In late 2021, the Apache Software Foundation disclosed a vulnerability that set off a panic across the global tech industry. The bug, known as Log4Shell, was found in the ubiquitous open-source logging library Log4j, and it exposed a huge swath of applications and services.

Nearly anything from popular consumer and enterprise platforms to critical infrastructure and IoT devices was exposed. Over 35,000 Java packages were impacted by Log4j vulnerabilities. That’s over 8% of the Maven Central repository, the world’s largest Java package repository.

When Log4j was discovered, CISA Director Jen Easterly said, “This vulnerability is one of the most serious that I’ve seen in my entire career, if not the most serious.”

Since Log4j surfaced, how has the security community responded? What lessons have we learned (or not learned)?

Significant Lingering Threat

Log4Shell is no longer a massive, widespread danger. Still, researchers warn that the vulnerability is still present in far too many systems. And actors will continue to exploit it for years to come.

Log4Shell was unusual because it was so easy to exploit wherever it was present. Developers use logging utilities to record operations in applications. To exploit Log4Shell, all an attacker has to do is get the system to log a special string of code. From there, they can take control of their victim to install malware or launch other attacks.

“Logging is fundamental to essentially any computer software or hardware operation. Whether it’s a phlebotomy machine or an application server, logging is going to be present,” said David Nalley, president of the nonprofit Apache Software Foundation, in an interview with Wired. “We knew Log4j was widely deployed, we saw the download numbers, but it’s hard to fully grasp since in open source you’re not selling a product and tracking contracts. I don’t think you fully appreciate it until you have a full accounting of where software is, everything it’s doing and who’s using it. And I think the fact that it was so incredibly ubiquitous was a factor in everyone reacting so immediately. It’s a little humbling, frankly.”

According to Nalley, they had software fixes out within two weeks. Alarmingly, Apache still sees up to 25% of downloads involving non-patched versions of Log4j.

Continued Log4j Attack Incidents

Threat actors continue to exploit the Log4j vulnerability to this day. CISA has released alerts regarding Iranian and Chinese actors using the exploit. From Iran, cyber threat actors took advantage of the Log4Shell vulnerability in an unpatched VMware Horizon server, installed crypto mining software, moved laterally to the domain controller, compromised credentials and implanted reverse proxies on several hosts to maintain persistence. Meanwhile, the top Common Vulnerabilities and Exposures (CVEs) most used by Chinese state-sponsored cyber actors since 2020 is Log4j.

Given the danger and ongoing threat, why do so many vulnerable versions of Log4j still persist? Could it be that some IT pros don’t really know what’s in their software?

The Risk of Open-Source Software

The problem isn’t software vulnerability alone. It’s also not knowing if you have vulnerable code hiding your applications. Surprisingly, many security and IT professionals have no idea whether Log4j is part of their software supply chain. Or even worse, they choose to ignore the danger.

Part of the challenge is due to the rise of open-source software (OSS). Coders leverage OSS to accelerate development, cut costs and reduce time to market. Easy access to open-source frameworks and libraries takes the place of writing custom code or buying proprietary software. And while many applications get built quickly, the exact contents might not be known.

In a Linux Foundation SBOM and Cybersecurity Readiness report, 98% of organizations surveyed use open-source software. Due to the explosion of OSS use, it’s clear that supply chain cybersecurity may be impossible to gauge for any given application. If you don’t know what’s in your supply chain, how can you possibly know it’s secure?

Security Starts With SBOM

The threat of vulnerabilities (both known and zero-day) combined with the unknown contents of software packages has led security regulators and decision-makers to push for the development of software bills of materials.

According to CISA:

A “software bill of materials” (SBOM) has emerged as a key building block in software security and software supply chain risk management. An SBOM is a nested inventory, a list of ingredients that make up software components.

If you have a detailed list of individual software components, you can assess risk exposure more accurately. Also, with a well-developed SBOM, you can match your list against CISA’s Known Exploited Vulnerabilities Catalog. Or, if you hear about an emerging mass exploit like Log4j, you can quickly confirm if your stack is at risk. If you don’t have an SBOM, you’re in the dark until you are notified by your vendor or until you get hacked.

Finding Millions of Vulnerabilities

If you were to scan your systems for software vulnerabilities, you might discover hundreds of thousands of weaknesses. Also, if you merged with another company recently, you inherit their risk burden as well. For larger enterprises, detected vulnerabilities can number in the millions.

Trying to patch everything at once would be impossible. Instead, proper triage is essential. For example, vulnerabilities nearest to mission-critical systems should be prioritized. Also, an organization should audit, monitor and test its software vulnerability profile often. And since IT teams might add applications at any moment, an up-to-date network inventory and scheduled vulnerability scanning are critical. Automated software vulnerability management programs can be a great help here.

Many companies don’t have the time or qualified resources to identify, prioritize and remediate vulnerabilities. The process can be overwhelming. Given the high risk involved, some organizations opt to hire expert vulnerability mitigation services.

Still More to Learn

While Log4j sent some into a frenzy, others didn’t even seem to notice. This gives rise to the debate about cyber responsibility. If my partner hasn’t patched a vulnerability, and it affects my operations, should my partner be held responsible?

In one survey, 87% of respondents said that given the level of cyber risk posed by Log4j, government regulatory agencies (such as the U.S. Federal Trade Commission) should take legal action against organizations that fail to patch the flaw.

Only time will tell how far the security community will take responsibility for vulnerabilities — whether by being proactive or by force.

More from Risk Management

Container Drift: Where Age isn’t Just a Number

Container orchestration frameworks like Kubernetes have brought about untold technological advances over the past decade. However, they have also enabled new attack vectors for bad actors to leverage. Before safely deploying an application, you must answer the following questions: How long should a container live? Does the container need to write any files during runtime?Determining the container’s lifetime and the context in which it runs is critical, especially when hosting an internet-facing service.What is Container Drift?When deploying an application within…

OneNote, Many Problems? The New Phishing Framework

There are plenty of phish in the digital sea, and attackers are constantly looking for new bait that helps them bypass security perimeters and land in user inboxes. Their newest hook? OneNote documents. First noticed in December 2022, this phishing framework has seen success in fooling multiple antivirus (AV) tools by using .one file extensions, and January 2023 saw an attack uptick as compromises continued. While this novel notes approach will eventually be phased out as phishing defenses catch up,…

The Role of Finance Departments in Cybersecurity

Consumers are becoming more aware of the data companies collect about them, and place high importance on data security and privacy. Though consumers aren’t aware of every data breach, they are justifiably concerned about what happens to the data companies collect. A recent study of consumer views on data privacy and security revealed consumers are more careful about sharing data. The majority of respondents (87%) say they wouldn’t do business with companies that appear to have weak security. Study participants…

What Does a Network Security Engineer Do?

Cybersecurity is complex. The digital transformation, remote work and the ever-evolving threat landscape require different tools and different skill sets. Systems must be in place to protect endpoints, identities and a borderless network perimeter. The job role responsible for handling this complex security infrastructure is the network security engineer. In a nutshell, the network security engineer is the person who is responsible for the design and implementation of the organization’s security system, ensuring there are no gaps or vulnerabilities for…