In late 2021, the Apache Software Foundation disclosed a vulnerability that set off a panic across the global tech industry. The bug, known as Log4Shell, was found in the ubiquitous open-source logging library Log4j, and it exposed a huge swath of applications and services.

Nearly anything from popular consumer and enterprise platforms to critical infrastructure and IoT devices was exposed. Over 35,000 Java packages were impacted by Log4j vulnerabilities. That’s over 8% of the Maven Central repository, the world’s largest Java package repository.

When Log4j was discovered, CISA Director Jen Easterly said, “This vulnerability is one of the most serious that I’ve seen in my entire career, if not the most serious.”

Since Log4j surfaced, how has the security community responded? What lessons have we learned (or not learned)?

Significant lingering threat

Log4Shell is no longer a massive, widespread danger. Still, researchers warn that the vulnerability is still present in far too many systems. And actors will continue to exploit it for years to come.

Log4Shell was unusual because it was so easy to exploit wherever it was present. Developers use logging utilities to record operations in applications. To exploit Log4Shell, all an attacker has to do is get the system to log a special string of code. From there, they can take control of their victim to install malware or launch other attacks.

“Logging is fundamental to essentially any computer software or hardware operation. Whether it’s a phlebotomy machine or an application server, logging is going to be present,” said David Nalley, president of the nonprofit Apache Software Foundation, in an interview with Wired. “We knew Log4j was widely deployed, we saw the download numbers, but it’s hard to fully grasp since in open source you’re not selling a product and tracking contracts. I don’t think you fully appreciate it until you have a full accounting of where software is, everything it’s doing and who’s using it. And I think the fact that it was so incredibly ubiquitous was a factor in everyone reacting so immediately. It’s a little humbling, frankly.”

According to Nalley, they had software fixes out within two weeks. Alarmingly, Apache still sees up to 25% of downloads involving non-patched versions of Log4j.

Continued Log4j attack incidents

Threat actors continue to exploit the Log4j vulnerability to this day. CISA has released alerts regarding Iranian and Chinese actors using the exploit. From Iran, cyber threat actors took advantage of the Log4Shell vulnerability in an unpatched VMware Horizon server, installed crypto mining software, moved laterally to the domain controller, compromised credentials and implanted reverse proxies on several hosts to maintain persistence. Meanwhile, the top Common Vulnerabilities and Exposures (CVEs) most used by Chinese state-sponsored cyber actors since 2020 is Log4j.

Given the danger and ongoing threat, why do so many vulnerable versions of Log4j still persist? Could it be that some IT pros don’t really know what’s in their software?

The risk of open-source software

The problem isn’t software vulnerability alone. It’s also not knowing if you have vulnerable code hiding your applications. Surprisingly, many security and IT professionals have no idea whether Log4j is part of their software supply chain. Or even worse, they choose to ignore the danger.

Part of the challenge is due to the rise of open-source software (OSS). Coders leverage OSS to accelerate development, cut costs and reduce time to market. Easy access to open-source frameworks and libraries takes the place of writing custom code or buying proprietary software. And while many applications get built quickly, the exact contents might not be known.

In a Linux Foundation SBOM and Cybersecurity Readiness report, 98% of organizations surveyed use open-source software. Due to the explosion of OSS use, it’s clear that supply chain cybersecurity may be impossible to gauge for any given application. If you don’t know what’s in your supply chain, how can you possibly know it’s secure?

Security starts with SBOM

The threat of vulnerabilities (both known and zero-day) combined with the unknown contents of software packages has led security regulators and decision-makers to push for the development of software bills of materials.

According to CISA:

A “software bill of materials” (SBOM) has emerged as a key building block in software security and software supply chain risk management. An SBOM is a nested inventory, a list of ingredients that make up software components.

If you have a detailed list of individual software components, you can assess risk exposure more accurately. Also, with a well-developed SBOM, you can match your list against CISA’s Known Exploited Vulnerabilities Catalog. Or, if you hear about an emerging mass exploit like Log4j, you can quickly confirm if your stack is at risk. If you don’t have an SBOM, you’re in the dark until you are notified by your vendor or until you get hacked.

Finding millions of vulnerabilities

If you were to scan your systems for software vulnerabilities, you might discover hundreds of thousands of weaknesses. Also, if you merged with another company recently, you inherit their risk burden as well. For larger enterprises, detected vulnerabilities can number in the millions.

Trying to patch everything at once would be impossible. Instead, proper triage is essential. For example, vulnerabilities nearest to mission-critical systems should be prioritized. Also, an organization should audit, monitor and test its software vulnerability profile often. And since IT teams might add applications at any moment, an up-to-date network inventory and scheduled vulnerability scanning are critical. Automated software vulnerability management programs can be a great help here.

Many companies don’t have the time or qualified resources to identify, prioritize and remediate vulnerabilities. The process can be overwhelming. Given the high risk involved, some organizations opt to hire expert vulnerability mitigation services.

Still more to learn

While Log4j sent some into a frenzy, others didn’t even seem to notice. This gives rise to the debate about cyber responsibility. If my partner hasn’t patched a vulnerability, and it affects my operations, should my partner be held responsible?

In one survey, 87% of respondents said that given the level of cyber risk posed by Log4j, government regulatory agencies (such as the U.S. Federal Trade Commission) should take legal action against organizations that fail to patch the flaw.

Only time will tell how far the security community will take responsibility for vulnerabilities — whether by being proactive or by force.