Container orchestration frameworks like Kubernetes have brought about untold technological advances over the past decade. However, they have also enabled new attack vectors for bad actors to leverage. Before safely deploying an application, you must answer the following questions: How long should a container live? Does the container need to write any files during runtime?

Determining the container’s lifetime and the context in which it runs is critical, especially when hosting an internet-facing service.

What is Container Drift?

When deploying an application within Kubernetes or OpenShift, a container image must first be created from a build file. This image serves as a static snapshot of what an application will look like when it’s first executed at runtime.

These images comprise several layers, each typically mapped to a single command in the original build file. One image layer, often called the “container layer”, provides a space in which a container can write. This writable space is useful for various reasons, such as updating the underlying operating system components of a container at runtime, storing application artifacts or storing log files that can be used for troubleshooting.

However, having this writable space poses several security risks. The longer the container writes to this space, the less the container has in common with the image it was created from. Experts often refer to this phenomenon as container drift.

A Security Blind Spot

As with any application running in a production environment, periodic security scans can detect software vulnerabilities. However, there are key differences when scanning a standalone application and scanning a program running within a container.

Due to performance concerns around running containers, security scans do not typically run against the container itself. Instead, the software either scans the associated image during the container build process or scans it periodically. Scanning the image not only avoids negatively impacting container runtime performance, but it also does not contain vulnerable software.

In a perfect world where containers are immutable, image scanning would mitigate the possibility of vulnerabilities being introduced at runtime. However, thanks to container drift, relying solely on image scanning can leave a huge gap in security.

Containers slowly change over time due to files being written to the container layer. This may be done programmatically, or developers may use a container orchestration framework application programming interface. These changes slowly modify the container, so the scanned image no longer accurately represents the container’s security posture. With no way to check the container itself, security analysts and product teams are dead in the water when determining the container’s risk.

While the writeable layer of a container is a security risk, the lifetime of a running container poses a more serious issue. Containers that exist for longer periods have a higher chance of deviating from the actual image they were instantiated from. Additionally, containers with a longer lifespan are likely to suffer from increased vulnerabilities, as the software libraries and applications within them will become outdated.

Mitigating Container Drift

With a better understanding of container drift, system owners and developers can proactively correct this issue in their environment. The most obvious step is to perform a rolling update using an updated image with the writeable layer disabled. While this might not be possible for all environments, it’s a good practice to disable the container layer where possible.

System administrators should also consider periodically restarting running containers to ensure that containers mirror the images they were instantiated from. This ensures that vulnerability scan reports accurately reflect a container’s security posture. In turn, this lets developers and system owners prioritize their patching efforts and know which components must be updated.

The last and possibly most important method of mitigating container drift is to have a software bill of materials (SBOM) for your container environment. Creating an SBOM can provide a baseline of software version information for all third-party libraries and application components running in your pods and containers. This document would be especially useful for cases where developers are utilizing the container layer for logging, and they need to know which version of Log4J is being used.

Securing Containerized Environments for the Future

With the ever-increasing number of bad actors that are present in the wild today, it is imperative that system owners and developers are actively maintaining a secure containerized environment. System owners need to understand that container image scanning simply isn’t sufficient to effectively secure a container during runtime, and additional measures such as frequently performing rolling updates and container refreshes must be taken. As with all types of emerging technology, bad actors are constantly searching for new and inventive ways to leverage security flaws and we must remain vigilant if we want to stay one step ahead of the bad guys!

More from Risk Management

Security Awareness Training 101: Which Employees Need It?

4 min read - To understand why you need cybersecurity awareness training, you must first understand employees' outsized roles in security breaches. “People remain — by far — the weakest link in an organization’s cybersecurity defenses,” noted Verizon on the release of their 2022 Data Breach Investigations Report (DBIR). They elaborate that 25% of all breaches covered in the report were the result of social engineering attacks, and when you add human errors and misuse of privilege, the human element accounts for 82% of…

4 min read

Secure-by-Design: Which Comes First, Code or Security?

4 min read - For years, developers and IT security teams have been at loggerheads. While developers feel security slows progress, security teams assert that developers sacrifice security priorities in their quest to accelerate production. This disconnect results in flawed software that is vulnerable to attack. While advocates for speed and security clash, consumers must often pay the price when threat actors strike. 48% of developers admitted they were still shipping code with vulnerabilities in 2022. It’s clearly time for a change. Many believe…

4 min read

Will Commercial Spyware Survive Biden’s Executive Order?

4 min read - On March 27, 2023, reports surfaced that 50 U.S. government employees had been targeted by phone spyware overseas. On the day of that report, President Joe Biden signed an executive order to restrict federal agencies’ use of commercial spyware. The timing of the order was linked to this specific phone-targeting exploit. But spyware infiltration of government officials — and by government officials — has been a recurring problem globally. Commercial spyware has long been entwined with statecraft and spycraft, both…

4 min read

How to Boost Cybersecurity Through Better Communication

4 min read - Security would be easy without users. That statement is as absurd as it is true. It’s also true that business wouldn’t be possible without users. It’s time to look at the big picture when it comes to cybersecurity. In addition to dealing with every new risk, vulnerability and attack vector that comes along, cybersecurity pros need to understand their own fellow employees - how they think, how they learn and what they really want. The human element — the individual and…

4 min read