Container orchestration frameworks like Kubernetes have brought about untold technological advances over the past decade. However, they have also enabled new attack vectors for bad actors to leverage. Before safely deploying an application, you must answer the following questions: How long should a container live? Does the container need to write any files during runtime?

Determining the container’s lifetime and the context in which it runs is critical, especially when hosting an internet-facing service.

What is container drift?

When deploying an application within Kubernetes or OpenShift, a container image must first be created from a build file. This image serves as a static snapshot of what an application will look like when it’s first executed at runtime.

These images comprise several layers, each typically mapped to a single command in the original build file. One image layer, often called the “container layer”, provides a space in which a container can write. This writable space is useful for various reasons, such as updating the underlying operating system components of a container at runtime, storing application artifacts or storing log files that can be used for troubleshooting.

However, having this writable space poses several security risks. The longer the container writes to this space, the less the container has in common with the image it was created from. Experts often refer to this phenomenon as container drift.

A security blind spot

As with any application running in a production environment, periodic security scans can detect software vulnerabilities. However, there are key differences when scanning a standalone application and scanning a program running within a container.

Due to performance concerns around running containers, security scans do not typically run against the container itself. Instead, the software either scans the associated image during the container build process or scans it periodically. Scanning the image not only avoids negatively impacting container runtime performance, but it also does not contain vulnerable software.

In a perfect world where containers are immutable, image scanning would mitigate the possibility of vulnerabilities being introduced at runtime. However, thanks to container drift, relying solely on image scanning can leave a huge gap in security.

Containers slowly change over time due to files being written to the container layer. This may be done programmatically, or developers may use a container orchestration framework application programming interface. These changes slowly modify the container, so the scanned image no longer accurately represents the container’s security posture. With no way to check the container itself, security analysts and product teams are dead in the water when determining the container’s risk.

While the writeable layer of a container is a security risk, the lifetime of a running container poses a more serious issue. Containers that exist for longer periods have a higher chance of deviating from the actual image they were instantiated from. Additionally, containers with a longer lifespan are likely to suffer from increased vulnerabilities, as the software libraries and applications within them will become outdated.

Mitigating container drift

With a better understanding of container drift, system owners and developers can proactively correct this issue in their environment. The most obvious step is to perform a rolling update using an updated image with the writeable layer disabled. While this might not be possible for all environments, it’s a good practice to disable the container layer where possible.

System administrators should also consider periodically restarting running containers to ensure that containers mirror the images they were instantiated from. This ensures that vulnerability scan reports accurately reflect a container’s security posture. In turn, this lets developers and system owners prioritize their patching efforts and know which components must be updated.

The last and possibly most important method of mitigating container drift is to have a software bill of materials (SBOM) for your container environment. Creating an SBOM can provide a baseline of software version information for all third-party libraries and application components running in your pods and containers. This document would be especially useful for cases where developers are utilizing the container layer for logging, and they need to know which version of Log4J is being used.

Securing containerized environments for the future

With the ever-increasing number of bad actors that are present in the wild today, it is imperative that system owners and developers are actively maintaining a secure containerized environment. System owners need to understand that container image scanning simply isn’t sufficient to effectively secure a container during runtime, and additional measures such as frequently performing rolling updates and container refreshes must be taken. As with all types of emerging technology, bad actors are constantly searching for new and inventive ways to leverage security flaws and we must remain vigilant if we want to stay one step ahead of the bad guys!