March 22, 2023 By Richard Howe 3 min read

Container orchestration frameworks like Kubernetes have brought about untold technological advances over the past decade. However, they have also enabled new attack vectors for bad actors to leverage. Before safely deploying an application, you must answer the following questions: How long should a container live? Does the container need to write any files during runtime?

Determining the container’s lifetime and the context in which it runs is critical, especially when hosting an internet-facing service.

What is container drift?

When deploying an application within Kubernetes or OpenShift, a container image must first be created from a build file. This image serves as a static snapshot of what an application will look like when it’s first executed at runtime.

These images comprise several layers, each typically mapped to a single command in the original build file. One image layer, often called the “container layer”, provides a space in which a container can write. This writable space is useful for various reasons, such as updating the underlying operating system components of a container at runtime, storing application artifacts or storing log files that can be used for troubleshooting.

However, having this writable space poses several security risks. The longer the container writes to this space, the less the container has in common with the image it was created from. Experts often refer to this phenomenon as container drift.

A security blind spot

As with any application running in a production environment, periodic security scans can detect software vulnerabilities. However, there are key differences when scanning a standalone application and scanning a program running within a container.

Due to performance concerns around running containers, security scans do not typically run against the container itself. Instead, the software either scans the associated image during the container build process or scans it periodically. Scanning the image not only avoids negatively impacting container runtime performance, but it also does not contain vulnerable software.

In a perfect world where containers are immutable, image scanning would mitigate the possibility of vulnerabilities being introduced at runtime. However, thanks to container drift, relying solely on image scanning can leave a huge gap in security.

Containers slowly change over time due to files being written to the container layer. This may be done programmatically, or developers may use a container orchestration framework application programming interface. These changes slowly modify the container, so the scanned image no longer accurately represents the container’s security posture. With no way to check the container itself, security analysts and product teams are dead in the water when determining the container’s risk.

While the writeable layer of a container is a security risk, the lifetime of a running container poses a more serious issue. Containers that exist for longer periods have a higher chance of deviating from the actual image they were instantiated from. Additionally, containers with a longer lifespan are likely to suffer from increased vulnerabilities, as the software libraries and applications within them will become outdated.

Mitigating container drift

With a better understanding of container drift, system owners and developers can proactively correct this issue in their environment. The most obvious step is to perform a rolling update using an updated image with the writeable layer disabled. While this might not be possible for all environments, it’s a good practice to disable the container layer where possible.

System administrators should also consider periodically restarting running containers to ensure that containers mirror the images they were instantiated from. This ensures that vulnerability scan reports accurately reflect a container’s security posture. In turn, this lets developers and system owners prioritize their patching efforts and know which components must be updated.

The last and possibly most important method of mitigating container drift is to have a software bill of materials (SBOM) for your container environment. Creating an SBOM can provide a baseline of software version information for all third-party libraries and application components running in your pods and containers. This document would be especially useful for cases where developers are utilizing the container layer for logging, and they need to know which version of Log4J is being used.

Securing containerized environments for the future

With the ever-increasing number of bad actors that are present in the wild today, it is imperative that system owners and developers are actively maintaining a secure containerized environment. System owners need to understand that container image scanning simply isn’t sufficient to effectively secure a container during runtime, and additional measures such as frequently performing rolling updates and container refreshes must be taken. As with all types of emerging technology, bad actors are constantly searching for new and inventive ways to leverage security flaws and we must remain vigilant if we want to stay one step ahead of the bad guys!

More from Risk Management

Working in the security clearance world: How security clearances impact jobs

2 min read - We recently published an article about the importance of security clearances for roles across various sectors, particularly those associated with national security and defense.But obtaining a clearance is only part of the journey. Maintaining and potentially expanding your clearance over time requires continued diligence and adherence to stringent guidelines.This brief explainer discusses the duration of security clearances, the recurring processes involved in maintaining them and possibilities for expansion, as well as the economic benefits of these credentialed positions.Duration of security…

Remote access risks on the rise with CVE-2024-1708 and CVE-2024-1709

4 min read - On February 19, ConnectWise reported two vulnerabilities in its ScreenConnect product, CVE-2024-1708 and 1709. The first is an authentication bypass vulnerability, and the second is a path traversal vulnerability. Both made it possible for attackers to bypass authentication processes and execute remote code.While ConnectWise initially reported that the vulnerabilities had proof-of-concept but hadn’t been spotted in the wild, reports from customers quickly made it clear that hackers were actively exploring both flaws. As a result, the company created patches for…

Researchers develop malicious AI ‘worm’ targeting generative AI systems

2 min read - Researchers have created a new, never-seen-before kind of malware they call the "Morris II" worm, which uses popular AI services to spread itself, infect new systems and steal data. The name references the original Morris computer worm that wreaked havoc on the internet in 1988.The worm demonstrates the potential dangers of AI security threats and creates a new urgency around securing AI models.New worm utilizes adversarial self-replicating promptThe researchers from Cornell Tech, the Israel Institute of Technology and Intuit, used what’s…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today