If your work touches on the world of software development, you’ve likely heard the saying ‘software is eating the world’ by engineer/investor Marc Andreessen. He argued that building software was becoming the business and that it has completely taken over companies. But while many can stand to benefit greatly from the brave new world where software is the product, for many organizations, the side effects of security vulnerabilities in software are ‘eating business results’ in the form of damaged brand reputation, loss of client trust and financial fines.

With at least 17,447 new vulnerabilities disclosed in 2020 alone, companies are struggling to stay ahead of software vulnerabilities and compliance issues in order to keep their products and services afloat. Fueling this perpetual race is the intense push to deliver products and product updates faster than ever, a shortage of application security professionals, and complicated multi-cloud architectures and deployment environments that make it harder to have visibility into everything that could go wrong at a given time.

Should You ‘Shift Left’?

To get ahead of this vicious development cycle, organizations should be adopting a security-minded approach. Some call this mindset ‘shift left’ or ‘shift left testing.’ It means testing software earlier in the process and re-testing along the development cycle in a continuous manner. But this task can be daunting and rather hard to scale with tightening deadlines and greater demands on everyone involved in the development process — from developers to DevOps to security teams.

One way to scale the ability to test more code in less time is using smart automation and analytic tools. By building security into the DevOps process during code, build and run stages — instead of a separate set of tasks executed by the security team — you can detect security issues, vulnerabilities and places where malicious behavior can begin at their source, at scale and most importantly before they propagate into production supply chains.

Infusing security analytics into the development and build/deploy process not only helps identify potential security breaches but also provides early identification of artifacts for remediation, thereby eliminating the ‘after the fact’ finger-pointing that typically arises in post mortem security investigations.

The Target: Software Supply Chains

One of the most damaging types of attacks in recent years start with a software supply chain compromise. In terms of tactics, most recent attacks have shifted from brute-forcing software vulnerabilities to introducing ‘taps’ and backdoors into software as a part of the development supply chain. These are designed to allow attackers to access and poison software updates, then exploit them to breach any number of companies that use them.

The supply chain can be compromised in part due to a lack of security monitoring and oversight for the coding and delivery of software (continuous integration/continuous delivery (CI/CD) pipelines), which creates a dangerous security gap. This gap widens because security testing does not test for changes in the software systems. This security gap is also becoming more dangerous with the increasing rate of software updates in the era of cloud computing. Attackers have been increasingly exploiting this process weakness, with notable supply chain attacks coming to mind, such as NotPetya and SolarWinds.

Let’s take the SolarWinds attack as an example. The SolarWinds’ Orion software update mechanism was breached several months before the attackers started spreading the SUNBURST backdoor using the update mechanism. Attackers were able to run experiments on the CI/CD pipelines, seemingly undisturbed, to discover the best attack vector. Eventually, they were able to make code changes that turned the SolarWinds software updates into a malware-spreading machine.

From an attack-prevention perspective (testing), it is possible that all code security checks passed. But, from a security-monitoring perspective, it is likely that the CI/CD was not properly monitored for a malicious threat from within the trusted networks. Monitoring for reconnaissance activity and software artifact changes might have detected the threat before it advanced.

Better Security Monitoring for CI/CD Pipelines

Security monitoring of CI/CD pipelines and code changes can help detect supply chain attacks and add another vital security layer to a business-critical cycle. Monitoring for malicious activity that’s inside the network usually means detection of activities caused by breached accounts, which can be harder to find without contextual data. The National Institute for Standards and Technology advises companies to monitor access to network resources and maintain logs that can enable security teams to examine anomalies.

Another place to look for signs of trouble is where code might have been modified after it was approved. One of the main tools available today is the Git-diff tool that helps look at code differences introduced after work has been finalized. But while this tool is very helpful, it is not designed to keep up with the increasingly rapid software development life cycle. Git-diff ignores the CI/CD environment, and it assumes that reviewers have enough time to thoroughly inspect all code changes. Other solutions in this domain are designed to digitally sign build artifacts. These are useful to ensure the integrity of artifacts in the build and deployment processes, but they fail to monitor changes in code content and context.

Security monitoring is not getting easier in the cloud, where code underpins everything that once was built from hardware. With most organizations reliant on cloud assets, infrastructure as code (IaC) is highly crucial to the modern cloud CI/CD environment. IaC is used to provision resources in the cloud account (e.g., IBM Terraform) and to control the deployment of workloads to those resources (e.g., Kubernetes, Helm). The criticality of code that builds cloud infrastructure is one element that increases the need for proper security monitoring that can help detect malicious activity by users who might appear legitimate.

Automation to Detect Anomalous Code

To address the problem of creating content- and context-aware monitoring of IaC code changes, IBM Research developed and patented a tool that deploys detective cybersecurity controls on code. This tool detects anomalous code content and context-changing activities that are security specific.

The following is a high-level description of this tool’s capabilities:

1. Creates a resource call-graph from the IaC
2. Learns the history of IaC security-related attributes from source control; for example, user-lists, IP-lists and security-related configurations.
3. Monitors changes in IaC attributes during CI/CD. For example, upon pull request (a pull request occurs when a developer asks for changes committed to an external repository to be considered for inclusion in a project’s main repository) to main branch:
   1. Inspects these changes to detect anomalies
   2. Performs taint analysis on the resource call-graph to propagate the detected anomaly
   3. Alerts on anomalies that taint a high-risk resource

For instance, if there is a user-list that rarely changes, and a change is detected where one user was replaced with another, this change can be marked as an anomaly. Then it can be checked to see if this user-list has privileged access to a sensitive resource, such as admin access to the logging resource. If this is the case, there is a security risk to address because this new user can now make changes to a security-sensitive resource (logging service), and an alert should be issued about it.

In a more interesting example, a huge pull request is made with many changes affecting IaC functions, variables and the CI/CD environment. One of the changes creates a new service authorization policy. In this case, it is necessary to understand the IaC semantics to monitor for changes. Also, the history of the service authorization policies needs to be examined to determine if this change is anomalous and a taint analysis needs to be conducted to determine if the policy inadvertently permits high-risk security activity.

IBM Research experimented with this new tool by artificially introducing suspicious activity into IaC code examples. We were able to detect anomalous changes in the IaC code examples for IBM Terraform. The following are three examples of anomaly detection in IBM Terraform code:

• A change was made to the CI/CD environment. This change sets an environment variable value that overrides a default value in the code. The change is undetected in source control because it does not change the code. It changes the IaC deployed due to the nature of environment variable handling. The change was detected in a pull request to the main branch, and an alert was issued as a pull request comment.
• Another use case is detecting anomalous behavior. A high number of updates were made to a certificate manager resource within several hours. This activity can indicate reconnaissance where a malicious actor is testing the boundaries of the security system. This activity was detected as part of the CI pipeline monitoring.
• A major pull request was made with changes that affect an allowed IP-list of a Cloud Object Storage database. Detection of the change required filtering out all the other changes made to this pull request.

Compared to a simple Git-diff, the anomalous code detection tool provides better security monitoring capabilities for each change in the IaC. The changes detected will usually escape compliance testing checks because those checks ignore changes. For example, ‘ensure number of admins is less than X’ will not detect a change in the admin list that replaces one administrator with another. Compared to run-time detective controls (like user and entity behavior analytics) this mechanism can help prevent malicious-code changes from being deployed.

Infusing the DevOps Process With Control

At IBM Research, we have seen how security-monitoring use cases can be injected into the DevOps process, and how DevSecOps can take more responsibility in the security monitoring use case. As the industry is at the highest risk than ever to supply chain attacks, the ‘shift left’ of more security use cases into DevSecOps is a highly valuable approach.

This blog is based on a patent filed by IBM in April 2021: “User and Entity Behavior Analytics of infrastructure as code in pre-deployment of cloud infrastructure, Fady Copty, Omri Soceanu, Lev Greenberg, Dov Murik”. We would like to thank Jenny Lerner for her contribution to the implementation and experimentation of this technology.