If your work touches on the world of software development, you’ve likely heard the saying ‘software is eating the world’ by engineer/investor Marc Andreessen. He argued that building software was becoming the business and that it has completely taken over companies. But while many can stand to benefit greatly from the brave new world where software is the product, for many organizations, the side effects of security vulnerabilities in software are ‘eating business results’ in the form of damaged brand reputation, loss of client trust and financial fines.

With at least 17,447 new vulnerabilities disclosed in 2020 alone, companies are struggling to stay ahead of software vulnerabilities and compliance issues in order to keep their products and services afloat. Fueling this perpetual race is the intense push to deliver products and product updates faster than ever, a shortage of application security professionals, and complicated multi-cloud architectures and deployment environments that make it harder to have visibility into everything that could go wrong at a given time.

Should You ‘Shift Left’?

To get ahead of this vicious development cycle, organizations should be adopting a security-minded approach. Some call this mindset ‘shift left’ or ‘shift left testing.’ It means testing software earlier in the process and re-testing along the development cycle in a continuous manner. But this task can be daunting and rather hard to scale with tightening deadlines and greater demands on everyone involved in the development process — from developers to DevOps to security teams.

One way to scale the ability to test more code in less time is using smart automation and analytic tools. By building security into the DevOps process during code, build and run stages — instead of a separate set of tasks executed by the security team — you can detect security issues, vulnerabilities and places where malicious behavior can begin at their source, at scale and most importantly before they propagate into production supply chains.

Infusing security analytics into the development and build/deploy process not only helps identify potential security breaches but also provides early identification of artifacts for remediation, thereby eliminating the ‘after the fact’ finger-pointing that typically arises in post mortem security investigations.

The Target: Software Supply Chains

One of the most damaging types of attacks in recent years start with a software supply chain compromise. In terms of tactics, most recent attacks have shifted from brute-forcing software vulnerabilities to introducing ‘taps’ and backdoors into software as a part of the development supply chain. These are designed to allow attackers to access and poison software updates, then exploit them to breach any number of companies that use them.

The supply chain can be compromised in part due to a lack of security monitoring and oversight for the coding and delivery of software (continuous integration/continuous delivery (CI/CD) pipelines), which creates a dangerous security gap. This gap widens because security testing does not test for changes in the software systems. This security gap is also becoming more dangerous with the increasing rate of software updates in the era of cloud computing. Attackers have been increasingly exploiting this process weakness, with notable supply chain attacks coming to mind, such as NotPetya and SolarWinds.

Let’s take the SolarWinds attack as an example. The SolarWinds’ Orion software update mechanism was breached several months before the attackers started spreading the SUNBURST backdoor using the update mechanism. Attackers were able to run experiments on the CI/CD pipelines, seemingly undisturbed, to discover the best attack vector. Eventually, they were able to make code changes that turned the SolarWinds software updates into a malware-spreading machine.

From an attack-prevention perspective (testing), it is possible that all code security checks passed. But, from a security-monitoring perspective, it is likely that the CI/CD was not properly monitored for a malicious threat from within the trusted networks. Monitoring for reconnaissance activity and software artifact changes might have detected the threat before it advanced.

Better Security Monitoring for CI/CD Pipelines

Security monitoring of CI/CD pipelines and code changes can help detect supply chain attacks and add another vital security layer to a business-critical cycle. Monitoring for malicious activity that’s inside the network usually means detection of activities caused by breached accounts, which can be harder to find without contextual data. The National Institute for Standards and Technology advises companies to monitor access to network resources and maintain logs that can enable security teams to examine anomalies.

Another place to look for signs of trouble is where code might have been modified after it was approved. One of the main tools available today is the Git-diff tool that helps look at code differences introduced after work has been finalized. But while this tool is very helpful, it is not designed to keep up with the increasingly rapid software development life cycle. Git-diff ignores the CI/CD environment, and it assumes that reviewers have enough time to thoroughly inspect all code changes. Other solutions in this domain are designed to digitally sign build artifacts. These are useful to ensure the integrity of artifacts in the build and deployment processes, but they fail to monitor changes in code content and context.

Security monitoring is not getting easier in the cloud, where code underpins everything that once was built from hardware. With most organizations reliant on cloud assets, infrastructure as code (IaC) is highly crucial to the modern cloud CI/CD environment. IaC is used to provision resources in the cloud account (e.g., IBM Terraform) and to control the deployment of workloads to those resources (e.g., Kubernetes, Helm). The criticality of code that builds cloud infrastructure is one element that increases the need for proper security monitoring that can help detect malicious activity by users who might appear legitimate.

Automation to Detect Anomalous Code

To address the problem of creating content- and context-aware monitoring of IaC code changes, IBM Research developed and patented a tool that deploys detective cybersecurity controls on code. This tool detects anomalous code content and context-changing activities that are security specific.

The following is a high-level description of this tool’s capabilities:

  1. Creates a resource call-graph from the IaC
  2. Learns the history of IaC security-related attributes from source control; for example, user-lists, IP-lists and security-related configurations.
  3. Monitors changes in IaC attributes during CI/CD. For example, upon pull request (a pull request occurs when a developer asks for changes committed to an external repository to be considered for inclusion in a project’s main repository) to main branch:
    1. Inspects these changes to detect anomalies
    2. Performs taint analysis on the resource call-graph to propagate the detected anomaly
    3. Alerts on anomalies that taint a high-risk resource

For instance, if there is a user-list that rarely changes, and a change is detected where one user was replaced with another, this change can be marked as an anomaly. Then it can be checked to see if this user-list has privileged access to a sensitive resource, such as admin access to the logging resource. If this is the case, there is a security risk to address because this new user can now make changes to a security-sensitive resource (logging service), and an alert should be issued about it.

In a more interesting example, a huge pull request is made with many changes affecting IaC functions, variables and the CI/CD environment. One of the changes creates a new service authorization policy. In this case, it is necessary to understand the IaC semantics to monitor for changes. Also, the history of the service authorization policies needs to be examined to determine if this change is anomalous and a taint analysis needs to be conducted to determine if the policy inadvertently permits high-risk security activity.

IBM Research experimented with this new tool by artificially introducing suspicious activity into IaC code examples. We were able to detect anomalous changes in the IaC code examples for IBM Terraform. The following are three examples of anomaly detection in IBM Terraform code:

  • A change was made to the CI/CD environment. This change sets an environment variable value that overrides a default value in the code. The change is undetected in source control because it does not change the code. It changes the IaC deployed due to the nature of environment variable handling. The change was detected in a pull request to the main branch, and an alert was issued as a pull request comment.
  • Another use case is detecting anomalous behavior. A high number of updates were made to a certificate manager resource within several hours. This activity can indicate reconnaissance where a malicious actor is testing the boundaries of the security system. This activity was detected as part of the CI pipeline monitoring.
  • A major pull request was made with changes that affect an allowed IP-list of a Cloud Object Storage database. Detection of the change required filtering out all the other changes made to this pull request.

Compared to a simple Git-diff, the anomalous code detection tool provides better security monitoring capabilities for each change in the IaC. The changes detected will usually escape compliance testing checks because those checks ignore changes. For example, ‘ensure number of admins is less than X’ will not detect a change in the admin list that replaces one administrator with another. Compared to run-time detective controls (like user and entity behavior analytics) this mechanism can help prevent malicious-code changes from being deployed.

Infusing the DevOps Process With Control

At IBM Research, we have seen how security-monitoring use cases can be injected into the DevOps process, and how DevSecOps can take more responsibility in the security monitoring use case. As the industry is at the highest risk than ever to supply chain attacks, the ‘shift left’ of more security use cases into DevSecOps is a highly valuable approach.

This blog is based on a patent filed by IBM in April 2021: “User and Entity Behavior Analytics of infrastructure as code in pre-deployment of cloud infrastructure, Fady Copty, Omri Soceanu, Lev Greenberg, Dov Murik”. We would like to thank Jenny Lerner for her contribution to the implementation and experimentation of this technology.

More from Application Security

Gozi strikes again, targeting banks, cryptocurrency and more

3 min read - In the world of cybercrime, malware plays a prominent role. One such malware, Gozi, emerged in 2006 as Gozi CRM, also known as CRM or Papras. Initially offered as a crime-as-a-service (CaaS) platform called 76Service, Gozi quickly gained notoriety for its advanced capabilities. Over time, Gozi underwent a significant transformation and became associated with other malware strains, such as Ursnif (Snifula) and Vawtrak/Neverquest. Now, in a recent campaign, Gozi has set its sights on banks, financial services and cryptocurrency platforms,…

Vulnerability management, its impact and threat modeling methodologies

7 min read - Vulnerability management is a security practice designed to avoid events that could potentially harm an organization. It is a regular ongoing process that identifies, assesses, and manages vulnerabilities across all the components of an IT ecosystem. Cybersecurity is one of the major priorities many organizations struggle to stay on top of. There is a huge increase in the number of cyberattacks carried out by cybercriminals to steal valuable information from businesses. Hence to encounter these attacks, organizations are now focusing…

X-Force releases detection & response framework for managed file transfer software

5 min read - How AI can help defenders scale detection guidance for enterprise software tools If we look back at mass exploitation events that shook the security industry like Log4j, Atlassian, and Microsoft Exchange when these solutions were actively being exploited by attackers, the exploits may have been associated with a different CVE, but the detection and response guidance being released by the various security vendors had many similarities (e.g., Log4shell vs. Log4j2 vs. MOVEit vs. Spring4Shell vs. Microsoft Exchange vs. ProxyShell vs.…

Unmasking hypnotized AI: The hidden risks of large language models

11 min read - The emergence of Large Language Models (LLMs) is redefining how cybersecurity teams and cybercriminals operate. As security teams leverage the capabilities of generative AI to bring more simplicity and speed into their operations, it's important we recognize that cybercriminals are seeking the same benefits. LLMs are a new type of attack surface poised to make certain types of attacks easier, more cost-effective, and even more persistent. In a bid to explore security risks posed by these innovations, we attempted to…