For full details on this research, see the X-Force Red whitepaper “Controlling the Source: Abusing Source Code Management Systems”. This material is also being presented at Black Hat USA 2022.

Source Code Management (SCM) systems play a vital role within organizations and have been an afterthought in terms of defenses compared to other critical enterprise systems such as Active Directory. SCM systems are used in the majority of organizations to manage source code and integrate with other systems within the enterprise as part of the DevOps pipeline, such as CI/CD systems like Jenkins. These SCM systems provide attackers with opportunities for software supply chain attacks and can facilitate lateral movement and privilege escalation throughout an organization.

This blog post will review a background on SCM systems, along with detailing ways to abuse some of the most popular SCM systems including GitHub Enterprise, GitLab Enterprise and Bitbucket to perform various attack scenarios. These attack scenarios include reconnaissance, manipulation of user roles, repository takeover, pivoting to other DevOps systems, user impersonation, and maintaining persistent access. X-Force Red’s source code management attack toolkit (SCMKit) will also be shown to perform and facilitate these attacks. Additionally, defensive guidance for protecting these SCM systems will be outlined.

Background

There are many ways to interact with and track source code, along with compiled source code assets. Some of the common terms used in this process are source control, version control, and source code management. The terms “source control” and “version control” are often used interchangeably with each other. However, there are differences between these two terms. Source control is specifically for tracking changes in source code, whereas version control also includes tracking changes for binary files and other file types. An example of this would be version control tracking changes to compiled executables, whereas source control would be tracking the changes to the underlying C# or C++ source files that were compiled into that executable. Git is a popular source control tool, and Subversion is a popular version control tool.

To use source control in a practical manner as part of the development process, source code management (SCM) systems are used. These systems allow tracking changes to source code repositories and allow developers to resolve conflicts when merging code commits from multiple people concurrently. Some of these SCM systems are more popular than others and have been adopted by enterprises, as they integrate into the development process in a more reliable manner. Some of the popular SCM systems include GitHub Enterprise, GitLab Enterprise and Bitbucket. Due to the role and importance of these SCM systems, they can be abused to facilitate software supply chain attacks and lateral movement within an organization.

SCM Systems and the DevOps Pipeline

SCM systems are heavily used during the “build” phase of a project in the DevOps pipeline, as shown in the below diagram. All other phases depend on the source code that is developed and maintained within the SCM system. Once a source code project is ready to be compiled and built, it will get pushed to a Continuous Integration (CI) server. After that, it will be tested, scanned, and deployed for use in production.

DevOps Pipeline Diagram

Software Supply Chain Attacks

An attack that has been gaining popularity recently is software supply chain attacks. In this attack, an attacker injects itself into the development process at one of the phases to deploy malicious code into production. This is typically performed in the “build” phase. For organizations that provide software to other organizations, this can enable the compromise of multiple organizations. One of the most notable software supply chain attacks was the SolarWinds breach, which impacted many organizations in the private and public sector.

The below diagram shows the opportunities an attacker has during the development process to implement a software supply chain attack. This research focuses on the highlighted areas of “B” and “C”, as it relates to the compromise of SCM systems. However, the compromise of these SCM systems can also lead to other scenarios such as “D” where an attacker can use an SCM system to compromise a build platform system.

Software Supply Chain Attack Opportunity Diagram

Lateral Movement to Other DevOps Systems

SCM systems can be used as an initial access point to other DevOps systems that are used in separate phases of the DevOps lifecycle. Being able to pivot to the build system to compromise the CI/CD platform or pivoting to the package repository system to compromise the distribution platform are other scenarios where an attacker could perform a software supply chain attack. Two detailed examples of this are shown in the whitepaper.

Attack Scenarios

The below attack scenarios are notable for an attacker to attempt against GitHub Enterprise, GitLab Enterprise, and Bitbucket. These have been useful as a part of X-Force Red’s Adversary Simulation engagements. This is not an exhaustive list of every single attack path available to execute on these SCM systems. The table, below, lists the attack scenarios that are detailed in the whitepaper. A few of these scenarios will be shown in subsequent sections.

Table of SCM attack scenarios

Repository Takeover

Using administrative access (specifically site admin role), an attacker can give themselves write access to any repository within GitHub Enterprise. In the below example, we are attempting to view a repository in GitHub Enterprise that our compromised administrative user (adumbledore) does not have access to.

Viewing locked repository

Using the site admin role, you can choose to unlock the repository via the “Unlock” button shown below. This will unlock the repository for the user for two hours by default.

Viewing screen to unlock repository

You must provide a reason to unlock the repository, and this reason is logged along with the request.

Adding reason to unlocking repository

Now you can see we have successfully unlocked the repository, and it is unlocked for two hours for the adumbledore user account. Code within this repository can then be accessed or modified.

Showing repository has been unlocked

User Impersonation

There are two options an attacker has if they have administrative access to GitLab Enterprise or GitHub Enterprise and would like to impersonate another user. The first option is to impersonate a user login via the web interface, and the second option is to create an impersonation token. This will be shown specifically with GitLab Enterprise.

Impersonate User Login

When viewing a user via the admin area in GitLab Enterprise, there is a button available in the top right-hand corner labeled “Impersonate”.

Impersonate user button in hpotter profile

After clicking the “Impersonate” button, you will be logged in as the user you are wanting to impersonate. In this instance, we are impersonating the hpotter user account.

Showing impersonation of hpotter

Impersonation Token

An attacker with admin access to GitLab Enterprise can also impersonate another user by creating an impersonation token. This can be performed via the web interface or the Users REST API. Using the web interface as an admin, you can navigate to the “Impersonation Tokens” section for the user account that you would like to impersonate; add the details for your token including name, expiration date, and scope of permissions.

Creating impersonation token

After you have created your impersonation token, the token value will be listed for use. The user that is impersonated cannot see this impersonation token when accessing GitLab Enterprise as themselves; it is only visible to other admin users.

Showing created impersonation token

An attacker can also create an impersonation token via the Users REST API as shown with the example curl command, below.

<strong>curl</strong> -k --request POST --header "PRIVATE-TOKEN: apiToken" --data "name=someName-impersonate" --data "expires_at=" --data "scopes[]=api" --data "scopes[]=read_user" --data "scopes[]=read_repository" --data "scopes[]=write_repository" --data "scopes[]=sudo" "https://gitlabHost/api/v4/users/userIDNumberToImpersonate/impersonation_tokens"

Output after creating impersonation token via API

Modifying CI/CD Pipeline

In Bitbucket, there is a feature called Bamboo that can be installed and configured to facilitate a CI/CD pipeline. If a repository is using a CI/CD pipeline with Bamboo, it will contain a directory named “bamboo-specs” within the root of the repository, along with a Bamboo configuration file. This configuration file will either be a YAML file (bamboo.yaml) or a Java spec file (pom.xml). If an attacker would like to discover any repositories that are configured with a CI/CD pipeline via Bamboo, they can search for “bamboo-specs” in either the web interface or REST API.

Discovering repos with CI/CD integration via Bamboo

If you have write access or admin access to a repository, the Bamboo configuration file can be modified. In this case, we are modifying the bamboo.yaml file to add our SSH key to the server where the Bamboo agent is running. This can be performed via the Git command line tool as well to commit the changes to the Bamboo configuration file.

Modifying Bamboo yaml file

This will immediately trigger the CI/CD pipeline to run as shown below.

Showing successful job status

When viewing the output from the pipeline, we can see our SSH key was added, and it printed the hostname of the server where the SSH key was added. At this point, we would be able to perform lateral movement to that server via our SSH key.

Viewing pipeline logs

Maintain Persistent Access

An attacker has three primary options in terms of maintaining persistent access to SCM systems. This can be performed either by creating a personal access token, impersonation token, or adding a public SSH key. All these options are detailed in the whitepaper.

The persistence option that will be shown is creating a personal access token in GitLab Enterprise. This can be performed via the web interface as a regular user or can be performed via the Users REST API as an administrator. The below screenshot shows creating a personal access token called “persistence-token”.

Creating personal access token for hpotter user

You can see the created personal access token and the token value, below.

Showing token value created

An attacker can also create a personal access token via the Users REST API as shown with the below example curl command. In GitLab Enterprise, this requires admin permissions.

<strong>curl</strong> -k --request POST --header "PRIVATE-TOKEN: apiToken" --data "name=hgranger-persistence-token" --data "expires_at=" --data "scopes[]=api" --data "scopes[]=read_repository" --data "scopes[]=write_repository" "https://gitlabHost/api/v4/users/UserIDNumber/personal_access_tokens"

Creating access token via API

SCMKit

Background

At X-Force Red, we wanted to take advantage of the REST API functionality available in the most common SCM systems seen during engagements and add the most useful functionality in a proof-of-concept tool called SCMKit. The goal of this tool is to provide awareness of the abuse of SCM systems, and to encourage the detection of attack techniques against SCM systems. SCMKit will be shown at Black Hat USA 2022 Arsenal.

SCMKit allows the user to specify the SCM system and attack module to use, along with specifying valid credentials (username/password or API key) to the respective SCM system. Currently, the SCM systems that SCMKit supports are GitHub Enterprise, GitLab Enterprise and Bitbucket Server. The attack modules supported include reconnaissance, privilege escalation and persistence. Other functionality that are available in the non-public version of SCMKit were not included in consideration for defenders, such as user impersonation and built-in credential searching. SCMKit was built in a modular approach so that new modules and SCM systems can be added in the future by the information security community. The tool and full documentation are available on the X-Force Red GitHub. A few examples will be shown in the next sections.

Reconnaissance

SCMKit has multiple modules available to perform reconnaissance of repositories, files, code, and other resources specific to various SCM systems such as GitLab Runners. The below example shows using the “codesearch” module in SCMKit. In this scenario, we are searching for any code in Bitbucket Server that contains “API_KEY” to try and discover API key secrets within source code.

Code search example for API key with SCMKit

There are several other reconnaissance modules that apply only to certain SCM systems. For example, there is a reconnaissance module to discover GitLab Runners that you have access to via the “runnerlist” module.

GitLab Runner reconnaissance example with SCMKit

Privilege Escalation

Another capability available in SCMKit is to add another regular user to the admin role. The example, below, shows adding a regular user under our control (hgranger, in this case) to the site admin role in GitHub Enterprise via the “addadmin” module.

Adding site admin example via SCMKit

You can see the change that took effect in GitHub Enterprise after performing the site admin addition via SCMKit, as the hgranger user is now a member of the site admins group.

Showing hgranger added as site admin

Persistence

There are two persistence modules within SCMKit that include the use of personal access tokens or SSH keys. This can be useful to maintain access to an SCM system. The below example shows creating an SSH key for the hgranger user account in Bitbucket via the “createsshkey” module.

Creating SSH key example with SCMKit

We can list all active SSH keys for a given user via the “listsshkey” module, as shown below. You will see the SSH key we added for the hgranger user listed.

Listing SSH keys example with SCMKit

Defensive Considerations

SCMKit

There are multiple static signatures that can be used to detect the usage of SCMKit. These can be found in the Yara rule on the SCMKit repository.

A static user agent string is used when attempting each module in SCMKit. The user agent string is “SCMKIT-5dc493ada400c79dd318abbe770dac7c”. A Snort rule is provided on the SCMKit repository.

Additionally, any access tokens or SSH keys that are created in SCM systems using SCMKit will be prepended with “SCMKit-” in the name. This can be filtered in the respective SCM system to indicate an access token or SSH key was created using SCMKit.

SCM Systems

Ensure that the below logs are being sent to your SIEM. This also lists the location of the logs on the server for each respective SCM system.

Logs of interest – GitHub Enterprise

Logs of interest – GitLab Enterprise

Logs of interest – Bitbucket Server

Below are the various filters you can apply to the logs to detect the attacks demonstrated in this research. Use these filters to build a baseline and detect anomalous activity in your environment.

Table of search queries for various attack types – GitHub Enterprise

Table of search queries for various attack types – GitLab Enterprise

Table of search queries for various attack types – Bitbucket Server

Additionally, the below items should be considered when configuring these SCM systems:

• Disable user impersonation
• Do not allow users to create personal access tokens or SSH keys with no expiration date
• Set automatic expiration date on all personal access tokens and SSH keys created/added
• Limit the number of administrative users. At minimum there should be two, and should not be more unless necessary
• Operate on a policy of least privilege in terms of access to repositories
• Require signed commits via GPG keys or S/MIME certificates
• Enable multi-factor authentication
• Ensure that code branches are deleted in a timely manner
• Require at least one approver for each code commit
• Increase logging level to detect reconnaissance where applicable

Conclusion

Source code management systems contain some of the most sensitive information in organizations and are a key component in the DevOps lifecycle. Depending on the role of an organization, compromise of these systems can lead to the compromise of other organizations. These systems are a high value to an attacker, and need more visibility from the information security community, as they are currently an afterthought compared to other systems such as Active Directory. It is X-Force Red’s goal that this research will bring more attention and inspire future research on defending these critical enterprise systems.

Acknowledgments

A special thank you to the below people for giving feedback on this research and providing whitepaper content review.

• Chris Thompson (@retBandit)
• Daniel Crowley (@dan_crowley)
• Dimitry Snezhkov (@Op_nomad)
• Patrick Fussell (@capt_red_beardz)
• Ruben Boonen (@FuzzySec)