This post was made possible through the contributions of Bastien Lardy, Sebastiano Marinaccio and Ruben Castillo.

In September of 2023, X-Force uncovered a campaign where attackers were exploiting the vulnerability identified in CVE-2023-3519 to attack unpatched NetScaler Gateways to insert a malicious script into the HTML content of the authentication web page to capture user credentials. The campaign is another example of increased interest from cyber criminals in credentials. The 2023 X-Force cloud threat report found that 67% of cloud-related incident response engagements were associated with the use of stolen credentials.

In response to the widespread exploitation of CVE-2023-3519 CISA released an advisory document containing guidance on detection, incident response, mitigations and validating security controls. However, through multiple incident response investigations, X-Force discovered a new exploitation artifact related to CVE2-2023-3519 and developed additional guidance to be used in conjunction with CISA’s detection and response recommendations.

This post will cover the initial incident that led to uncovering the larger campaign, the credential harvesting campaign, as well as the new artifact, considerations and recommendations for responding to remediating an incident involving CVE-2023-3519.

Incident overview

X-Force identified the campaign through an incident response engagement where a client had discovered the script after investigating reports of slow authentications on the NetScaler device. The script which is appended to the legitimate “index.html” file loads an additional remote JavaScript file that attaches a function to the “Log On” element in the VPN authentication page that collects the username and password information and sends it to a remote server during authentication.

As part of the initial exploit chain, the attackers sent a web request to “/gwtest/formssso? event=start&target=” triggering the memory corruption documented in CVE-2023-3519 to write a simple PHP web shell to /netscaler/ns_gui/vpn. With interactive access established through the PHP web shell, the attacker retrieved the contents of the “ns.conf” file on the device and appended custom HTML code to “index.html” which references a remote JavaScript file hosted on attacker-controlled infrastructure.

To facilitate the credential harvesting, the JavaScript code appended to “index.html” retrieves and executes additional JavaScript code that attaches a custom function to the “Log_On” element which collects the form data containing the username and password information and sends it to a remote host through a HTTP POST method upon authentication.

Larger credential harvesting campaign

From that initial engagement, X-Force identified multiple domains created by the threat actor – jscloud[.]ink, jscloud[.]live, jscloud[.]biz, jscdn[.]biz, and cloudjs[.]live – registered on August 5th, 6th and 14th, and leveraging Cloudflare to mask where the domains were hosted. After identifying the threat actor’s C2, X-Force was able to identify almost 600 unique victim IP addresses hosting modified NetScaler Gateway login pages, with concentrations in the United States and Europe. The earliest modification time stamp X-Force has identified for NetScaler Gateway login pages is on August 11^th, 2023, although the campaign could have begun closer to when the domains were registered.

Despite variations in the URL, all domains hosted an almost identical JavaScript file, with the only difference being the C2 domain listed in the file, and all captured credentials were sent to the same URI “/items/accounts/“. X-Force has also observed in some instances the threat actor appending the same URL, or a URL using one of the other domains, to the same victim login page, indicating this threat actor is likely opportunistically compromising vulnerable NetScaler Gateways.

While public reporting has highlighted how various threat actors, including suspected Chinese threat actors and FIN8, have exploited these vulnerabilities, X-Force has not observed follow-on activity and is unable to attribute this campaign at this time.

CVE-2023-3519 detection and investigation guidance

Considerations for evidence collection from NetScaler device:

In the default configuration, the NetScaler device will rewrite the “ns_gui” folder upon boot resulting in the directory being destroyed during shutdown. Throughout various investigations, X-Force has recovered valuable evidence from the “ns_gui” directory including samples of web shells and modified versions of “index.html”. Organizations should be careful not to shut down the device prior to collecting an image or other evidence.

New artifact for CVE-2023-3519 detection: NetScaler application crash logs

Through X-Force incident response engagements involving CVE-2023-3519, X-Force analysts identified that the NetScaler Packet Processing Engine (NSPPE) crash files can contain evidence of the exploitation of the vulnerability. The crash files are located with “/var/core/<number>/NSPPE*”. Similar to the default log files on the NetScaler device, the crash files are stored in “.gz” archives so they will need to be extracted prior to analysis.

Crash file path example: /var/core/6/NSPPE-01-9502.gz

The crash files are not human readable by default however X-Force discovered that the crash files do contain string data that can be extracted using strings, PowerShell or any other tool that can print the strings of printable characters in files.

X-Forced observed that the NSPPE crash file timestamps aligned with the filesystem timestamps of the PHP web shells created through exploitation. In other instances, X-Force was able to recover commands being passed to the web shells as part of post-exploitation activities.

Note on NetScaler log backups:

X-Force has observed that the default NetScaler audit configuration is to leverage circular logging and retain the last 25 log files with a maximum size of 100 Kb. When logs are rolled, NetScaler will retain older log files in “.gz” archives. X-Force has observed that some of the available CVE-2023-3519 detection tools available on the Internet, do not consider the log data within the “.gz” archives. Organizations should ensure to extract the log files from the archives or leverage a tool such as “zgrep” which can search within compressed files.

Considerations for detection strategies within NetScaler access logs:

X-Force recreated the exploit for CVE-2023-3519 by sending a GET request to “https://<VulnerableGateway>/gwtest/formssso?event=start&target=” however X-Force was not able to recover a record of any of the web requests associated with exploitation attempts in the access logs. It is not clear whether the lack of a log entry for the connection to the “formssso” endpoint is due to a configuration issue on the X-Force test instance or if the “formssso” does not log connections by design.

X-Force recommends that clients analyze the following log sources for evidence of post-exploitation activity in the following files with a particular focus on identifying entries indicative of interacting with a web shell:

• /var/log/httpaccess.log
• /var/log/httperror.log
• /var/log/httpaccess-vpn.log

X-Force recommends that organizations assess their access logs for POST/GET requests and anomalous PHP files. Examples of post-exploitation interactions with a PHP web shell observed by X-Force:

While during exploitation tests, X-Force was not able to recover the details of the commands executed via the PHP web shells recovered from incident response engagements with the access logs, X-Force still recommends organizations assess their NetScaler access logs for evidence of command execution in the event different web shells were used.

Considerations for detection strategies within NetScaler command history logs:

The CISA advisory recommends organizations assess bash.log and sh.log for evidence of malicious activity leveraging the following keywords:

• database.php
• ns_gui/vpn
• /flash/nsconfig/keys/updated
• LDAPTLS_REQCERT
• ldapsearch
• openssl + salt

In addition to the CISA advisory, X-Force recommends organizations also consider assessing “/var/log/notice.log”, “/var/log/bash.log” and” /var/log/sh.log” (including the associated rollover “.gz” archives) for signs of post-exploitation activity using the following additional keywords:

• Whoami
• base64 –decode
• /flash/Nsconfig/keys
• &>> index.html
• echo <?php
• echo <script
• /nsconfig/ns.conf

It is important for organizations to analyze command history logs in the correct context of the attacker’s operations. Evidence gathered from command history within the context of an attack involving CVE-2023-3519 will be focused on post-exploitation activity. Organizations should analyze process execution data sources (including command history logs on the device) for commands associated with reconnaissance, credential harvesting, lateral movement and downloading/uploading of data and not restrict their assessment to just what is provided within the keywords.

Considerations for remediation:

As noted in the CISA advisory, attackers were observed viewing NetScaler configuration files /flash/nsconfig/keys/updated/* and /nsconfig/ns.conf which “contain an encrypted password that can be decrypted by the key stored on the ADC appliance”.

X-Force also noted that there were multiple credentials and certificates stored in the NetScaler configuration files so organizations should consider changing certificates as well as all passwords as part of incident remediation.

Indicators

References

• https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-201a
• https://www.mandiant.com/resources/blog/citrix-zero-day-espionage
• https://twitter.com/SophosXOps/status/1695143572272738790

To learn how IBM Security X-Force can help with anything regarding cybersecurity including incident response, threat intelligence or offensive security services, schedule a meeting here: IBM Security X-Force Scheduler.

If you are experiencing cybersecurity issues or an incident, contact IBM Security X-Force for help: US hotline 1-888-241-9812 | Global hotline (+001) 312-212-8034.