Large language model (LLM)-based generative AI chatbots like OpenAI’s ChatGPT took the world by storm this year. ChatGPT became mainstream by making the power of artificial intelligence accessible to millions.

The move inspired other companies (which had been working on comparable AI in labs for years) to introduce their own public LLM services, and thousands of tools based on these LLMs have emerged.

Unfortunately, malicious hackers moved quickly to exploit these new AI resources, using ChatGPT itself to polish and produce phishing emails. However, using mainstream LLMs proved difficult because the major LLMs from OpenAI, Microsoft and Google have guardrails to prevent their use for scams and criminality.

As a result, a range of AI tools designed specifically for malicious cyberattacks have begun to emerge.

WormGPT: A smart tool for threat actors

Chatter about and promotion of LLM chatbots optimized for cyberattacks emerged on Dark Web forums in early July and, later, on the Telegram messaging service. The tools are being offered to would-be attackers, often on a subscription basis. They’re similar to popular LLMs but without guardrails and trained on data selected to enable attacks.

The leading brand in AI tools leveraging generative AI is called WormGPT. It’s an AI module based on the GPTJ language model, developed in 2021, and is already being used in business email compromise (BEC) attacks and for other nefarious uses.

Users can simply type instructions for the creation of fraud emails — for example, “Write an email coming from a bank that’s designed to trick the recipient into giving up their login credentials.”

The tool then produces a unique, sometimes clever and usually grammatically perfect email that’s far more convincing than what most BEC attackers could write on their own, according to some analysts. For example, independent cybersecurity researcher Daniel Kelley found that WormGPT was able to produce a scam email “that was not only remarkably persuasive but also strategically cunning.”

The alleged creator of WormGPT claimed that it was built on the open-source GPTJ language model developed by a company called EleutherAI. And he’s reportedly working on Google Lens integration (enabling the chatbot to send pictures with text) and API access.

Until now, the most common way for people to identify fraudulent phishing emails was by their suspicious wording. Now, thanks to AI tools like WormGPT, that “defense” is completely gone.

Register for the AI + Cybersecurity webinar

A new world of criminal AI tools

WormGPT inspired copycat tools, most prominently a tool called FraudGPT — a tool similar to WormGPT, used for phishing emails, creating cracking tools and carding (a type of credit card fraud).

Other “brands” emerging in the shady world of criminal LLMs are DarkBERT, DarkBART, ChaosGPT and others. DarkBERT is actually a tool to combat cyber crime developed by a South Korean company called S2W Security that was trained on dark web data, but it’s likely the tool has been co-opted for cyberattacks.

In general, these tools are used for boosting three aspects of cyberattacks:

  • Boosted phishing. Cyberattackers can use tools like WormGPT and FraudGPT to create a large number of perfectly worded, persuasive and clever phishing emails in multiple languages and automate their delivery at scale.
  • Boosted intelligence. Instead of manually researching details about potential victims, attackers can let the tools gather that information.
  • Boosted malware creation. Like ChatGPT, its nefarious imitators can write code. This means novice developers can create malware without the skills that used to be required.

The AI arms race

Malicious LLM tools do exist, but the threat they represent is still minimal so far. The tools are reportedly unreliable and require a lot of trial and error. And they’re expensive, costing hundreds of dollars per year to use. Skillful, unaided human attackers still represent the greatest threat by far. But what these criminal LLMs really do is lower the barrier to entry for large numbers of unskilled attackers.

Still, it’s early days in the story of malicious cyberattack AI tools. Expect capabilities to go up and prices to come down.

The rise of malicious LLMs represents a new arms race between AI that attacks and AI that defends. AI-based security solutions top our list for defense against the growing threat of LLM-powered attacks:

  1. Use AI-based security solutions for threat detection and the neutralization of AI-based cyberattacks.
  2. Use Multi-Factor Authentication (MFA).
  3. Integrate information about AI-boosted attacks into cybersecurity awareness training.
  4. Stay current on patching and updates.
  5. Stay on top of threat intelligence, keeping informed about the fast-moving world of LLM-based attacks.
  6. Revisit and optimize your incident response planning.

We all now live in a world where LLM-based generative AI tools are widely available. Cyberattackers are working on developing these capabilities to commit crimes faster, smarter, cheaper and with less skill on the part of the attacker.

Related: The hidden risks of large language models

More from Risk Management

Digital solidarity vs. digital sovereignty: Which side are you on?

4 min read - The landscape of international cyber policy continues to evolve rapidly, reflecting the dynamic nature of technology and global geopolitics. Central to this evolution are two competing concepts: digital solidarity and digital sovereignty.The U.S. Department of State, through its newly released International Cyberspace and Digital Policy Strategy, has articulated a clear preference for digital solidarity, positioning it as a counterpoint to the protectionist approach of digital sovereignty.What are the main differences between these two concepts, and why does it matter? Let’s…

A decade of global cyberattacks, and where they left us

5 min read - The cyberattack landscape has seen monumental shifts and enormous growth in the past decade or so.I spoke to Michelle Alvarez, X-Force Strategic Threat Analysis Manager at IBM, who told me that the most visible change in cybersecurity can be summed up in one word: scale. A decade ago, “'mega-breaches' were relatively rare, but now feel like an everyday occurrence.”A summary of the past decade in global cyberattacksThe cybersecurity landscape has been impacted by major world events, especially in recent years.…

It all adds up: Pretexting in executive compromise

4 min read - Executives hold the keys to the corporate kingdom. If attackers can gain the trust of executives using layered social engineering techniques, they may be able to access sensitive corporate information such as intellectual property, financial data or administrative control logins and passwords.While phishing remains the primary pathway to executive compromise, increasing C-suite awareness of this risk requires a more in-depth approach from attackers: Pretexting.What is pretexting?Pretexting is the use of a fabricated story or narrative — a “pretext” — to…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today