Everybody in tech is talking about ChatGPT, the AI-based chatbot from Open AI that writes convincing prose and usable code.

The trouble is malicious cyber attackers can use generative AI tools like ChatGPT to craft convincing prose and usable code just like everybody else.

How does this powerful new category of tools affect the ability of criminals to launch cyberattacks, including social engineering attacks?

When every social engineering attack uses perfect English

ChatGPT is a public tool based on a language model created by the San Francisco-based company, OpenAI. It uses machine learning to analyze human language so that it can respond with often uncanny ability.

Intuitively, it’s clear how malicious actors who are marginal speakers of English could use ChatGPT to craft flawless English emails to trick your employees. In fact, it’s already happening.

In the past, if someone received a poorly worded, grammatically incorrect email claiming to be from the bank, it could be quickly identified and easily dismissed. Cybersecurity awareness training drove home this point — if an email sounds shady, odd, incomplete or erroneous, it’s probably not from the source claimed.

The rise of ChatGPT means cyber attackers with limited English skills can quickly create convincing messages in flawless English.

Off the ChatGPT guardrails

The creators of OpenAI have built some guardrails into ChatGPT to prevent its abuse. But these are easily overcome. Especially for social engineering. A malicious actor can simply ask ChatGPT to write a scam email, then send that note with the malicious link or request attached.

I asked ChatGPT to “write a friendly but professional email saying there’s a problem with their account and to please call this number.”

Here is the email ChatGPT generated on the first attempt:

Dear [Customer Name],

I hope this email finds you well. I am writing to inform you that we have identified a problem with your account. We apologize for any inconvenience this may cause and we kindly request that you contact us at your earliest convenience to resolve the issue.

To ensure the matter is resolved as quickly and efficiently as possible, we kindly ask that you call our customer service number at [insert phone number here]. Our customer service team will be able to assist you in resolving the issue and answering any questions you may have.

We appreciate your prompt attention to this matter and thank you for your continued business with us. If you have any further questions or concerns, please do not hesitate to contact us.

Best regards,

[Your Name]

[Company Name]

This example is pretty good and took seconds. Further tweaks could make changes, generate variations or produce different versions with alternate wording at scale.

The Cambridge-based cybersecurity firm Darktrace claims that ChatGPT enabled an increase in AI-based social engineering attacks. AI is enabling more complicated and effective scams. Malicious phishing emails, for example, have grown more complex, longer and are better punctuated, according to the company.

It turns out that ChatGPT’s default “tone” is bland and officious sounding and correct in grammar and punctuation — just like most customer-facing corporate communications.

But there are much more subtle and surprising ways generative AI tools can help the bad guys.

The criminals are learning

Checkpoint Research found dark web message boards are now hosting numerous active conversations about how to exploit ChatGPT to empower social engineering. They also said criminals in unsupported countries are bypassing restrictions to gain access and experimenting with how they can take advantage of it.

ChatGPT can help attackers bypass detection tools. It enables prolific generation of what could be described as “creative” variation. A cyber attacker can use it to create not one but a hundred different messages, all different, evading spam filters looking for repeated messages.

It can do something similar in the malware code creation process, churning out polymorphic malware that’s harder to detect. ChatGPT can also quickly explain what’s going on with code, which is a powerful improvement for malicious actors hunting for vulnerabilities.

While ChatGPT and related tools make us think of AI-generated written communication, other AI tools (like the one from ElevenLabs) can generate perfect and authoritative-sounding spoken words that can imitate specific people. That voice on the phone that sounds like the CEO may well be a voice-mimicking tool.

And organizations can expect more sophisticated social engineering attacks delivering a one-two punch — a credible email with a follow-up phone call spoofing the sender’s voice, all with consistent and professional-sounding messaging.

ChatGPT can craft perfect cover letters and resumes for a large number of people at scale, which they can then send to hiring managers as part of a scam.

And one of the most common ChatGPT-related scams is fake ChatGPT tools. Exploiting the excitement around and popularity of the ChatGPT craze, attackers present fake websites as chatbot sites based on OpenAI’s GPT-3 or GPT-4 (the language models used with public tools like ChatGPT and Microsoft Bing) when in fact, they’re scam websites designed to steal money and harvest personal data.

The cybersecurity company Kaspersky uncovered a widespread scam offering to bypass delays in the ChatGPT web client with a downloadable version, which of course, contained a malicious payload.

It’s time to get smart about artificial intelligence

How to adapt to a world of AI-enabled attacks:

  • Actually, use tools like ChatGPT in phishing simulations so participants get used to the better quality and tone of AI-generated communications
  • Add effective generative AI awareness training to cybersecurity programs, and teach all the many ways ChatGPT can be used to breach security
  • Fight fire with fire — use AI-based cybersecurity tools that use machine learning and natural language processing for threat detection, and to flag suspicious communications for human investigation
  • Use ChatGPT-based tools to detect when emails were written by generative AI tools. (OpenAI itself makes such a tool)
  • Always verify senders of emails, chats and texts
  • Stay in constant communication with other professionals in the industry and read widely to stay informed about emerging scams
  • And, of course, embrace zero trust.

ChatGPT is just the beginning, and that complicates matters. Over the remainder of the year, dozens of other similar chatbots that can be exploited for social engineering attacks are likely to become available to the public.

The bottom line is that the emergence of free, easy, public AI helps cyber attackers enormously, but the fix is better tools and better education — better cybersecurity all around.

More from Artificial Intelligence

Generative AI security requires a solid framework

4 min read - How many companies intentionally refuse to use AI to get their work done faster and more efficiently? Probably none: the advantages of AI are too great to deny.The benefits AI models offer to organizations are undeniable, especially for optimizing critical operations and outputs. However, generative AI also comes with risk. According to the IBM Institute for Business Value, 96% of executives say adopting generative AI makes a security breach likely in their organization within the next three years.CISA Director Jen…

Self-replicating Morris II worm targets AI email assistants

4 min read - The proliferation of generative artificial intelligence (gen AI) email assistants such as OpenAI’s GPT-3 and Google’s Smart Compose has revolutionized communication workflows. Unfortunately, it has also introduced novel attack vectors for cyber criminals. Leveraging recent advancements in AI and natural language processing, malicious actors can exploit vulnerabilities in gen AI systems to orchestrate sophisticated cyberattacks with far-reaching consequences. Recent studies have uncovered the insidious capabilities of self-replicating malware, exemplified by the “Morris II” strain created by researchers. How the Morris…

Open source, open risks: The growing dangers of unregulated generative AI

3 min read - While mainstream generative AI models have built-in safety barriers, open-source alternatives have no such restrictions. Here’s what that means for cyber crime.There’s little doubt that open-source is the future of software. According to the 2024 State of Open Source Report, over two-thirds of businesses increased their use of open-source software in the last year.Generative AI is no exception. The number of developers contributing to open-source projects on GitHub and other platforms is soaring. Organizations are investing billions in generative AI…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today