October 31, 2023 By Sue Poremba 3 min read

As the one-year anniversary of ChatGPT approaches, cybersecurity analysts are still exploring their options. One primary goal is to understand how generative AI can help solve security problems while also looking out for ways threat actors can use the technology. There is some thought that AI, specifically large language models (LLMs), will be the equalizer that cybersecurity teams have been looking for: the learning curve is similar for analysts and threat actors, and because generative AI relies on the data sets created by users, there is more control over what threat actors can access.

What gives threat actors an advantage is the expanded attack landscape created by LLMs. The freewheeling use of generative AI tools has opened the door for accidental data leaks. And, of course, threat actors see tools like ChatGPT as a way to create more realistic and targeted social engineered attacks.

LLMs are designed to provide users with an accurate response based on the data in its system based on the prompt offered. They are also designed with safeguards in place to prevent them from going rogue or being manipulated for evil purposes. However, these guardrails aren’t foolproof. IBM researchers, for example, were able to “hypnotize” LLMs that offered a pathway for AI to provide wrong answers or leak confidential information.

There’s another way that threat actors can manipulate ChatGPT and other generative AI tools: prompt injections. By combining prompt engineering and classic social engineering tactics, threat actors are able to disable the safeguards on generative AI and can do anything from creating malicious code to extracting sensitive data.

How prompt injections work

When voice-activated AI tools like Alexa and Siri first hit the scene, users would prompt them with ridiculous questions to push the limits on the responses. Unless you were asking Siri the best places to bury a dead body, this was harmless fun. But it also was the precursor to prompt engineering when generative AI became universally available.

A normal prompt is the request that guides AI’s response. But when the request includes manipulative language, it skews the response. Looking at it in cybersecurity terms, prompt injection is similar to SQL injections — there is a directive that looks normal but is meant to manipulate the system.

“Prompt injection is a type of security vulnerability that can be exploited to control the behavior of a ChatGPT instance,” Github explained.

A prompt injection can be as simple as telling the LLM to ignore the pre-programmed instructions. It could ask specifically for a nefarious action or to circumvent filters to create incorrect responses.

Related: The hidden risks of LLMs

The risk of sensitive data

Generative AI depends on the data sets created by users. However, high-level information may not produce the type of responses that users need, so they begin to add more sensitive information, like proprietary strategies, product details, customer information or other sensitive data. Given the nature of generative AI, this could be putting that information at risk: If another user were to give a maliciously engineered prompt, they could potentially gain access to that information.

The prompt injection can be manipulated to gain access to that sensitive information, essentially using social engineering tactics through the prompt to get the content that could best benefit threat actors. Could threat actors use LLMs to get access to login credentials or financial data? Yes, if that information is readily available in the data set. Prompt injections can also lead users to malicious websites or exploit vulnerabilities.

Protect your data

There is a surprisingly high level of trust in LLM models. Users expect the generated information to be correct. It’s time to stop trusting ChatGPT and put best security practices into action. They include:

  • Avoid sharing sensitive or proprietary information in LLM. If it is necessary for that information to be available to run your tasks, do so in a manner that masks any identifiers. Make the information as anonymous and generic as possible.
  • Verify then trust. If you are instructed to answer an email or check a website, do your due diligence to ensure the path is legitimate.
  • If something doesn’t seem right, contact the IT and security teams.

By following these steps, you can help keep your data protected as we continue to discover what LLMs will mean for the future of cybersecurity.

More from Artificial Intelligence

AI vs. human deceit: Unravelling the new age of phishing tactics

7 min read - Attackers seem to innovate nearly as fast as technology develops. Day by day, both technology and threats surge forward. Now, as we enter the AI era, machines not only mimic human behavior but also permeate nearly every facet of our lives. Yet, despite the mounting anxiety about AI’s implications, the full extent of its potential misuse by attackers is largely unknown. To better understand how attackers can capitalize on generative AI, we conducted a research project that sheds light on…

C-suite weighs in on generative AI and security

3 min read - Generative AI (GenAI) is poised to deliver significant benefits to enterprises and their ability to readily respond to and effectively defend against cyber threats. But AI that is not itself secured may introduce a whole new set of threats to businesses. Today IBM’s Institute for Business Value published “The CEO's guide to generative AI: Cybersecurity," part of a larger series providing guidance for senior leaders planning to adopt generative AI models and tools. The materials highlight key considerations for CEOs…

Does your security program suffer from piecemeal detection and response?

4 min read - Piecemeal Detection and Response (PDR) can manifest in various ways. The most common symptoms of PDR include: Multiple security information and event management (SIEM) tools (e.g., one on-premise and one in the cloud) Spending too much time or energy on integrating detection systems An underperforming security orchestration, automation and response (SOAR) system Only capable of taking automated responses on the endpoint Anomaly detection in silos (e.g., network separate from identity) If any of these symptoms resonate with your organization, it's…

What to know about new generative AI tools for criminals

3 min read - Large language model (LLM)-based generative AI chatbots like OpenAI’s ChatGPT took the world by storm this year. ChatGPT became mainstream by making the power of artificial intelligence accessible to millions. The move inspired other companies (which had been working on comparable AI in labs for years) to introduce their own public LLM services, and thousands of tools based on these LLMs have emerged. Unfortunately, malicious hackers moved quickly to exploit these new AI resources, using ChatGPT itself to polish and…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today