March 27, 2020 By Mark Stone 4 min read

Technologies like artificial intelligence (AI) and neural networks are driven by deep learning — machine learning algorithms that get “smarter” with more data. The deepfake, a severe cybersecurity threat, wouldn’t be possible without deep learning.

Deepfakes aside, we need to be aware that several machine learning models, including state-of-the-art neural networks, are vulnerable to adversarial examples. The threat to the enterprise can be critical.

What is an adversarial example, and why should we care? These machine learning models, as intelligent and advanced as they are, misclassify examples (or inputs) that are only marginally different from what they normally classify correctly. For instance, by modifying an image ever so slightly — even altering just one pixel in some cases — image recognition software can be defeated.

With more companies relying on deep learning to process data than ever before, we need to be more aware of these types of attacks. The underlying strategies behind adversarial attacks are fascinating. Colorful toasters are even involved.

Adversarial Attacks 101

Luba Gloukhova, founding chair of Deep Learning World and editor-in-chief of the Machine Learning Times, finds herself at the hub of the machine learning and deep learning industries. I met Gloukhova, an independent speaker and consultant, at a tech conference in February, where she told me that the more she understands about the capabilities of deep learning, the more potential security risks we are exposing ourselves to become apparent.

“As I saw some of the potential shortcomings of this technology, it got me venturing down this path of adversarial attacks on adversarial examples, and it got me really interested in this industry,” Gloukhova said.

According to Gloukhova, an adversarial attack is one in which inputs to a deep learning neural network ultimately result in unexpected outputs. The example here is the input itself.

“The adversarial examples are generated by a slight modification to the input to a network which is trained to recognize it,” she explained. “The slight modification in one way or another creates completely unexpected results and unexpected outputs.”

This unexpected output exposes vulnerabilities in the architectures of deep neural networks because a bad actor can craft the adversarial examples to disguise malicious content as legitimate to the human eye. One fundamental analogy she often uses is that of a hacker trying to spread malicious code, where the code (input) could fly under the radar of most recognition systems.

“This adversarial attack could contain malicious content that’s hidden because of slight modifications that make it look as if it’s not problematic,” she said.

Toaster? Banana? Other?

Back to that colorful toaster. Back in 2017, at the 31st Conference on Neural Information Processing Systems in Long Beach, a group of data scientists presented evidence of their toaster theory. By placing a sticker of a toaster nearby or on an object (input) for image recognition software, the software (or neural network) would classify it as a toaster. In their experiment, a banana was used as the input.

On its own, it gets classified as a banana with high accuracy. But as soon as we add the toaster sticker, the network thinks it’s a toaster. “It misleads the classification model, and what’s interesting is that the authors of the paper have shown that it is successful in attacking models that have completely different architectures and were trained with completely different differences,” said Gloukhova.

What’s most concerning about these types of adversarial attacks is that the bad actor doesn’t need any information about the model it’s attacking. Fortunately, we’re not highly reliant on this technology at this point.

Threats Well Beyond Image Recognition

Still, Gloukhova warns that the computer security threat is significant, and she doesn’t want to paint a picture that only image recognition systems are vulnerable. In fact, a research paper from Cornell University demonstrates how essentially the same adversarial attacks could be applied to natural language processing systems based on deep neural networks. In the experiment, they were able to take any audio waveform and produce another that is over 99.9 percent similar and transcribe it as any phrase of their choosing.

As our reliance on autonomous vehicles grows, deep neural networks will be a critical component of the transportation ecosystem. This network, not unlike other deep learning networks, is susceptible to adversarial examples. According to Gloukhova, a serious threat has been outlined in research papers that demonstrate how by simply placing some inconspicuous spray paint or a particular sticker on a stop sign, the vehicle may fail to classify it.

“The deep neural network that resides within that vehicle’s recognition system could make it misinterpret the actual stop sign as, say, a yield sign or a speed limit sign,” she warns. “So my goal is to communicate that these adversarial examples do exist. They are a vulnerability of these deep neural networks. We need to be aware of the limits of this technology, especially as our fascination with it and our reliance on it grows.”

We are likely a long way away from this scenario playing out at scale in the real world, but planning to counter these threats now can better position us for safety.

How the Enterprise Can Respond

So what does all this mean for the enterprise? On the surface, the issue can seem too complex to get our minds around, but some of the strategies Gloukhova promotes revolve more around mindset and culture than technology. Of course, getting into the developmental and architectural weeds will still be necessary.

“By leveraging these complex architectures and huge training data sets, we’ve chosen to opt for models and model architectures that make them open to adversarial attacks,” she said. “A big takeaway for this is the overall awareness of this vulnerability.”

She hopes that developers and executives will be more cautious about the kinds of activation functions and neural network architectures they choose, which may come down to using smaller training data sets or adopting less fancy architectures.

Gloukhova recommended developer tactics like non-linear activation functions, adversarial training, JPEG compression, defensive distillation and gradient masking. However, these technical solutions are not so simple, and they require a lot more computing power and some sacrifices.

One of the top priorities for any organization should be to hire the right people for deep learning. “It does take the kind of people that are aware of these threats and are aware of the countermeasures,” Gloukhova said. “I’m excited to spread the word, especially toward the C-Suite. They need to see the value in this kind of knowledge so it can trickle down through the organization to the developers, and maybe it will create some lasting change.”

More from Artificial Intelligence

Generative AI security requires a solid framework

4 min read - How many companies intentionally refuse to use AI to get their work done faster and more efficiently? Probably none: the advantages of AI are too great to deny.The benefits AI models offer to organizations are undeniable, especially for optimizing critical operations and outputs. However, generative AI also comes with risk. According to the IBM Institute for Business Value, 96% of executives say adopting generative AI makes a security breach likely in their organization within the next three years.CISA Director Jen…

Self-replicating Morris II worm targets AI email assistants

4 min read - The proliferation of generative artificial intelligence (gen AI) email assistants such as OpenAI’s GPT-3 and Google’s Smart Compose has revolutionized communication workflows. Unfortunately, it has also introduced novel attack vectors for cyber criminals. Leveraging recent advancements in AI and natural language processing, malicious actors can exploit vulnerabilities in gen AI systems to orchestrate sophisticated cyberattacks with far-reaching consequences. Recent studies have uncovered the insidious capabilities of self-replicating malware, exemplified by the “Morris II” strain created by researchers. How the Morris…

Open source, open risks: The growing dangers of unregulated generative AI

3 min read - While mainstream generative AI models have built-in safety barriers, open-source alternatives have no such restrictions. Here’s what that means for cyber crime.There’s little doubt that open-source is the future of software. According to the 2024 State of Open Source Report, over two-thirds of businesses increased their use of open-source software in the last year.Generative AI is no exception. The number of developers contributing to open-source projects on GitHub and other platforms is soaring. Organizations are investing billions in generative AI…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today