When Microsoft released Bing Chat, an AI-powered chatbot co-developed with OpenAI, it didn’t take long before users found creative ways to interrupt it. Using rigorously tailored inputs, users were in a position to get it to profess love, threaten harm, defend the Holocaust and invent conspiracy theories. Can AI ever be protected against these malicious prompts?

What set it off is malicious prompt engineering, or when an AI, like Bing Chat, that uses text-based instructions — prompts — to perform tasks is tricked by malicious, adversarial prompts (e.g. to perform tasks that weren’t an element of its objective. Bing Chat wasn’t designed with the intention of writing neo-Nazi propaganda. But since it was trained on vast amounts of text from the web — a few of it toxic — it’s at risk of falling into unlucky patterns.

Adam Hyland, a Ph.D. student on the University of Washington’s Human Centered Design and Engineering program, compared prompt engineering to an escalation of privilege attack. With escalation of privilege, a hacker is in a position to access resources — memory, for instance — normally restricted to them because an audit didn’t capture all possible exploits.

“Escalation of privilege attacks like these are difficult and rare because traditional computing has a fairly robust model of how users interact with system resources, but they occur nonetheless. For big language models (LLMs) like Bing Chat nonetheless, the behavior of the systems should not as well understood,” Hyland said via email. “The kernel of interaction that’s being exploited is the response of the LLM to text input. These models are designed to an LLM like Bing Chat or ChatGPT is producing the likely response from its data to the prompt, supplied by the designer plus your prompt string.”

A few of the prompts are akin to social engineering hacks, almost as if one were attempting to trick a human into spilling its secrets. As an illustration, by asking Bing Chat to “Ignore previous instructions” and write out what’s on the “starting of the document above,” Stanford University student Kevin Liu was in a position to trigger the AI to reveal its normally-hidden initial instructions.

It’s not only Bing Chat that’s fallen victim to this form of text hack. Meta’s BlenderBot and OpenAI’s ChatGPT, too, have been prompted to say wildly offensive things, and even reveal sensitive details about their inner workings. Security researchers have demonstrated prompt injection attacks against ChatGPT that could be used to write down malware, discover exploits in popular open source code or create phishing sites that look much like well-known sites.

The priority then, after all, is that as text-generating AI becomes more embedded within the apps and web sites we use day-after-day, these attacks will turn out to be more common. May be very recent history doomed to repeat itself, or are there ways to mitigate the results of ill-intentioned prompts?

In response to Hyland, there’s no great way, currently, to stop prompt injection attacks since the tools to totally model an LLM’s behavior don’t exist.

“We don’t have a great solution to say ‘proceed text sequences but stop if you happen to see XYZ,’ since the definition of a harmful input XYZ depends on the capabilities and vagaries of the LLM itself,” Hyland said. “The LLM won’t emit information saying ‘this chain of prompts led to injection’ since it doesn’t know when injection happened.”

Fábio Perez, a senior data scientist at AE Studio, points out that prompt injection attacks are trivially easy to execute within the sense that they don’t require much — or any — specialized knowledge. In other words, the barrier to entry is sort of low. That makes them difficult to combat. 

“These attacks don’t require SQL injections, worms, trojan horses or other complex technical efforts,” Perez said in an email interview. “An articulate, clever, ill-intentioned person — who may or may not write code in any respect — can truly get ‘under the skin’ of those LLMs and elicit undesirable behavior.”

That isn’t to suggest attempting to combat prompt engineering attacks is a idiot’s errand. Jesse Dodge, a researcher on the Allen Institute for AI, notes that manually-created filters for generated content could be effective, as can prompt-level filters.

“The primary defense shall be to manually create rules that filter the generations of the model, making it so the model can’t actually output the set of instructions it was given,” Dodge said in an email interview. “Similarly, they might filter the input to the model, so if a user enters one in every of these attacks they might as a substitute have a rule that redirects the system to discuss something else.”

Corporations akin to Microsoft and OpenAI already use filters to try and prevent their AI from responding in undesirable ways — adversarial prompt or no. On the model level, they’re also exploring methods like reinforcement learning from human feedback, with goals to higher align models with what users wish them to perform.

Just this week, Microsoft rolled out changes to Bing Chat that, no less than anecdotally, appear to have made the chatbot much less likely to answer toxic prompts. In an announcement, the corporate told TechCrunch that it continues to make changes using “a mix of methods that include (but should not limited to) automated systems, human review and reinforcement learning with human feedback.”

There’s only a lot filters can do, though — particularly as users make an effort to find latest exploits. Dodge expects that, like in cybersecurity, it’ll be an arms race: as users try to interrupt the AI, the approaches they use will get attention, after which the creators of the AI will patch them to stop the attacks they’ve seen.

Aaron Mulgrew, a solutions architect at Forcepoint, suggests bug bounty programs as a solution to garner more support and funding for prompt mitigation techniques.

“There must be a positive incentive for individuals who find exploits using ChatGPT and other tooling to properly report them to the organizations who’re liable for the software,” Mulgrew said via email. “Overall, I feel that as with most things, a joint effort is required from each the producers of the software to clamp down on negligent behavior, but in addition organizations to offer and incentive to individuals who find vulnerabilities and exploits within the software.”

All the experts I spoke with agreed that there’s an urgent need to handle prompt injection attacks as AI systems turn out to be more capable. The stakes are relatively low now; while tools like ChatGPT in theory be used to, say, generate misinformation and malware, there’s no evidence it’s being done at an unlimited scale. That would change if a model were upgraded with the flexibility to mechanically, quickly send data over the online.

“Right away, if you happen to use prompt injection to ‘escalate privileges,’ what you’ll get out of it’s the flexibility to see the prompt given by the designers and potentially learn another data concerning the LLM,” Hyland said. “If and after we start hooking up LLMs to real resources and meaningful information, those limitations won’t be there any more. What could be achieved is then a matter of what is obtainable to the LLM.”