NIST researchers warn of top AI security threats

As dozens of states race to establish standards for how their agencies use AI to increase efficiency and streamline public-facing services, researchers at the National Institute of Standards and Technology found that artificial intelligence systems, which rely on large amounts of data to perform tasks, can malfunction when exposed to untrustworthy data, according a report published last week.

The report, part of a broader effort by the institute to support the development of trustworthy AI, found that cyber criminals can deliberately confuse or “poison” AI systems to make them malfunction by exposing them to bad data. And what’s more, according to the study, there’s no one-size-fits-all defense that developers or cybersecurity experts can implement to protect AI systems.

“Data is incredibly important for machine learning,” NIST computer scientist Apostol Vassilev, one of the publication’s authors, told StateScoop. “‘Garbage in, garbage out’ is a well known kind of catchphrase in the trade.”

To perform tasks like autonomously driving vehicles or interacting with customers as online chatbots, AI is trained on vast quantities of data, which help the technology predict how best to respond in a variety of situations. Autonomous vehicles, for example, are trained on images of highways and streets with road signs, among other datasets. A chatbot might be exposed to records of online conversations.

Researchers warned that some AI training data — such as websites with inaccurate information or undesirable interactions with the public — may not be trustworthy and could cause AI systems to perform in an unintended manner. Chatbots, for example, might learn to respond with abusive or racist language when their guardrails get circumvented by carefully crafted malicious prompts. 

Joseph Thacker, a principal AI engineer and security researcher at AppOmni, security management software used by state and local governments, said it’s important to consider the security protocols needed to safeguard against every potential attack — like the ones outlined in NIST’s report.

“We’re gonna need everyone’s help to secure it,” Thacker told StateScoop. “And I think people should be thinking that through.”

‘Malicious intent’

The NIST report outlined four types of attacks on AI — poisoning, evasion, privacy and abuse — and classified them based on criteria such as the attacker’s goals and objectives, capabilities and system knowledge.

Poisoning occurs when an AI system is trained on corrupted data, such as by slipping numerous instances of inappropriate language into conversation records so that a chatbot interprets those instances as a common enough occurrence to use in its own customer interactions.

“Using a generative AI example, if you have a malicious intent and try to modify some of this input data that is fed into the model during training, where the model learns how to classify what is a cat, what is a dog and all these things, it can actually learn perturbations that could cause the model to misclassify, ” explained Apostol Vassilev, one of the NIST computer scientists who wrote the report.

But Thacker, who specializes in application security, hacking and AI, argued that while data poisoning is possible, its window is limited to the tool’s training phase and the other types of attacks — evasion, privacy and abuse in the form of prompt injections — are therefore more likely.

“If you can evade the filter, then that is an attack on the system, because you’re bypassing the set protection,” Thacker said of prompt injections, when bad actors trick the system into voluntarily offering someone else’s data.

Thacker said prompt injection attacks aim to force a chatbot to provide sensitive training data it’s programmed to withhold.

“If you’re able to extract data directly out of the model that went into the training of it — and a lot of times it’s trained on all the data on the internet, which will often contain a lot of people’s private information,” Thacker said. “ If you’re able to get the large language model to then output that sensitive information, it violates the privacy of that person.”

So what can be done?

Vassilev said a top challenge for state and local governments is incorporating large language models into their workflows securely. And while there are ways to mitigate attacks against AI, he cautioned agencies not to fall into a false sense of security, because there’s no foolproof method of protecting AI from misdirection.

“You can’t just say ‘Okay, I got this model and apply this technique and I’m done.’ What you need to do is continue to monitor, assess and react when problems occur,” said Vassilev, who also acknowledged that researchers should also develop better cybersecurity defenses. “In the meantime, you guys have to be alert and aware of all of these things. And monitor continuously.”

Thacker, who helps tech companies find these kinds of vulnerabilities in their software, insisted there are some common-sense ways to protect against AI security threats, including prohibiting access to sensitive data.

“Don’t connect systems that have access to sensitive data, like Social Security numbers or other personal information,” Thacker said. “If a government agency wants to enable its employees to work more efficiently through the use of AI, like ChatGPT or a similar service, don’t put in [training] data that’s sensitive. And don’t hook that up to a system which allows access to that data either.”

But Thacker also sounded a note of optimism, predicting that AI’s security features will become more common, similar to the ubiquity of two-factor authentication.

“A lot of people don’t realize everything that’s beneath the waters when they kind of are using a website or using a [software-as-a-service] application” he said. “I think that AI security is going to be integrated through the tech stack of your traditional security, and then your cloud security and then your SaaS security.”