Tenable Research has disclosed multiple vulnerabilities in OpenAI’s ChatGPT that could allow attackers to steal private information from users’ memories and chat history without their knowledge. The vulnerabilities, discovered through months of investigation and responsibly disclosed in March 2024, have since been fully patched by OpenAI.
The vulnerabilities center around prompt injection, a known weakness in large language models (LLMs) where attackers can manipulate AI behavior by hiding malicious instructions in data the system processes. Tenable’s findings demonstrated how this theoretical risk could be weaponized for large-scale data theft.
The research revealed that attackers could exploit users through seemingly innocent actions, such as asking ChatGPT to summarize a blog post or simply asking a question that triggers a web search. In some cases, no user action was required at all beyond normal ChatGPT usage.
Among the seven discovered vulnerabilities, Tenable highlighted several novel techniques:
- 0-Click Attacks: Malicious websites indexed by search engines could inject prompts when users ask ChatGPT innocent questions. For example, asking about a niche topic could trigger a search that finds an attacker-controlled site, executing hidden instructions without any user action.
- Comment Section Exploits: Attackers could hide malicious prompts in blog comment sections. When users ask ChatGPT to summarize the article, the AI would process the comments and execute the hidden instructions.
- Memory Persistence: Perhaps most concerning, attackers could manipulate ChatGPT’s memory feature to create lasting compromises that continue leaking data across future sessions, even after the initial attack.
- Tenable also discovered a vulnerability in ChatGPT’s safety mechanisms, specifically the url_safe endpoint designed to block suspicious links. By exploiting whitelisted domains like bing.com, attackers could bypass these protections entirely.
The research revealed that when ChatGPT needs current information, it delegates browsing tasks to an alternative LLM called SearchGPT, which has fewer security protections. While SearchGPT lacks access to user memories as an isolation measure, Tenable developed techniques to breach this barrier through “Conversation Injection” essentially causing ChatGPT to prompt-inject itself by processing SearchGPT’s manipulated responses.
Real-World Attack Scenarios
Tenable demonstrated proof-of-concept attacks showing how the vulnerabilities could enable:
- Executive impersonation for business email compromise schemes
- Malware delivery through trusted-appearing ChatGPT responses
- Credential harvesting and financial fraud
- Persistent surveillance across all future ChatGPT interactions
- Misinformation campaigns exploiting false message histories
With hundreds of millions of users interacting with ChatGPT weekly, the potential attack surface was enormous.
Tenable responsibly disclosed all findings last year and all reported vulnerabilities had been resolved. OpenAI confirmed the fixes address issues that affected how certain search queries were routed without diving into specifics about the scope of affected users or elaborate on their implementation details.
Microsoft had previously disclosed CVE-2024-38197 as a related medium-severity spoofing issue. Tenable’s research expanded on those findings by demonstrating more severe exploitation paths.
The research underscores fundamental challenges in securing AI systems that interact with external data sources. Prompt injection remains an unsolved architectural problem with LLMs, and according to Tenable, systematic fixes are unlikely in the near future. Also notes is AI vendors’ reliance on SEO scores, which aren’t security boundaries, for sources, which highlight the broader issue of AI systems trusting external content without adequate validation.
For detailed technical analysis, proof-of-concept demonstrations, and vulnerability breakdowns, visit Tenable’s research post here.
Leave a Reply