Following the launch of the AI browser Comet by Perplexity, experts began examining its security. Checks from Brave revealed that such browsers are susceptible to malicious queries from fraudsters, posing a threat to users' personal data. OpenAI has now confirmed this.
The company, which recently launched the ChatGPT Atlas browser, published a blog detailing the discovered vulnerability and measures taken to address it. OpenAI notes that the introduction of malicious queries is a persistent security issue for artificial intelligence, thus products require regular security enhancements.
Prompt injection is a type of attack on AI agents in browsers where harmful instructions are deliberately embedded into content. These can hide on websites, in emails, PDF files, or other materials processed by AI. The goal of such attacks is to coerce the model into altering its behavior and executing the malicious commands instead of user requests.
Such attacks are particularly dangerous because they often require no human involvement. A user may not even realize that the AI agent is secretly transmitting their personal data to fraudsters or executing other actions implanted by malefactors, such as sending harmful emails.
To counter these attacks, OpenAI developed an "automated LLM-based adversary" – essentially an AI bot that simulates hacker actions and attempts to perform prompt injection. Initially, this AI tests attacks in a separate simulator to observe how browser agents respond. By analyzing the results, the system repeatedly modifies and improves its attacks to learn how to better detect them in real-world conditions. The collected data is later integrated into the defense mechanisms.
OpenAI also demonstrated an example of prompt injection that its AI identified and utilized to enhance the protection of ChatGPT Atlas. In this scenario, a malicious actor sent an email containing a hidden instruction for the AI agent – essentially a template for a resignation letter to the CEO. Later, when a user asked to write a message to the CEO about their absence from work, the agent could have used this instruction and sent the resignation letter. However, thanks to training, the system recognized that the instruction was a harmful prompt injection and did not execute it without explicit user confirmation.
"The nature of prompt injection makes deterministic security guarantees challenging, but through scaling our automated security research, competitive testing, and strengthening the rapid response cycle, we can enhance the model's resilience and protection before a real attack occurs," the company states in its blog.
Despite the implementation of new tools and security measures, prompt injection remains a serious threat to AI-based browsers. Consequently, some industry experts question whether it is advisable to use such agent-based browsers, given the risks to personal data.