AI Security Breaches Unveiled: Critical Findings from Recent Study
A recent investigation has unveiled alarming security breaches among several AI providers, revealing significant vulnerabilities in the protection of user data. The key outcomes of the study include:
- Evidence indicating that at least three AI service providers were compromised by attackers who exploited their access to extract user prompts.
- Discovery of multiple resellers within the inference marketplace that logged user prompts and transmitted them to unauthorized individuals using leaked credentials.
- Instances where AI providers neglected to adhere to end-user privacy settings, including provisions for Zero Data Retention (ZDR) and “no data collection” policies.
In a detailed report, Moore, the Chief Research Officer at security firm Dragos, stated, “We sent nearly two million AI prompts containing unique, trackable secrets to AI providers, then watched where those secrets surfaced. What we found revealed security breaches, prompt theft, and widespread failures to protect customer data.”
Despite the encryption of prompts during transit, they are decrypted at various points within the extensive inference reseller network. Initially, a prompt is transmitted to an AI router, which decodes it to identify the most suitable inference providers for the request. Subsequently, the prompt may be directed to a range of available models, potentially hosted by different providers. This process allows prompts to be transferred between various AI routers, each of which may log the prompt and retain it according to their policies, sometimes indefinitely.
The Challenges of AI Security
The most alarming finding from the study was the breach affecting three distinct hosting providers. Attackers compromised one provider by reusing “a couple hundred” AWS keys. Moore elaborated:
“Each key was sent in exactly one request to one provider, all within a few short windows in late October, and every request we sent during those windows was stolen. Some of the stolen keys came from requests that the provider rejected before a model ever ran—this tells us that the attacker was copying requests at the provider’s front door, and not somewhere deeper in the inference stack. In the other two breaches, some of the stolen requests were marked as ZDR. A privacy flag doesn’t help when the attacker is already inside.”
Entities linked to the breach of these three providers tested each compromised key shortly after they were generated, exploring what resources were accessible. Investigators found that the attackers scoured S3 buckets, secrets managers, and parameter stores in search of additional credentials. They also attempted to activate hosted models in AWS Bedrock and added their own IAM users to maintain persistent access. Overall, the stolen keys were utilized “tens of thousands” of times in AWS API calls.
















Discussion about this post