Skip to content
konsole
Threat intelligence

Threat report · Data leakage · Cost

Where AI usage leaks data and money: exposed infrastructure, shared chats, shadow AI, LLMjacking and runaway spend, 2024 to 2026

A sourced report on the non-exotic ways organisations lose data and budget through LLM use: databases left open, chat links indexed by Google, employees pasting source code into personal accounts, stolen model credentials billed to the victim, and agents that nobody capped. With the figures, the regulators’ responses and the controls that address each.

Konsole threat research 15 min read

THREAT INTELLIGENCETRACKINGData leakage and runaway spendChat histories and keys left on the open internetShared conversations indexed by search enginesStolen model credentials and six-figure loopsDATED · SOURCED · MAPPED TO CONTROLS
PromptsPASTED CUSTOMER DATA, SOURCE CODE, CONTRACTSShared linksCONVERSATIONS INDEXED BY SEARCH ENGINESExposed infraDATABASES AND DASHBOARDS LEFT OPENStolen keysMODEL CREDENTIALS RUN AT YOUR EXPENSERunaway spendLOOPS AND RETRIES NOBODY CAPPED
Five surfaces through which AI usage leaks. The first three lose data, the last two lose money, and the same gateway controls address all five.

Infrastructure left open

On 29 January 2025 Wiz Research found two DeepSeek hosts with a ClickHouse database reachable from the internet with no authentication. It allowed full SQL control and held a log table with more than a million entries of plaintext chat history, API secrets and backend details. DeepSeek closed it within hours of being told. The exposure fed directly into regulatory action: Italy’s data protection authority blocked the DeepSeek app the next day. [1][2]

The same pattern repeats at the layer of apps built on top of models. In January 2026 a misconfigured Firebase backend behind the Chat & Ask AI app allowed public reads of roughly 300 million messages from more than 25 million users; it was fixed within hours of the report. In August 2026 Cybernews found an unsecured Azure storage bucket belonging to the aggregator 302.ai holding 17.6 million files and 22.7 terabytes, including archives of conversation data and enterprise customer-service bot transcripts; it was closed in September. In November 2025 OpenAI disclosed that a breach at its analytics vendor Mixpanel exposed names, email addresses and approximate locations of API users, though no prompts, keys or payment data. [3][4][5]

Self-hosted model servers are now a category of their own. A SentinelOne and Censys scan in January 2026 found about 175,000 Ollama hosts answering on the public internet; LeakIX counted 12,269 with no authentication the following month; Intruder found 31% of the Ollama APIs it queried would run a prompt for anyone. In May 2026 Cyera disclosed CVE-2026-7482 in Ollama, estimating around 300,000 vulnerable instances. Anyone can run inference on the owner’s hardware, pull or register models, and in some configurations read memory. Google’s threat intelligence group confirmed in November 2026 that attackers scan for exposed Ollama endpoints and enrol them into resold compute pools. [6][7][8][9]

Share buttons that published private conversations

In July 2025 Fast Company reported that ChatGPT conversations shared with a “make this chat discoverable” option had been indexed by Google. About 4,500 were found by that method; 404 Media later assembled a dataset of around 100,000. The content included mental-health, legal and workplace material. OpenAI removed the option within a day, called it a short-lived experiment and worked with search engines to de-index. [10][11]

Three weeks later Forbes found that Grok’s share button generated public URLs with no warning and that Google had indexed more than 370,000 of them, including names, at least one password, uploaded spreadsheets and instructions for harmful activities. In June 2025 the Meta AI app’s Discover feed had already shown users’ medical questions, relationship problems and home addresses to the public, posted through a share flow many did not understand was public; Meta added a warning. In none of these cases was anything hacked. The product made publishing easy and the consequences invisible. [12][13][14]

Shadow AI: what employees paste, and how often

The origin case is Samsung in April 2023, when engineers pasted internal source code and meeting notes into ChatGPT and the company banned generative AI on its devices. Three years of measurement since then show the behaviour has scaled with adoption rather than fallen away. [15][16]

MeasurementFigureSource and date
Enterprise GenAI use through personal accounts72% of use (2025); 47% of users, down from 78% (2026)Netskope Cloud and Threat Report, 2025 and Jan 2026 [17][18]
GenAI data-policy violationsDoubled in 2025; 223 per organisation per month on average, 2,100 in the top quartileNetskope, Jan 2026 [18][19]
Largest violation categorySource code, 46% of violationsNetskope, 2025 [17]
Prompts containing sensitive data8.5% (Q4 2024); 4.37% of prompts and 22% of uploaded files (Q2 2025); 2.6% of 22.5 million prompts and uploads (full-year 2025), 71% of exposures via ChatGPTHarmonic Security [20][21][22]
Employees who paste data into GenAI; share in unmanaged accounts77%; 82% of those pastes in personal accounts; 22% of pastes contain PII or payment dataLayerX, Oct 2025 [23][24]
AI interactions involving sensitive data34.8% (2025); 39.7% (2026, 222 companies)Cyberhaven Labs [25][26]
Organisations reporting a breach of an AI model or application; share lacking AI access controls13%; 97% of thoseIBM Cost of a Data Breach, Jul 2025 [27]
Extra breach cost where shadow AI was involvedUS$670,000 above the US$4.44 million average; shadow AI in 20% of breachesIBM, Jul 2025 [27][28]

Two cautions on these numbers. They come from security vendors measuring their own customers, so they describe organisations that already bought monitoring, and they define “sensitive” differently (Harmonic’s falling percentage partly reflects a larger denominator as routine use grew). The direction is consistent across all four vendors and IBM: the volume of corporate data entering models is rising, a large share goes through accounts the company cannot see, and the organisations that get breached are overwhelmingly the ones with no access control on AI use.

LLMjacking: your keys, their usage, your bill

Sysdig coined the term in May 2024 after watching attackers use cloud credentials stolen from a vulnerable web server to call Bedrock, OpenAI, Azure and Anthropic models through a reverse proxy, and modelled the worst-case cost at more than US$46,000 a day. By September 2024 they had observed more than 85,000 Bedrock requests from one campaign, 61,000 of them in a three-hour window, and raised the modelled ceiling to US$100,000 a day with the newest models; one victim reportedly owed about US$30,000 after three hours. A Permiso honeypot key hijacked for an AI chat service generated US$3,500 of charges in two days. [29][30][31][32]

The market matured fast. Within days of DeepSeek’s releases in December 2024 and January 2025 the proxies had added it; Sysdig found more than 55 misused DeepSeek keys and one proxy that ran up nearly US$50,000 in four and a half days, reselling access for US$30 a month. Microsoft’s Digital Crimes Unit sued the group it called Storm-2139 in December 2024 and named four developers in February 2025: they had scraped Azure OpenAI keys that customers leaked publicly, built tooling to strip the content controls and sold access used to generate non-consensual imagery. [33][34][35]

Google’s Threat Intelligence Group published the most complete picture on 2 November 2026. Fraudulent charges exceeded US$100,000 in a single day in individual incidents and reached about US$1 million at some organisations before keys were revoked. One token-relay service pushed nearly US$1 million of token costs onto a single victim and sold “unlimited” access for about US$60 a month. One actor made more than 500,000 API calls in under eight hours. The access vectors were unglamorous: misconfigured service-account keys, hijacked session tokens, stolen GitHub personal access tokens, keys scraped from public repositories and validated with one-token probe requests to stay under volume alerts, and the exposed Ollama endpoints described above. Google’s recommendations were spending caps, per-model key scoping, secrets scanning, phishing-resistant MFA, patching self-hosted gateways and alerting when a model service is enabled in an account that never used one. [9][36]

Where the keys come from

GitGuardian’s annual scan of public GitHub found 23.8 million new hard-coded secrets in 2024 and 28.6 million in 2025, the largest year-on-year jump it has recorded. Secrets for AI services rose 81% to 1.28 million, including 113,000 DeepSeek keys in the provider’s first year. The 2026 report also found 24,008 secrets inside MCP configuration files, 2,117 of them still valid, and that repositories using AI coding assistants were roughly twice as likely to leak a secret. Sixty-four percent of secrets that were valid in 2022 were still usable in January 2026. [37][38][39]

Keys also leak into the training data itself. Truffle Security scanned the December 2024 Common Crawl, 400 terabytes and 2.67 billion pages that feed many models including DeepSeek’s, and found 11,908 live secrets of 219 kinds, from AWS root keys to Slack webhooks. Lasso Security’s earlier finding of 1,681 valid Hugging Face and GitHub tokens, 655 with write access to models used by 723 organisations, shows the same problem on the model-hosting side. [40][41][42]

Runaway spend without an attacker

Not every oversized bill involves a thief. OWASP lists Unbounded Consumption, including “denial of wallet”, as LLM10 in its 2025 Top 10, and its agentic list treats uncapped autonomy as a first-class risk. The mechanics are described in Konsole’s own cost-control webinar: an agent calls the same tool repeatedly, pursues a task with no valid end state, or asks the model to reason about the same problem in a loop; a retry policy hammers an endpoint that is returning an error; a conversation re-sends its whole history on every turn so that turn ten costs many times turn one. [43][44]

The economics make the gap between a controlled and an uncontrolled call enormous. At November 2026 list prices a frontier model costs on the order of US$8 to US$10 per million input tokens and US$40 to US$50 per million output tokens, while a small model costs about ten cents and fifty cents. A key or an agent that can reach the frontier model when the task needed the small one is paying up to a hundred times more per token, which is why Google’s LLMjacking report and every cost-governance guide say the same thing: scope keys to a model, cap spend per key, and alert in real time rather than at invoice time. Well-sourced public post-mortems of accidental loops at large scale are rare, which is itself a finding: companies disclose breaches because they must, and disclose bills because they choose to. [45][46]

Smart routing20–60× CHEAPER MODELS FOR SIMPLE TASKSPrompt cachingUP TO 90% OFF REPEATED TOKENSBudgets + alertsPER USER · TEAM · APP · MODELContext discipline60–80% FEWER INPUT TOKENSCircuit breakersTOKEN CAPS · TOOL-CALL LIMITS · KILL SWITCH
The five cost levers from the webinar. Budgets and circuit breakers are the two that stop both LLMjacking and accidental loops.

What regulators have done so far

  • Italy, OpenAI, €15 million, December 2024. The Garante fined OpenAI for training ChatGPT without an adequate legal basis, transparency failures and no age verification. In March 2026 the Court of Rome annulled the decision on procedural grounds (Ireland had become the lead authority), without ruling on the merits. [47][48]
  • Italy, Luka (Replika), €5 million, May 2025; Character.AI, €158,000, July 2026. Both for consent and age-verification failures; Character.AI also for a late data-protection impact assessment. [49][50]
  • Italy, DeepSeek, January 2025. App blocked the day after the Wiz disclosure; still in force. Berlin’s authority reported the app to Apple and Google in June 2025; South Korea issued a corrective order. [51]
  • EU AI Act. Prohibited practices applied from February 2025 and general-purpose model obligations from August 2025. The Digital Omnibus on AI, in force 27 July 2026, moved standalone high-risk obligations from August 2026 to December 2027. [52][53]

Enforcement so far has targeted consent, transparency and children rather than the leakage mechanisms in this report. That will not hold. Chat logs are personal data, exposed chat logs are a breach, and the first regulator to treat an indexed share link or an open log database as a notifiable incident will set the precedent.

The controls that map to each failure

FailureControlWhere it lives
Sensitive data in prompts reaches a third-party modelDetect and mask PII, PHI and secrets before the request leaves your network; restore in the responseGateway, inline on every call (masking)
Employees use personal accounts the company cannot seeRoute sanctioned access through one endpoint with per-user tokens; log every call with its sourceGateway and identity
Stolen or leaked model keys billed to youKeys held in a vault, never in code; developers get short-lived tokens; per-key model scope and spend cap; secrets scanning on repositoriesGateway (platform) and CI
Agent loops and retry stormsToken budget per run, tool-call ceiling, loop detection, hard stop with a human-held kill switchGateway (cost control)
Model servers and databases left openBind to localhost, authenticate at a gateway, scan your own ranges for listening model portsNetwork and platform
Share features publish private contentDisable public sharing for workplace accounts; treat shared links as data exportsPolicy and tenant settings

Frequently asked questions

If the model provider promises not to train on our data, is pasting customer records still a leak?
Yes. A no-training promise says nothing about retention, logging, vendor sub-processors, legal holds, the provider’s own breaches, or the employee’s personal account that the promise does not cover. The DeepSeek, Mixpanel and wrapper-app incidents all leaked data that was never used for training. Masking before the request leaves your network is the only control that works regardless of what the provider does.
How do attackers find model API keys?
Mostly from places developers left them: public GitHub repositories (28.6 million new secrets in 2025 per GitGuardian), MCP configuration files, Docker images, web pages in Common Crawl, and cloud credentials taken from compromised servers. They validate keys with one-token requests to avoid triggering volume alerts, then resell access through proxies.
What is LLMjacking and how big are the losses?
Using stolen credentials to run models billed to the victim. Sysdig documented it from May 2024 with modelled costs of US$46,000 to US$100,000 a day. Google’s November 2026 report confirmed real incidents above US$100,000 in a day and around US$1 million per victim before revocation, with access resold for about US$60 a month.
How common is it for employees to put sensitive data into AI tools?
Measurements vary with definitions, but every vendor that measures it finds the same direction. Netskope counted 223 data-policy violations per organisation per month in 2025, double the year before. LayerX found 77% of employees paste data into GenAI tools and 82% of those pastes happen in personal accounts. Cyberhaven found about 40% of AI interactions involve sensitive data.
Which of these incidents led to regulatory penalties?
The DeepSeek exposure led directly to Italy blocking the app. The fines to date (OpenAI €15 million, later annulled on procedure; Replika €5 million; Character.AI €158,000) were about legal basis, transparency and age verification rather than leakage. The EU AI Act’s high-risk obligations were pushed to December 2027 in July 2026.

Key takeaways

  • Mask before the model. A no-training clause does not cover the eight other ways a prompt can leak.
  • Never let a developer hold a provider key. Vault it, scope it to a model, cap its spend, hand out short-lived tokens.
  • Count your model endpoints. Hundreds of thousands of self-hosted servers answer to anyone, and attackers scan for them.
  • Cap every agent run in tokens and tool calls, and alert in real time. The invoice is not a monitoring system.
  • Measure your own shadow AI. The organisations that got breached were the 97% with no AI access controls.

Sources

Numbers in square brackets in the text refer to this list. Links open the publisher’s page.

  1. [1]Wiz Research uncovers exposed DeepSeek database leaking sensitive information · Wiz · 29 Jan 2025
  2. [2]DeepSeek AI exposed databases with user chat history, API keys · BleepingComputer · Jan 2025
  3. [3]AI chat app leak exposes 300 million messages tied to 25 million users · Malwarebytes · Feb 2026
  4. [4]302.ai data leak exposed millions of files · Cybernews · Sep 2026
  5. [5]OpenAI API users’ data exposed in Mixpanel breach · The Register · 27 Nov 2025
  6. [6]Researchers find 175,000 publicly exposed Ollama servers · The Hacker News · Jan 2026
  7. [7]Ollama exposed · LeakIX · Feb 2026
  8. [8]Exposed AI infrastructure is rampant · Intruder · 2026
  9. [9]LLMjacking: monetizing compromised cloud models · Google Threat Intelligence Group / Mandiant · 2 Nov 2026
  10. [10]OpenAI is removing ChatGPT conversations from Google · Engadget · Aug 2025
  11. [11]OpenAI pulls chat sharing tool after Google search privacy scare · TechRadar · Aug 2025
  12. [12]xAI published hundreds of thousands of Grok chatbot conversations · Forbes · 20 Aug 2025
  13. [13]Hundreds of thousands of Grok chats accidentally published · Techzine · Aug 2025
  14. [14]Meta AI chats public: sensitive data privacy · eWeek · Jun 2025
  15. [15]Samsung bans ChatGPT use after employee leak · TechRadar · May 2023
  16. [16]Samsung bans staff AI use over data leak concerns · CSO Online · May 2023
  17. [17]Cloud and Threat Report: Generative AI 2025 · Netskope Threat Labs · 2025
  18. [18]Cloud and Threat Report 2026 · Netskope · Jan 2026
  19. [19]GenAI data violations doubled in 2025 · Help Net Security · 7 Jan 2026
  20. [20]Almost 10% of GenAI prompts include sensitive data · Security Boulevard (Harmonic Q4 2024) · Jan 2025
  21. [21]What 22 million enterprise AI prompts reveal about shadow AI in 2025 · Harmonic Security · Jan 2026
  22. [22]Report: massive amounts of sensitive data being shared with GenAI tools · Security Boulevard · Jan 2026
  23. [23]New research: AI is already the number one data exfiltration channel · The Hacker News (LayerX) · Oct 2025
  24. [24]Enterprise AI and SaaS Data Security Report · LayerX · Oct 2025
  25. [25]Sensitive data flowing into AI tools · Cyberhaven Labs · 2025
  26. [26]2026 AI Adoption and Risk Report · Cyberhaven · Feb 2026
  27. [27]13% of organizations reported breaches of AI models or applications, 97% lacking proper AI access controls · IBM Newsroom · 30 Jul 2025
  28. [28]Cost of a Data Breach 2025: navigating AI · IBM X-Force · Jul 2025
  29. [29]LLMjacking: stolen cloud credentials used in new AI attack · Sysdig Threat Research · 6 May 2024
  30. [30]The growing dangers of LLMjacking · Sysdig · 18 Sep 2024
  31. [31]LLMjacking attacks escalate, driving costs to over $100,000 a day · CyberInsider · Sep 2024
  32. [32]A single cloud compromise can feed an army of AI sex bots · Krebs on Security · Oct 2024
  33. [33]LLMjacking targets DeepSeek · Sysdig · Feb 2025
  34. [34]Disrupting a global cybercrime network abusing generative AI · Microsoft On the Issues · 27 Feb 2025
  35. [35]Microsoft names and shames four developers behind Azure OpenAI abuse · The Register · 28 Feb 2025
  36. [36]Google: LLMjacking attacks rack up $1 million in cloud AI charges · BleepingComputer · Nov 2026
  37. [37]State of Secrets Sprawl 2025 · GitGuardian · Mar 2025
  38. [38]The State of Secrets Sprawl 2026 · GitGuardian · Mar 2026
  39. [39]AI coding assistants twice as likely to leak secrets as overall leaks rise 34% · SC World · Mar 2026
  40. [40]Research finds 12,000 live API keys and passwords in DeepSeek’s training data · Truffle Security · 27 Feb 2025
  41. [41]12,000 API keys and passwords found in public datasets used for LLM training · The Hacker News · Feb 2025
  42. [42]Exposed Hugging Face API tokens · The Register · 4 Dec 2023
  43. [43]LLM10:2025 Unbounded Consumption · OWASP GenAI Security Project · 2025
  44. [44]OWASP Top 10 for Agentic Applications for 2026 · OWASP GenAI Security Project · 9 Dec 2025
  45. [45]Claude model pricing · Anthropic · Nov 2026
  46. [46]LLMjacking: when attackers run Claude on your AWS bill · DoiT · 2025
  47. [47]Italy’s privacy watchdog fines OpenAI €15 million · Euronews · 20 Dec 2024
  48. [48]OpenAI prevails in landmark Italian AI and GDPR enforcement case · Wilson Sonsini · Mar 2026
  49. [49]Italy fines Replika developer €5 million · Insurance Journal (Reuters) · 20 May 2025
  50. [50]Italy privacy watchdog fines Character.AI owner over age-check failures · The Star (Reuters) · 9 Jul 2026
  51. [51]Generative AI and GDPR enforcement in Europe · Cross-Border Data Forum · 2026
  52. [52]EU AI Omnibus enters into force, amending the AI Act · White & Case · Jul 2026
  53. [53]The Digital Omnibus on AI enters into force today · Lewis Silkin · 27 Jul 2026

Keep reading

Get started

Ship the AI.
Keep the control.

The harness is up in under two minutes. Free to start, no card needed.

Harness listening · deploy in under 2 minutes

Start free Developer plan · 50K tokens/mo · 3 seats

Book a demo

We’ll walk it through with your stack in mind.

anupama@konsole.one, sheoli@konsole.one

A product of ClearTrust · Secure · Private · Model-agnostic