# LLMjacking and shadow AI: how stolen keys and pasted data are leaking out of AI tools

In one week an AI inference provider said attackers reached customer datasets and partial model weights, and two US healthcare bodies told patients that staff had put their records into AI tools. One leak came in through stolen keys, the other walked out through a chat box. Both can be prevented.

Two kinds of AI data leak made the news in early November 2026. The first came from outside. Fireworks AI, a large inference platform, expanded its disclosure of a security incident and said an unauthorised party had reached, for some customers, datasets and incomplete model weights. The company links the activity to a group that resells stolen access to language models, a crime known as LLMjacking. The investigation is still open.

The second came from inside. Weill Cornell Medicine told patients that health information from a radiology study had been entered into AI tools without authorisation. In Washington State, a staff member at a state mental health facility used a personal ChatGPT account to draft case notes containing patients' names, diagnoses and medications. Nobody broke in. Data simply left through a text box.

These look like separate problems owned by separate teams. They share a root: AI tools are now where valuable data and valuable credentials gather, and few organisations watch them as closely as email or databases. This article explains what happened, how LLMjacking works, how to detect and prevent it, and how to stop staff pasting sensitive data into tools you do not control.

## What happened

### Fireworks AI widens its breach disclosure (9 November 2026)

Fireworks first published a security bulletin on 9 October. It said it had identified "unauthorized use of internal Fireworks credentials" on 6 October. An unauthorised party had accessed environment variables used in evaluation jobs in a limited number of customer accounts, and API keys and tokens stored there as secrets were believed captured. At that point Fireworks had found no evidence of access to customer models, datasets or inference traffic. It revoked the credentials involved and keys the actor had created, and told affected customers to rotate third-party credentials at the issuing provider, because deleting a stored secret does not revoke it.

On 9 November Fireworks published an update, as reported by Cyber Security News and others. On 23 October a third-party cloud provider hosting Fireworks research and development machines had its own incident. On 24 October the unauthorised party used that route to reach Fireworks development machines and run an unauthorised workload. Fireworks says it detected this the same day and has seen no further successful access to its customer platform since 25 October.

The update widened what was exposed: resource metadata and billing details such as addresses, invoice history and credit usage. Fireworks says payment card numbers were not stored or affected. For a limited group of customers, secrets, datasets and a subset of incomplete model weights were accessed. It reports no detected impact on its Inference, Training SDK or Nexus services. It notified law enforcement, engaged Palo Alto Networks Unit 42 for forensic analysis and disabled self-serve dataset uploads and downloads. According to these reports, Fireworks attributes the activity to a financially motivated group that sells unauthorised LLM access through illicit gateways. That is the company's own attribution.

Forbes reported on 9 November that the attack exposed customer data and AI models and that two customers had moved away, according to a summary by The Neuron. Bloomberg reported, as relayed by Seeking Alpha, that Fireworks has more than 10,000 customers; a spokesperson said it was working with customers "including Notion, Cursor, and others". Fireworks raised a Series D at a reported $17.5 billion valuation in July 2026.

Fireworks was not alone. Wiz counted at least nine AI developer platforms that disclosed incidents in October, which it called "Black October". Several blamed credential access, but Wiz found no single shared vulnerability.

### Google maps the LLMjacking economy (2 November 2026)

Google Threat Intelligence Group and Mandiant published research on LLMjacking, reported by BleepingComputer and CSO Online on 3 November. The main entry points were stolen cloud identity tokens, service account keys, GitHub tokens, session tokens and LLM API keys, plus scanning for exposed model endpoints and a flaw in the open-source LiteLLM gateway. Access was resold through "LLM mixers", relay services that pass subscribers' prompts through victims' accounts; one advertised plans at about $60 a month. Google said some organisations saw fraudulent charges above $100,000 in a day and up to $1 million in total. These figures come from other incidents, not Fireworks.

### Weill Cornell Medicine notifies radiology patients (5 November 2026)

Weill Cornell Medicine posted a notice dated 5 November, reported by Becker's Hospital Review and the HIPAA Journal. On 6 September it learned that patient information used in a radiology research study had been entered into AI and machine learning tools "without authorization and in violation of its policies" between 10 and 20 June 2026.

The data included medical record numbers, radiology accession numbers, dates of service, facility and provider names, and clinical details that may include diagnoses, procedures and medications. Names, contact details and Social Security numbers were not involved. Weill Cornell says it has found no misuse. It suspended the research activity, asked for the data to be deleted and added staff education. The notice does not name the tools, say who entered the data or give a patient count.

### Washington DSHS staff used ChatGPT for case notes (reported 2 November 2026)

The Seattle Times reported on 2 November, and the HIPAA Journal on 9 November, that a staff member at Lake Burien Transitional Care Facility, a state facility for young people with intensive mental health needs, used a personal ChatGPT account to draft case note summaries. A colleague reported it in July. The Department of Social and Health Services (DSHS) found earlier instances and, in mid-August, contacted the families of 12 patients. A similar incident at Western State Hospital affected four patients.

DSHS policy already banned AI use without agency approval. The employee was told to delete all prompts before deleting the account. One family's letter said it is "unknown how the information is stored within the ChatGPT server."

## How LLMjacking works

LLMjacking is the theft and resale of someone else's access to AI models. Sysdig's threat research team coined the term in May 2024, after attackers used stolen AWS credentials to run cloud-hosted models at the victim's expense.

Think of a company fuel card. A thief copies the number and sells fill-ups to strangers at half price. The strangers get cheap fuel, the thief gets cash and the company gets the bill. Here the fuel is model tokens, the card is an API key or cloud credential, and the strangers are customers of a relay service.

### What attackers want

**Compute.** The simplest goal is free inference, used directly or resold. The cost lands on your invoice.

**Data.** Inference platforms hold fine-tuning datasets, evaluation sets, uploaded files and sometimes prompt logs. A training dataset is often a distilled copy of a company's best internal knowledge.

**Models and secrets.** A fine-tuned model embodies paid compute and private data, so weights, adapters and checkpoints have value. Secrets stored on the platform, such as keys for other services, let an attacker move sideways. Fireworks' first incident was exactly that.

### The resale chain

Google describes a division of labour. One group finds access by scanning for open endpoints, harvesting keys from public code and infostealer logs, or breaking into machines that hold credentials. Another aggregates keys and routes traffic through gateways. A mixer sells subscriptions to users who may not know where the capacity comes from. Wiz saw stolen keys tested within 30 minutes, often with tiny requests that do not trip alarms. Heavy use comes later.

### Common entry points

| Entry point | What it looks like | Main fix |
|---|---|---|
| Leaked keys | Keys in public repositories, notebooks, CI logs or infostealer dumps | Secret scanning, short-lived credentials, fast rotation |
| Exposed endpoints | Model servers or gateways reachable from the internet without authentication | Private networking, authentication everywhere, patching |
| Over-broad service accounts | One identity that can enable model services, read storage and create users | Least privilege, deny model access by default |
| Secrets inside jobs | Keys passed to training or evaluation jobs as plain environment variables | Inject narrow secrets at run time |
| Third parties | A supplier hosting your machines is breached | Segment and monitor supplier hosts |

Self-hosting is not automatically safer. A [vLLM](/tools/vllm) or [Triton](/tools/triton) server on a public IP with no authentication is an open buffet.

## How to detect it

LLMjacking is a money crime first, so the earliest signal is often in billing. Think of a water meter: if it is spinning at 3 a.m. while everyone sleeps, something is leaking, even if no tap looks open.

**Spend anomalies.** Track cost per key, per project and per hour. A month-end invoice is far too late when fraud can reach $100,000 in a day. Use hard caps where possible; see [cost and latency budgets](/tools/cost-and-latency-budgets).

**Unusual models and regions.** Attackers favour the most capable, most expensive models. Alert on first use of a model and on traffic from regions you do not use.

**Odd request shapes.** Watch for many tiny validation calls, off-hours traffic on batch keys and sudden jumps in requests per second.

**Identity changes.** Google flags new credentials, new users, permission changes and attempts to disable logging, especially alongside new model usage. Fireworks found keys the actor had created, a classic way to stay in.

**Inventory gaps.** You cannot spot misuse of a key you do not know exists. Treat any unlisted key as an incident.

[Model monitoring](/tools/model-monitoring) usually tracks accuracy and drift; for inference it should also cover cost, volume and caller identity. Traces built with [OpenTelemetry](/tools/opentelemetry) can carry key, tenant, model and token counts for every call.

## How to prevent it

Prevention is mostly ordinary cloud security, applied to a service many teams treated as a simple API.

**Short-lived credentials.** A key that expires in an hour is worth little to a reseller. Prefer workload identity federation or instance roles to static keys, and rotate any static key on a schedule.

**Scoped keys.** Give each key one application, one environment, a list of allowed models and a spending cap. Google advises denying all permissions not explicitly required, including access to cloud LLM services. The same principle for agents is covered in [governing AI agent access](/articles/governing-ai-agent-access) and [agent identity and delegation](/tools/agent-identity-and-delegation).

**Secrets out of jobs.** Fetch secrets at run time from a secrets manager, limited to the job's identity. Fireworks said it is reviewing how secrets reach evaluation jobs, which shows how easy this gap is to miss.

**Egress control.** Decide which machines may call which model providers and block the rest. This stops attackers using your keys from your machines and stops staff sending data to unapproved tools. It is central to [Zero Trust](/tools/zero-trust).

**Per-tenant isolation.** On a shared platform, keep each customer's data, jobs, secrets and logs in separate boundaries, with separate encryption keys where possible, so one stolen credential exposes one tenant.

**Treat development machines as production.** Fireworks' second route in was a development environment at a third-party provider. Such hosts often hold real credentials with weaker controls.

**Logging the attacker cannot erase.** Send cloud audit and inference logs to an account the workload cannot write to, and alert on attempts to switch them off.

## The shadow AI data-leak problem

"Shadow AI" means staff using AI tools the organisation has not approved, usually consumer chatbots on personal accounts. It is rarely malicious. A clinician facing a pile of case notes will use a tool that summarises them in seconds unless something better is offered.

Picture an office whose copier is broken. Staff carry confidential files to the copy shop down the street. Banning the copy shop helps a little; fixing the office copier helps much more.

The HIPAA Journal, citing a Wolters Kluwer Health survey, reports that 40% of hospital workers knew of colleagues using unauthorised AI tools and 20% admitted using them. It says consumer AI tools lack the necessary safeguards and must not be used with protected health information, while many enterprise tools can be used compliantly. The difference lies in the contract and controls: who stores prompts, for how long, whether they train models and whether a business associate agreement exists.

### Four controls that work together

**Approved tools.** Offer a sanctioned assistant with an enterprise agreement, controlled retention, single sign-on and audit logs. If it is slow or weak, people return to the copy shop.

**Data loss prevention (DLP).** DLP inspects data leaving the organisation. For AI, that means browser and network controls that recognise consumer AI sites, detect identifiers in pasted or uploaded text, and block, warn or redirect to the approved tool. Weill Cornell's data had no names, yet record numbers and clinical details can still identify a person, so rules must cover such identifiers too.

**Redaction and pseudonymisation.** Strip or replace identifiers before a prompt leaves your network. A gateway can swap a record number for a token and swap it back in the answer. [Format-preserving encryption](/tools/format-preserving-encryption) keeps a value's shape for downstream systems, and input [guardrails](/tools/guardrails) can enforce this on every call. Free text defeats redaction often enough that it reduces risk rather than removing it.

**Training and clear rules.** DSHS already had a policy; it did not stop a busy employee. Training should use real examples, explain why consumer and enterprise tools differ, and name the tool to use instead. Research teams need explicit rules, since a study can feel like an exception.

All four belong under one [AI governance](/tools/ai-governance) process with an owner, a tool inventory and a fast route to approve new tools.

## What it means in practice

**AI platforms are high-value targets.** An inference provider holds keys, datasets, fine-tuned weights and billing data for thousands of customers. October's cluster of incidents suggests organised effort.

**Your provider's breach can become yours.** Fireworks' first incident exposed secrets customers had stored on the platform, including keys to other services. Store only what is scoped and rotatable.

**Cost is a security signal.** Finance and security should share AI spend dashboards. A billing spike is often the first sign of compromise.

**Data minimisation defends against both leaks.** A dataset never uploaded cannot be stolen from a provider, and identifiers removed before a prompt cannot leak through it.

**Regulated sectors will see more notices.** Both healthcare cases surfaced months after the event through internal reports. Others will find similar cases once they look.

## What to do now

1. **Build a key inventory.** List every AI provider, key, owner, purpose and expiry. Revoke anything without an owner.
2. **Rotate and shorten.** Rotate long-lived AI keys now and move to short-lived or federated credentials where supported.
3. **Scope every key.** Restrict each to named models, one environment and a hard spending cap; deny model services by default elsewhere.
4. **Alert on spend by the hour.** Alert per key on cost jumps, first use of a model and traffic from unused regions.
5. **Move secrets out of job configuration** and audit existing training and evaluation jobs for embedded keys.
6. **Close exposed endpoints.** Scan your address ranges for model servers and gateways, require authentication and patch gateways quickly.
7. **If you use Fireworks, follow its bulletin.** Rotate third-party credentials at the issuer and check for unfamiliar keys, users, jobs and datasets.
8. **Give staff an approved AI tool** with an enterprise agreement, controlled retention and single sign-on.
9. **Deploy AI-aware DLP** that catches identifiers in pastes and uploads to unapproved tools and redirects users.
10. **Test your controls.** Add credential theft, spend abuse and data pasting to your [red-teaming](/tools/red-teaming) exercises.

## Risks and open questions

**The Fireworks investigation is not finished.** Its scope has already grown once, from secrets in some accounts to datasets and partial weights. The LLMjacking attribution comes from Fireworks, and the cloud provider has not been publicly named.

**Numbers are missing.** Weill Cornell has given no patient count, and Fireworks has said only "a limited number" and "a limited group" of customers.

**Deletion is hard to prove.** Neither Weill Cornell nor DSHS can easily verify what a provider kept in logs or backups. Contracts, not technology, must close this gap.

**Controls can be bypassed.** Free-text notes are hard to scan, and personal phones sit outside the network. DLP reduces risk; it does not remove it.

**Stolen fragments have unclear value.** It is not yet known whether incomplete weights are useful to an attacker or competitor.

**Watch next:** further Fireworks bulletins and Unit 42 findings, whether other "Black October" platforms report data or model theft, providers moving to short-lived scoped keys by default, and any US guidance on staff use of consumer AI tools.

## Frequently asked questions

### What is LLMjacking in simple terms?

It is using stolen cloud or API credentials to run AI models at someone else's expense, often by reselling the access. Attackers may also take the datasets, fine-tuned models and secrets stored alongside the credentials.

### Did hackers steal complete AI models from Fireworks?

According to Fireworks' 9 November update, as reported, a subset of incomplete model weights and datasets was accessed for a limited group of customers. Theft of complete production models has not been established, and the investigation continues.

### How do I know if my AI keys are being abused?

Look for sudden cost rises, calls to models you do not use, traffic from unfamiliar regions, many tiny test requests and keys or users you did not create. Hourly spend alerts per key catch most cases early.

### Can healthcare staff put patient data into ChatGPT?

Not into a consumer account. Entering protected health information into a tool without the required safeguards and agreements can be an impermissible disclosure under HIPAA. Enterprise tools with proper agreements can be compliant, subject to employer policy.

### Is banning AI tools the best way to stop shadow AI?

Not on its own. Bans without a good alternative push use onto personal devices where nobody can see it. An approved tool, DLP, redaction and training work better together.

## Summary

Early November 2026 showed both ways AI tools leak. Fireworks AI expanded its breach disclosure to datasets and incomplete model weights for some customers, reached through stolen internal credentials and a third-party cloud provider's incident, and linked the activity to an LLMjacking group; Unit 42 is helping with an investigation that remains open. Google's research showed stolen keys resold through relay services, with some victims billed up to $1 million. Weill Cornell Medicine and Washington DSHS notified patients after staff entered health data into AI tools. The defences overlap: inventory and scope every key, use short-lived credentials, watch spend hourly, control egress, isolate tenants, keep tamper-proof logs, and give staff an approved tool backed by DLP, redaction and training.

## Sources

- [Fireworks AI: October 2026 security incident](https://fireworks.ai/security/bulletins/october-2026-security-incident)
- [Cyber Security News: Hackers breach Fireworks AI through third-party cloud provider](https://cybersecuritynews.com/fireworks-ai-data-breach/)
- [The Neuron: Fireworks AI was hacked, how criminals profit from stolen AI](https://www.theneuron.ai/news/fireworks-ai-breach-stolen-ai-access-llmjacking/)
- [Seeking Alpha via TradingView: Fireworks AI hit by second cyber incident](https://www.tradingview.com/news/seekingalpha:b5cb06a44094b:0-nvidia-backed-fireworks-ai-hit-by-second-cyber-incident-exposes-data-report/)
- [Intelligent CIO: Fireworks raises US$1.505 billion Series D](https://www.intelligentcio.com/north-america/2026/07/31/fireworks-raises-us1-505-billion-series-d-as-enterprise-demand-for-specialised-ai-accelerates/)
- [Wiz: Black October for AI developer platforms](https://wiz.io/blog/black-october-for-ai-developer-platforms)
- [BleepingComputer: Google says LLMjacking attacks rack up $1 million in cloud AI charges](https://www.bleepingcomputer.com/news/security/google-llmjacking-attacks-rack-up-1-million-in-cloud-ai-charges/)
- [CSO Online: Enterprises pay $100,000 a day for AI they never ordered](https://csoonline.com/article/4243863/enterprises-pay-100000-a-day-for-ai-they-never-ordered.html)
- [The Hacker News: Researchers uncover LLMjacking scheme](https://thehackernews.com/2024/05/researchers-uncover-llmjacking-scheme.html)
- [Becker's Hospital Review: Weill Cornell patient data entered into AI tools without authorization](https://www.beckershospitalreview.com/healthcare-information-technology/ai/weill-cornell-patient-data-entered-into-ai-tools-without-authorization/)
- [HIPAA Journal: Healthcare providers discover unauthorized PHI disclosures to AI tools](https://www.hipaajournal.com/healthcare-providers-unauthorized-disclosure-phi-ai-tools/)
- [The Seattle Times via Yakima Herald-Republic: AI use leads to potential privacy breach at WA mental health facilities](https://www.yakimaherald.com/news/northwest/ai-use-leads-to-potential-privacy-breach-at-wa-mental-health-facilities/article_2011d3ac-633e-5bc7-a0aa-0a15436a4993.html)