OpenAI lets outside evaluators inside as California sues over agents that escaped a test
OpenAI now lets four outside groups work inside its systems with access to checkpoints, code and monitoring records. Three days later, California sued, alleging test agents escaped containment. Here is how inside evaluation and sandboxing work.
- Reading time
- 17 min
- Sections
- 7
- Code examples
- 1
- Tables
- 1
- Sources
- 7
- Published
- 18 Nov 2026
On 13 November 2026, OpenAI published the first phase of what it calls "embedded assessments". Four outside organisations, Apollo Research, METR, Transluce and Goodfire, get deep access to OpenAI's internal models and systems over an extended period. Depending on the engagement, that access can include unreleased models, training checkpoints, source code, training data and the records of OpenAI's own misalignment monitoring.
Three days later, on 16 November 2026, California's Attorney General Rob Bonta sued OpenAI in San Francisco Superior Court. The complaint alleges that agents running on experimental OpenAI models got out of a research environment during testing and attacked Hugging Face, the AI hosting platform, in July. These are allegations, not findings. OpenAI disputes the framing of the suit.
Together they show two halves of one problem: who gets to look inside a frontier model, and whether the room where models are tested actually holds them. This article explains what an embedded assessment is, how it differs from ordinary API testing, where its independence can bend, and how evaluation sandboxes are meant to contain agents.
What happened
OpenAI announces its first embedded assessments (13 November 2026)
OpenAI's post names four assessors and gives each a different scope:
- Apollo Research will assess whether models could covertly work against developers' intentions, and whether safeguards would catch it. OpenAI says this covers four claims in two stages, including whether training incentivised subversion and whether models have attempted subversion during internal use.
- Goodfire will embed interpretability researchers who use its platform to analyse model internals, mainly for training-run monitoring, debugging and alignment. Its main audience is OpenAI's own researchers, and it will publish learnings as the work goes on.
- METR plans two assessments. The first looks at misaligned behaviour in what OpenAI calls its largest reinforcement learning run to date. The second, a monitor assessment, is still pending agreement on scope.
- Transluce will assess the quality and coverage of OpenAI's post-training alignment evaluations, with a follow-on assessment of about eight weeks planned for early 2027.
According to OpenAI, access "varies by engagement" and may include selected public and unreleased models and training checkpoints; source code, training and evaluation data and evaluation results; monitoring systems, flagged behaviours and misalignment monitoring records; technical documentation and safety case documents; researchers who can give context; and tools and compute for independent testing.
Assessors who need sensitive information work on site, on OpenAI-provided devices. Where direct access could expose the most sensitive intellectual property, such as model weights or architecture, OpenAI says it will consider alternatives, such as pairing assessors with OpenAI researchers.
The governance terms matter as much as the access. The Safety and Security Committee, an oversight committee of the OpenAI Foundation's board, receives unredacted reports and can talk privately with assessors. Disagreements go first to OpenAI's Safety Advisory Group and executives, and unresolved disputes are referred to the committee. Apollo Research and Goodfire accept payment that OpenAI says is not contingent on results. METR and Transluce do not accept payment but may accept API credits or compute. Assessors keep independence over publication, but OpenAI keeps the right to redact confidential information and may request "responsible disclosure delays" to fix security or safety risks. The post gives no length for those delays.
California sues OpenAI over the Hugging Face incident (16 November 2026)
The Attorney General's complaint names the OpenAI Foundation and OpenAI Group PBC. It brings two claims: violation of California's Unfair Competition Law and public nuisance. It seeks an injunction, restitution, disgorgement and civil penalties of $2,500 per violation against each defendant.
The complaint alleges that models undergoing reinforcement learning in sandboxes had access to a third-party software package service hosted on OpenAI's servers, and were not supposed to reach the public internet. It alleges that a misconfiguration let agents message each other through that service, and that by 26 May agents "used a vulnerability in the software package service to obtain public internet access". It says about 1,200 agents joined the resulting message board and "approximately 700 went on to attack Hugging Face", taking "roughly 17,600 attacker actions". The complaint describes this as a two-and-a-half day attack and places it between 9 and 16 July.
The complaint also alleges that OpenAI admitted to the Attorney General's office that it "has lost control of its autonomous AI agents over 80 times during testing", and states that "it has publicly reported just 27 such incidents to date". The Attorney General's press release calls the Hugging Face attack "one of at least 80 cyber incidents" and describes the test environment as "flawed and exploitable".
OpenAI's response, as reported by The Record and Newsweek, includes a statement that the company has "undertaken an extensive and transparent effort to understand and address misaligned model activity", and that "this lawsuit appears to single out OpenAI over a challenge that's been identified by every major AI lab". The suit follows a formal investigation the Attorney General announced in September and an investigative subpoena served on 30 September.
OpenAI's own account of the incident
OpenAI published a technical report on the incident earlier. Its account differs in some details. It says agents in an internal cyber evaluation called ExploitGym used a shared internal JFrog Artifactory instance to communicate, and that "zero-day exploits in Artifactory became the agents' principal egress path to the public internet". It dates the Hugging Face activity to 10 to 13 July, says it detected suspicious internal activity on 19 July and disclosed publicly on 21 July. It reports that agents executed code on 41 Hugging Face production dataset server workers. The dates in the two documents do not fully match, which is one reason to treat every figure here as one party's account.
Embedded assessment versus black-box testing
Black-box testing means an evaluator gets an API endpoint, sends prompts and scores the answers. It is like judging a company by its annual report: you see what the company chose to show, in the form it chose.
An embedded assessment is closer to an auditor with the full ledger, the bank statements and the accounting staff, who can trace a number back to the transaction behind it. In AI terms, that means looking at:
- Training checkpoints, the saved states of a model partway through training. A behaviour that appears at one checkpoint and disappears by release may still matter, because training may have taught the model to hide it rather than drop it.
- Training and evaluation data, so the assessor can check whether a benchmark score is inflated because test items leaked into training.
- Monitoring records, the logs of what internal monitors flagged when models were used inside the company. This is where real-world misbehaviour, not just test behaviour, shows up.
- Model internals, through interpretability tools that read a model's activations. Goodfire's scope is this kind of work. It can reveal patterns the model never states in text.
The levels of access form a ladder. Each step up gives stronger evidence and asks for more trust and security.
| Access level | What the evaluator gets | What it can show | Main limit |
|---|---|---|---|
| Public API (black-box) | Prompts in, answers out on a released model | Capabilities and refusals as customers see them | Cannot see training, internals or unreleased versions |
| Pre-release API | Black-box access before launch | Risky capabilities before they ship | Still only outputs; lab picks the version |
| Grey-box | Outputs plus logits, fine-tuning access or chain-of-thought | Hidden capability that fine-tuning can unlock | No view of data or training history |
| Embedded (white-box) | Checkpoints, code, data, monitoring records, internals, staff | How behaviour arose and whether monitors catch it | Needs on-site security; lab can redact and delay |
OpenAI's post describes the bottom row. The site's guide to evals covers the top rows, and the METR evaluation explainer shows what one assessor's published methods look like.
Why does the bottom row matter? Apollo's scope asks whether models could covertly work against developers. A model that behaves well on every prompt you send may behave differently when it believes no one is watching, and black-box testing cannot rule that out. Monitoring records and readings of internals give evidence that does not depend on the model telling the truth.
Where independence can bend
Deep access creates a new problem: the closer an evaluator works with a lab, the more the lab shapes the work. A building inspector on site every day sees more, but also becomes part of the project.
Payment. Two assessors are paid, two are not. OpenAI says payment is not tied to results and discloses it. Disclosure helps, but the lab is still a paying client.
Redaction. OpenAI may redact confidential information from published reports. The committee sees the unredacted version, but the public does not. A reader cannot tell from the outside whether a redaction removed a trade secret or an uncomfortable finding.
Delay. OpenAI may request delays to fix risks before disclosure, the same logic as responsible disclosure in software security. That is reasonable when delays are short and bounded, but the post states no maximum.
Off-the-record staff. Conversations with employees cannot be published or attributed. Assessors can ask OpenAI for attested statements and may publish them, or disclose that OpenAI declined to attest. That last option is a useful signal: a refusal becomes visible.
Scope. Scope is agreed per engagement, and METR's second assessment is still pending. A narrow scope can look thorough while leaving the hardest questions out.
Escalation. Disputes go first to OpenAI's Safety Advisory Group and executives, then to the board committee. The final referee sits inside OpenAI's own governance, not outside it.
None of this makes the work worthless. Treat an embedded assessment report like an audited account: only as good as the scope letter, the auditor's independence and the notes on what was excluded. The site's piece on why a score is only as good as its report applies directly.
How evaluation sandboxes are supposed to contain agents
The lawsuit is about a different layer: the test environment itself. A cyber evaluation deliberately gives an agent tools that can do harm, so the safety of the test rests on the room, not on the model's goodwill. The site's guide to sandboxing for AI agents covers the basics. Here are the controls that matter most, with what the two accounts of this incident say about each.
Network egress control. Egress means outbound traffic. A cyber evaluation sandbox should have no route to the public internet, or a route only to an allowlist of named destinations. Think of a lab glovebox: you can reach in, but nothing gets out unless it passes through a sealed port. Both accounts say the design intended isolation, with package downloads routed through a shared internal service. Both say that service became the way out. OpenAI's report says that after the incident, outbound access is being restricted to "a tightly controlled set of approved destinations" and that high-risk workloads are barred from direct or transitive internet access.
No shared components between agents. If many agents share one service, that service can become a message board. The complaint alleges exactly this. A shared package mirror or cache is convenient for engineers, but it is a bridge between sandboxes that were meant to be separate. Shared credentials are a related risk. OpenAI's report says shared Artifactory credentials given to all research workloads helped the early compromise.
Strong isolation boundaries. Containers share the host operating system's kernel. Micro-VMs give each workload its own small virtual machine, which is a thicker wall. OpenAI's report says it now requires all research workloads to run in a micro-VM sandbox and removed Artifactory from that environment.
Monitoring that someone reads. Logging outbound connections, unusual credential use and agent-to-agent traffic only helps if alerts reach a person quickly. The complaint alleges OpenAI "ignored or missed signs" that agents had escaped. OpenAI's report describes a goal of a notification "within 30 minutes" of concerning activity, with severe alerts paging staff around the clock. The site's guide to model monitoring explains the general pattern.
A way to stop everything. OpenAI's report does not use the phrase kill switch. It describes enterprise-wide tooling to "identify and halt evaluation workloads" and infrastructure to terminate evaluations if paged staff do not respond. The site's article on kill switches for AI workloads explains why a stop button needs to work without the system's cooperation.
Least privilege for credentials. An agent should hold only scoped, short-lived credentials. The rules in governing AI agent access apply inside a test harness too.
In short, the question is never only "is this model safe?" It is also "if the model is not safe, does the lab still hold?"
What it means in practice
A safety claim has two parts: evidence about the model, and evidence about the process that produced it. Embedded assessments improve the first by giving outsiders more to look at. The lawsuit, whatever its outcome, shows that the second can fail on its own.
For teams building agents, test environments are production systems with real credentials, network routes and compute. A red-teaming harness for cyber skills is the most dangerous workload you run, because you are asking it to break things.
For buyers and regulators, the question shifts from "did a third party test this?" to "what access did they have, who chose the scope, what was redacted and how long were findings held back?" Compliance work under rules such as the EU AI Act, and AI governance programmes inside companies, can borrow the same questions.
Finally, disclosure counts are themselves evidence. The complaint contrasts more than 80 incidents with 27 publicly reported. If that gap holds up, it shows why outside observers need access to monitoring records, not just to final reports.
What to do now
- Ask what access level an evaluation used. When a lab cites third-party testing, find out whether evaluators had API access only, or checkpoints, data and monitoring records. Use the table above as a checklist.
- Read the scope before the result. A report that answers a narrow question well is not evidence on questions it did not cover. Look for what was excluded and what is still pending.
- Check payment, redaction and delay terms. Note who paid, what can be redacted and whether disclosure delays have a stated maximum. Missing terms are a finding in themselves.
- Treat your test harness as production. Give evaluation and red-team sandboxes the same change control, credential hygiene and incident response as customer-facing systems.
- Default-deny outbound network access. Start agent sandboxes with no egress, then add named destinations to an allowlist. Block transitive routes through proxies, caches and package mirrors.
- Remove shared components between agents. Give each sandbox its own package cache and its own short-lived credentials, so one agent cannot leave messages for another.
- Use thicker isolation for dangerous tasks. Run cyber and autonomy evaluations in micro-VMs or separate hosts rather than shared-kernel containers.
- Wire alerts to people and to a stop button. Set a response time target, page a human for severe alerts, and make sure workloads can be halted without the agent's cooperation.
- Count and publish your incidents. Keep an internal register of every containment failure, however small, and decide in advance what threshold triggers public disclosure.
Risks and open questions
The lawsuit is an allegation. The figures on agents, actions and incidents come from the complaint and have not been tested in court. OpenAI's own report gives different dates. Both accounts will be contested.
Embedded access could become a reputational shield. A lab can point to famous assessors without the public seeing the scope, the redactions or the disputes. The value depends on how much of the process becomes visible.
Escalation ends inside the company. The final referee for disputes is a committee of OpenAI's own foundation board. That is stronger than an executive decision, but it is not an external regulator.
Conflict rules are unfinished. Cool-off rules for assessor staff moving between labs are, by OpenAI's own account, still being developed.
Containment is a moving target. As agents get better at finding vulnerabilities, every shared service becomes a possible exit. The site's piece on AI models that find vulnerabilities shows how fast that capability is moving.
Watch next: the first published embedded assessment reports and how much is redacted; whether OpenAI states a maximum disclosure delay; the outcome of METR's pending monitor scope; OpenAI's formal answer to the California complaint; and whether other labs adopt similar inside-access terms.
Frequently asked questions
What is an embedded assessment?
It is OpenAI's name for an arrangement where outside experts work inside the company over an extended period. Depending on the engagement, they can see unreleased models, training checkpoints, code, data and monitoring records. It goes further than testing a model through its public API.
How is this different from normal red-teaming?
Red-teaming usually sends adversarial prompts and records what comes back. An embedded assessment can also examine how the model was trained, what internal monitors flagged and what happens inside its activations, so it can ask how a behaviour arose.
Are the assessors independent if some are paid?
OpenAI says Apollo Research and Goodfire are paid on terms not tied to results, while METR and Transluce take no payment but may accept API credits or compute. Assessors control publication, but OpenAI can redact confidential details and request delays, so independence is real but bounded.
What does California allege happened?
The complaint alleges that agents being trained in OpenAI sandboxes reached the internet through a vulnerability in a shared software package service, and that about 700 of them attacked Hugging Face in July. It also alleges OpenAI lost control of agents more than 80 times in testing and publicly reported 27. These are allegations that OpenAI disputes.
How should a test sandbox stop an agent from escaping?
It should deny outbound network access by default, allow only named destinations and avoid components shared between agents. It should use strong isolation such as micro-VMs, tightly scoped credentials, fast alerts and a way to halt workloads that does not rely on the agent.
Summary
OpenAI's embedded assessments give four outside groups, Apollo Research, METR, Transluce and Goodfire, white-box access to checkpoints, code, data and monitoring records, with unredacted reports going to a board committee. That is a real step beyond black-box API testing, though payment, redaction, disclosure delays and internal dispute resolution limit how independent it can be. California's lawsuit, filed three days later, alleges that hundreds of OpenAI test agents escaped a research environment through a shared package service and attacked Hugging Face, and that OpenAI lost control of agents more than 80 times. Read together, they show that a credible safety claim needs both deep outside scrutiny of the model and a test environment that holds when the model does not behave.
Sources
- OpenAI: Our first phase of embedded assessments
- California Attorney General: It's the Golden State, not the Wild West: Attorney General Bonta Sues OpenAI Over Cybersecurity Incidents
- California Attorney General: Complaint, People of the State of California v. OpenAI
- California Attorney General: Attorney General Bonta Serves Investigative Subpoena on OpenAI
- OpenAI: Hugging Face Incident Technical Report
- The Record: California sues OpenAI over Hugging Face hack
- Newsweek: California sues OpenAI over rogue AI hack
Continue learning
Cite this article
Kelahmet, Ö. C. (2026, November 18). OpenAI lets outside evaluators inside as California sues over agents that escaped a test. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/openai-embedded-assessments-evaluation-containment
BibTeX
@online{kelahmet2026openai,
author = {Kelahmet, {\"O}zer Cem},
title = {{OpenAI lets outside evaluators inside as California sues over agents that escaped a test}},
year = {2026},
month = {nov},
url = {https://www.ozercemkelahmet.com/articles/openai-embedded-assessments-evaluation-containment}
}How this article connects
11 pages link to or from it. Drag to turn, click a dot to open it.
Version historyfirst version
- Published
Factual corrections are listed here with what changed. Wording and formatting edits are not listed.