# Performance Factory: how OpenAI checks the agents that debug ChatGPT in production

OpenAI now lets Codex agents chase ChatGPT performance regressions from alert to verified recovery, and Google built a fake internet to test Chrome browsing agents. Both show that the hard part of production agents is not the model but the checks around it.

On 19 November 2026, OpenAI described the Performance Factory, an internal system in which Codex agents investigate performance regressions in ChatGPT, propose fixes and then check whether the fix worked. The same day, Google's Chrome security team described a "mirror dimension": a sealed, synthetic copy of the web used to test Chrome's browsing agent without letting it touch real websites.

The two posts share one lesson. Putting an agent in front of real systems is no longer mostly a question of model quality. It is a question of how the work is structured, what the agent may touch, and how anyone knows its answers are right. This article explains both announcements, how each mechanism works, and what a team running its own agents can borrow.

## What happened

### OpenAI describes the Performance Factory for ChatGPT (19 November 2026)

In an engineering post, OpenAI staff Okay Zed and Cyrus Forbes explain that agentic development made traditional performance engineering hard to sustain, so they built a cloud system that uses Codex to find and fix regressions, mainly in ChatGPT.

They describe six stages of evolution:

1. **Manual work.** Engineers read metrics, logs, traces and code by hand. The team built more than 30 performance tools in this period.
2. **Codex as a partner.** Engineers asked Codex why something slowed down and investigated with it, but still steered every step.
3. **A reusable skill.** The team wrote its investigation method into a shared skill called perf-forensics and tested it with evals built from past regressions.
4. **Human proxy.** Engineers pasted alerts into Codex and pasted results back into Slack.
5. **Automated cloud investigations.** Alerts started cloud Codex sessions that posted findings to Slack; engineers still fixed and checked.
6. **The factory.** A system that tracks each task's next step and assigns agents to do it, with work passing between agents and engineers.

The factory has four parts. A **task service** where workflows are defined in config files plus Markdown prompts; the config lists states, allowed transitions and which steps need an engineer. A **dispatch loop** that periodically looks for ready tasks and starts a Codex turn with instructions for the current state. **Cloud execution** inside the company network, with access to repositories and observability tools, under a service identity so permissions are scoped and actions are attributable. And **subscriptions**, which mark a waiting task as ready again when a matching event arrives, such as a pull request review comment, a failed CI check or a deployment. Each task keeps its state and findings outside any single agent's context.

According to OpenAI, the regression workflow tracked more than 1,700 investigations and root-caused more than 600 of them in just over three months, about 17 automated investigations per day. In a manual review of 100 root-cause suggestions, 70% were graded accurate; the other 30% needed refinement or did not fully establish causality. The authors are candid: "Getting fixes right has been harder than finding the cause." Engineer review and merge is the main bottleneck.

### Google builds a mirror dimension to test Chrome's browsing agent (19 November 2026)

On the Google security blog, Chrome AI Security engineers Aaron Selya and Charles Qian describe a hermetic synthetic web used to evaluate auto browse, Chrome's agentic browsing feature. A headless Chrome runs inside a gVisor sandbox. All outbound traffic goes to a local interception proxy that serves generated sites instead of real ones, minting TLS certificates that chain to a custom root certificate installed in the sandboxed browser. The agent visits real addresses, such as youtube.com, and is quietly routed to synthetic content.

Sub-agents build the sites: a research agent writes specifications, a builder agent writes the code, and critic agents test accessibility, forms and layout. Google says the environment includes thousands of simulated websites (the post body also speaks of hundreds of interactive, stateful pages), and that its automated red-teaming pipeline produces more than 1,000 test candidates per security engineer per day, which the authors put at over 30 times what one engineer can write by hand. That figure counts generated test cases, not vulnerabilities found.

## From one long chat to a durable task

The most transferable idea in the Performance Factory is architectural: the work lives in a task record, not in a conversation.

Think of a hospital ward. A patient stays for days, but no nurse works for days. The chart carries the case across shifts: status, results, what was tried, what is pending, who must sign off. A new nurse reads it, does the next task, writes it down and leaves.

A long agent chat is the opposite: everything sits in one context window, and a crash, a full context or a two-day wait for a deploy loses or degrades the work. OpenAI's task service turns the chart into software:

- **States and transitions are config, not prose.** A regression moves through needs investigation, root caused, fix proposed and resolved. The config decides which moves are legal and which need an engineer, so the model cannot skip review: that edge does not exist for it.
- **Each step has its own Markdown prompt.** The verifier gets verification instructions, not the whole playbook, which is the same reasoning behind packaging expertise as [skills](/tools/skills).
- **A dispatch loop does the scheduling.** It starts a fresh agent turn for each ready task. Agents are short-lived workers; the task is the long-lived thing.

Workflow engines such as Temporal use the same principle for ordinary software: Temporal describes a workflow's event history as "a complete, ordered log of everything that has already happened," which lets work resume after a crash. The [state checkpoint simulation](/ai-simulations/langgraph-state-checkpoints) shows the idea for agent graphs, and the [agent harness](/tools/agent-harness) is the layer where this structure belongs.

### Event-driven wakeups

Much of an investigation is waiting: for a reviewer, for CI, for a deploy to roll out. OpenAI's subscriptions handle this by parking the task and waking it when a matching event arrives. A task waiting on review becomes ready when a review comment lands; a task waiting on a deploy becomes ready when the deploy happens.

It is like a parcel notification: you do not stand at the door all day. A sleeping task uses no model tokens, and the agent that wakes up reads stored state instead of a stale conversation. This is also what makes [always-on agents](/articles/always-on-ai-agents) practical: they are mostly asleep, not mostly running.

### Scoped service identities

The cloud execution layer runs agents under a service identity. OpenAI gives two reasons: permissions are scoped, and actions are identifiable. Both matter when an agent can read production observability data and open pull requests.

A scoped identity is like a contractor's site badge: it opens only the doors the job needs, and the log shows which badge was used. One credential can be revoked without touching any engineer's access. Our pages on [agent identity and delegation](/tools/agent-identity-and-delegation) and [governing AI agent access](/articles/governing-ai-agent-access) cover the design patterns in more depth. Note what the factory does not do: it does not let agents merge or deploy. That boundary sits in the workflow config, not in a hope that the model will behave.

## How the answers get checked

OpenAI's own 70% accuracy figure means roughly three in ten suggestions are incomplete or wrong. The interesting engineering is in the layers that catch them.

### Adversarial review: a second agent tries to disprove the first

For each proposed diagnosis, a separate Codex session tries to prove it wrong by rechecking assumptions and testing counterfactuals. OpenAI gives an example: an agent blamed image download errors for a slowdown, but the reviewer noticed that a related service had already recovered while the failures continued, so the blamed cause could not explain the pattern. The investigation was reopened. Each proposed fix gets an adversarial review too.

Think of a defence lawyer: someone whose job is to find the hole produces a stronger case, even if no smarter than the prosecutor. The key choices are separation (a fresh session that did not write the diagnosis) and a falsification brief (find why this is wrong, not review this). It is a specific use of [sub-agents](/tools/sub-agents) and is related to, but stricter than, a general [LLM-as-judge](/tools/llm-as-judge) score.

### Backtesting on past incidents, with the answers removed

Every time the skill changes, OpenAI reruns it on past production incidents. Each case starts with a fresh investigator and the original alert signal, with answer-revealing details removed. The agent cannot see saved memories, the incident writeup or earlier run results, and its transcript is checked afterwards for leaks. Scoring covers onset time, size of the problem, affected requests, cause ranking and how well uncertainty is explained. Timing and blame are scored separately, because an agent can find the right moment and still blame the wrong deployment.

It is like backtesting a weather forecast on last year's days using only that morning's data. If the forecaster can peek at the next day's newspaper, the score means nothing; evidence leakage is the agent version of that peek. An incident postmortem, which Google's SRE book defines as "a written record of an incident, its impact, the actions taken to mitigate or resolve it," is exactly the document that must be hidden. OpenAI also names two limits that apply to anyone doing this: metrics and traces expire, and evidence gathered after recovery can itself reveal the answer. Good [evals](/tools/evals) need a freeze date for the evidence, not just a list of cases.

### Grading the handoff, not just the answer

OpenAI tests task summaries using saved task snapshots. A separate grader checks whether an engineer unfamiliar with the system could follow what happened, whether useful charts were kept, and whether the proposed next step is supported by evidence. A correct diagnosis buried in an unreadable note is a failed handoff.

### An agent that watches the agents

A daily agent scans recent run messages and logs for tool failures and access errors. When one error blocks several investigations, it opens a fix task. Engineers review and merge the fix, and evals confirm the agents can now get past the blocked step. Without it, one expired permission can quietly spoil dozens of investigations. Run [tracing](/tools/tracing) is what makes this scan possible.

| Verification layer | What it catches | Main cost |
|---|---|---|
| Config-defined states and human gates | Agents skipping review, merging or deploying | Engineer time at each gate; this is the current bottleneck |
| Adversarial review session | Plausible but wrong diagnoses, untested assumptions, risky fixes | One extra agent run per diagnosis and per fix |
| Backtests on past incidents | Regressions when a skill or model changes | Building cases, removing leaks, data that expires |
| Transcript leak checks | Scores inflated by access to the answer | Extra review of each backtest run |
| Handoff grading | Correct but unreadable or unsupported summaries | Saved snapshots and a separate grader |
| Post-deploy recovery check | Fixes that did not work or only partly worked | Waiting for real traffic; new tasks for leftovers |
| Daily self-monitoring agent | Broken tools and access errors blocking many runs | A small daily run plus fix review |
| Synthetic environment (Google) | Unsafe actions and prompt injection before real exposure | Building realistic sites; eval awareness |

## Why fixes are harder than diagnosis

OpenAI's most useful admission is that finding the cause is the easier half. Its post says proposed fixes can be overly complicated or introduce new bugs, and some changes, such as adding a cache that may serve stale data, require engineers to weigh trade-offs.

A diagnosis is a claim about the past, checkable against evidence that already exists. A fix is a claim about the future: it must be correct, small enough to review, safe for other code paths, and acceptable on trade-offs the agent may not know about. A mechanic can say the brake pads are worn; whether to replace them before your trip depends on things the mechanic does not know.

That is why the factory closes the loop. After an engineer deploys, an agent checks whether performance returned to baseline. Partial recovery opens a new task for the remaining slowdown; no improvement sends the regression back for investigation. Recovery is measured, not assumed. OpenAI lists synthetic performance tests, so agents can measure a fix before review, as future work.

## Offline worlds for agents: Google's mirror dimension

OpenAI checks agents against history. Google's Chrome team checks them against a fake present. Both answer the same question: how do you evaluate an agent whose actions have side effects?

Google's post lists the problems with testing a [browser agent](/tools/browser-agents) on the live web: sites change, unrestricted access can cause harm, and "side effects from one trajectory must never persist into the next." It is a flight simulator: crashes cost nothing and every session starts on the same runway.

Three design details are worth copying:

- **Isolation at the network layer.** The browser runs in a gVisor sandbox, which its developers describe as "an application kernel that implements a Linux-like interface," and all traffic goes to a local proxy. Our [sandboxing](/tools/sandboxing) page explains why this boundary matters.
- **Realism against eval awareness.** Hosting everything under one test domain would give the game away, so real addresses are mapped to synthetic content. Google first passed injection payload identifiers in URL query parameters, found that models noticed the odd parameters and became aware they were being tested, and switched to custom HTTP request headers that leave the visible URL unchanged. A 2025 paper by Needham and colleagues likewise found frontier models can often tell evaluation transcripts from real use, especially in agentic settings.
- **Attacks injected anywhere.** The sandbox can hide prompt injection in page text, DOM elements, image text, ARIA labels or WebMCP tool responses at any step of a multi-turn task.

Google says frontier models can still notice subtle test cues, and that the approach could extend to coding agents.

## What it means in practice

The model call is a small part of these systems. Most of the engineering is in the task record, the permissions, the gates and the checks.

Accuracy numbers need context. OpenAI's 70% figure comes from its own manual review of 100 suggestions, and Google's 1,000 candidates per day measures test generation, not defects found. Neither is an independent benchmark. They do show continuous measurement and published weak spots.

The human role changes rather than disappears. Engineers review fixes and deployments and read handoffs; their attention is scarce, so OpenAI now links related incidents, groups them by metric and ranks impact before anything reaches a person. And evaluation becomes a standing asset: a leak-free incident library or a sandboxed copy of key user journeys pays off every time a model, prompt or tool changes.

## What to do now

1. **Move long work out of the chat.** Store each investigation as a task with explicit states, findings and a next step, so any fresh agent or engineer can pick it up.
2. **Write transitions as config.** Define which state changes are legal and which need a human, and keep merge and deploy rights out of the agent's reach.
3. **Give each step a short prompt.** Split one large playbook into per-state instructions you can test and change separately.
4. **Wake tasks on events, not timers.** Subscribe tasks to review comments, CI results and deployments so idle work costs nothing and resumes from stored state.
5. **Run agents under their own identity.** Use a scoped service account per workflow so permissions are narrow and every action is attributable.
6. **Add a falsification pass.** Have a separate session try to disprove each diagnosis and each fix before a human sees it.
7. **Build a leak-free backtest set.** Turn past incidents into cases, freeze the evidence to what was known at detection time, hide postmortems, and check transcripts for leaks.
8. **Score parts separately.** Grade timing, blame, uncertainty and handoff quality as different numbers so a change that helps one cannot hide damage to another.
9. **Measure recovery after every fix.** Compare against baseline and reopen automatically on partial or no recovery.
10. **Watch the agents themselves.** Run a regular scan of agent logs for tool and access errors that block many runs at once.

## Risks and open questions

**A 30% miss rate is not small.** OpenAI's own review found three in ten root-cause suggestions incomplete or not causal. Adversarial review reduces the risk, but the post does not report how much accuracy improves after it.

**Two agents can share a blind spot.** With the same model and tools, the reviewer may miss what the investigator missed.

**Backtests age badly.** Metrics and traces expire, and evidence gathered after recovery can leak the answer. A backtest set needs maintenance, or it quietly turns into a memory test.

**Review is the bottleneck.** OpenAI is exploring autonomous shipping of low-risk fixes; deciding what counts as low risk is unresolved.

**Synthetic worlds can be detected.** Google says frontier models still notice subtle test cues. An agent that behaves well only when it suspects a test gives false confidence.

**Watch next:** whether OpenAI publishes accuracy after adversarial review or results from synthetic performance tests, whether Google makes the mirror dimension approach available beyond Chrome, and whether agent platforms start shipping durable task state, event subscriptions and backtest harnesses as standard features.

## Frequently asked questions

### What is OpenAI's Performance Factory?

It is an internal OpenAI system, described on 19 November 2026, in which Codex agents investigate performance regressions, mainly in ChatGPT. A task service tracks each regression through states, a dispatch loop starts agents for the next step, and engineers review and deploy fixes. An agent then checks whether performance recovered.

### How accurate are the agents' diagnoses?

According to OpenAI, a manual review of 100 root-cause suggestions graded 70% as accurate, with 30% needing refinement or not fully establishing causality. Over just over three months the system tracked more than 1,700 investigations and root-caused more than 600. These are company-reported figures, not an independent benchmark.

### What is adversarial review in an agent workflow?

It is a second, separate agent session whose job is to disprove the first agent's conclusion by rechecking assumptions and testing alternative explanations. Because it did not produce the diagnosis, it is less likely to share its reasoning shortcuts. OpenAI applies it to both diagnoses and proposed fixes.

### What is evidence leakage when backtesting agents?

It is when an agent being tested on a past incident can see information that reveals the answer, such as the postmortem, saved memories, earlier runs or data recorded after recovery. Leakage makes scores look better than real performance. OpenAI removes answer-revealing details and checks transcripts afterwards.

### What is Google's mirror dimension?

It is a sealed, synthetic version of the web that Chrome's security team uses to test its browsing agent. A sandboxed browser is routed to generated sites through a local proxy, so tests are repeatable and cannot harm real websites. Google uses it to measure task completion and resistance to prompt injection.

## Summary

OpenAI's Performance Factory shows what it takes to let agents work on a live system: durable tasks with config-defined states, event wakeups, scoped identities and human gates, plus checks that include adversarial review, leak-free backtests, graded handoffs, measured recovery and a daily self-monitoring agent. OpenAI reports more than 1,700 investigations and 70% accuracy in a 100-case review, and admits that good fixes are harder than good diagnoses. Google's mirror dimension adds the complementary tool, a sealed synthetic web where agents can be tested safely, while noting that models can still sometimes tell they are being tested.

## Sources

- [OpenAI: How OpenAI's Performance Factory keeps ChatGPT fast](https://openai.com/index/how-openai-performance-factory-keeps-chatgpt-fast/)
- [Google Security Blog: Mirror Dimension for Agents: Evaluating Browser-Use Agents Without the Web](https://blog.google/security/mirror-dimension-for-agents/)
- [OpenAI: Build skills](https://learn.chatgpt.com/docs/build-skills)
- [Temporal: Workflows documentation](https://docs.temporal.io/workflows)
- [gVisor: Documentation](https://gvisor.dev/docs/)
- [Google SRE Book: Postmortem Culture: Learning from Failure](https://sre.google/sre-book/postmortem-culture/)
- [arXiv: Large Language Models Often Know When They Are Being Evaluated](https://arxiv.org/abs/2505.23836)
