Agent Simulation via Trace Replay: Raindrop Approach + OpenAI Paper
Simulation means replaying real production traces against a proposed change to an agent, then comparing what changed. You take a captured trace, cut it at a point, let the changed agent continue from there, and score the result with deterministic diffs and a judge.
The question it answers is "what does my change change?" The change can be a new model, an added or removed tool, a prompt edit, or a harness swap. Raindrop uses this as a pre-merge regression check. OpenAI uses the same mechanic to forecast how often misbehavior will happen after a model release.
Sources and how they relate
The two sources use the same replay mechanic for different purposes. Ben Hylak (Raindrop CTO) cites the OpenAI paper as precedent and borrows its tool-simulation and realism-check ideas.
| Raindrop talk (Hamel Husain's evals course) | OpenAI paper, July 2026 | |
|---|---|---|
| Purpose | Regression check on each change | Forecast misbehavior rates before release |
| What changes | Anything: model, tool, prompt, harness | The model |
| Judge asks | Is the outcome the same? Did the known bug recur? | Does this response show misbehavior X? |
| Reference | Old trace (good, bad, or just current behavior) | Misbehavior criteria, not the old response |
| Scale | 5 to 50+ traces, picked to touch the change | ~1.3M conversations graded |
| Validated against | Replaying main should equal main | Real production rates after launch |
The idea itself is not new. It is traffic replay testing (for example Twitter's Diffy, 2015) and snapshot testing, adapted to agents. The agent-specific hard parts are simulating the environment and keeping the agent from noticing it is being tested.
The core loop
Capture a trace, cut it, let the changed agent continue, and compare. Raindrop's full cycle is log, detect issues, simulate the fix, ship, repeat.
flowchart LR A[Production traces] --> B[Select traces
that touch the change] B --> C[Cut at branch point] C --> D[Replay with change
in simulated world] D --> E[Diffs + judge
vs old trace] E --> F[Review differing traces] F --> G[Ship or fix]
- Capture. Keep full traces from production: user messages, system prompt, tool schemas, tool calls and results, retrieved content. Missing context here breaks the replay later.
- Cut. Pick the branch point, usually right before the agent turn your change affects. Everything before it is fixed input.
- Replay. Run the changed agent from that point. It can take as many internal steps (tool calls, sub-agents) as it needs within that one user turn.
- Run several times. Outputs vary run to run, so replay each trace a few times and look at rates.
- Compare. Cheap diffs first, then a judge where exact comparison does not work.
- Look at the traces. Read the ones that changed. Ben says this tells you more than the eval suite.
Choosing traces
Which traces you pick matters more than how many. A thousand replays tell you nothing about a removed tool if none of them called that tool.
| Change | Traces to pick | Rough count (Ben) |
|---|---|---|
| Remove or change one tool | Traces where that tool was called | 5 to 10 |
| Fix a known bug | The bad traces that showed the issue, plus a random sample | Enough to cover the failure |
| Prompt or model swap | Random sample plus targeted edge cases | Dozens |
| Harness rewrite | Broad random sample | 50+ |
- Always mix in a random sample. Replaying only failures overfits to them.
- Traces do not need to be good. They can be verified good (expect the same answer), known bad (expect the bug gone), or just current behavior (expect no drift).
- For rate estimates, sample randomly. The paper's advantage over hand-picked evals came from a representative sample of traffic. Hand-picked failures give the wrong rates even when every prompt is real.
Building the world
The hardest part is recreating the environment the agent saw when the trace ran. Neither of the obvious shortcuts works.
- Cached tool results fail. Once the new agent makes a different call, the recorded results no longer fit. Ben says the agent will "instantly call your bluff" and say it is in a simulation.
- Live production fails. State has changed since the trace ran. A delivery bot asked "how far away is my order?" a week ago would find nothing pending today. Replaying against live systems is also unsafe.
What Raindrop does (the "Swiss cheese" model):
- Run real code as far as possible. They check out the customer's agent into a sandbox and run it unchanged. Deterministic parts, like an internal SQL dialect parser, run for real. They also spin up sandboxes if the agent uses them.
- Simulate at the network boundary. Past that point an LLM generates tool responses.
- Seed from the original trace. Each original tool result is a window into the world at that time. The simulator fills the holes between those windows so the answers stay consistent.
- Add real data when needed. For a brand-new data source, users can drop in JSON files or connect a read-only MCP (for example Supabase) so the world builder can pull real schemas and data.
The paper confirms how much this matters. Plain simulated tool outputs were picked as the real one only 11.6% of the time in a blind comparison. Giving the simulator richer context raised that to 49.5%, close to the 50% that means indistinguishable.
Evals over traces
An eval here is a pure function over traces: f(new trace, old trace) returns pass or fail. Once defined, it runs on any trace, unlike input/output pairs that have to be collected one by one.
Raindrop layers it this way: simulation is the base infrastructure, anomaly detection flags anything weird, and evals are the scores you define and track.
Start deterministic, then move up to judges.
| Level | Examples | Cost |
|---|---|---|
| Deterministic diffs | Final answer changed; new tool errors; different tools called; step count; cost; latency | Near zero |
| Threshold checks | Steps within 20% of before; output length within 20%; cost did not rise | Near zero |
| Cheap classifier or small judge | Is the final answer the same? Is it more verbose, yes or no? | Low |
| Strong judge | Is it better? Is the writing good? | High, and hard to define |
- "Is the answer the same?" is Ben's go-to eval. It is cheap, and the traces that fail it are the ones to read.
- Bug-fix check: replay the bad traces and ask "does the new output still show the issue?" Any yes means the fix did not work.
- Prefer binary checks over 1 to 5 scales.
- Set pass-rate thresholds, for example "not more verbose in 80% of traces," rather than pass/fail per trace.
- Escalate disagreements. Raindrop runs a cheap classifier, checks it against past labels, and sends only disagreements to a stronger model.
In the paper, the judge checks each response against misbehavior criteria (deception, reward hacking and so on) rather than comparing to the old response.
Validating the simulator
The replay setup has to be tested before its results mean anything. There are three checks.
- Main equals main. Replay production traces with the current, unchanged code. The output should match production. Raindrop calls this the world-building phase and runs it before testing any PR.
- Blind discriminator. Show an LLM the original trace and the simulated one without labels, and ask which is simulated. Over many pairs, if it does better than chance, the simulation is giving itself away. Ben credits the OpenAI paper for this.
- Watch for eval awareness. Check whether the agent says or acts like it knows it is being tested. Examples Ben gives: "this isn't working because it's a simulation, so I'll answer like this," or "the user is probably testing my politeness." Either one invalidates the result.
Common giveaways:
- Empty environments: no git history, no issues, no past conversations.
- Data dated after the trace's timestamp.
- Tool results that do not match what the agent just asked for.
Why it matters, from the paper: on hand-built alignment evals, models reasoned about being watched or graded 62 to 76% of the time, versus 0.12% in production. In simulated deployment, the rate of the model noticing it was being tested came within 2.6x of production.
Multi-turn and user simulation
Both sources simulate the environment, not the user. Replays stop after one user turn.
| Kind of multi-turn | Works? | Notes |
|---|---|---|
| Cutting deep into a conversation (20 messages of history, then the next turn) | Yes | Still one new turn, just with long fixed history |
| Agent's own steps within a turn (tool calls, sub-agents) | Yes | This is what world simulation is for |
| Continuing past the cut, with the user responding | Not reliably | Requires predicting what the real user would say |
A goal-driven simulated user (as in τ-bench) helps answer "can the agent still complete this task?" It is weak for predicting real user experience, for these reasons:
- Simulated users are too cooperative and clear. Real users get confused, impatient, change direction, or hide their goal until late.
- The simulator knows the goal from hindsight and tends to volunteer information the real user never gave. That makes the agent look better than it is.
- It adds a second source of variance. A different result could be the change or the simulated user.
- An overly tidy user is another signal that the agent is being tested.
The paper also stops at one regenerated turn.
Pitfalls checklist
The first five come from Ben's list. The last three come from the paper or from gaps in the talk.
- Replaying against live systems (state has changed, and it is unsafe)
- Serving cached tool responses (the agent notices)
- Replaying only failures (overfits; mix in a random sample)
- Running each trace once (outputs vary; replay several times)
- Not reading the simulated traces yourself
- Traces missing context needed to rebuild the branch point (system prompt, tool schemas, retrieved content)
- Trusting the judge without checking it against labeled examples
- Stale traces: usage shifts over time, so the replay set drifts from current traffic
What the paper adds
The talk gives the practice. The paper gives evidence that replaying real traffic predicts production better than hand-built evals.
| Result | Simulated deployment | Hand-built evals |
|---|---|---|
| Predicted direction of change (misbehaviors moving 1.5x or more) | 92% correct | 54% correct |
| Predicted vs actual production rate | Pearson r = 0.91 (log scale) | Not reported |
| Test awareness vs production | Within 2.6x | 18 to 19x higher |
| Using public data (WildChat) instead of private traffic | Mean error under 3x | Not compared |
The hand-built baselines were:
- Challenging Prompts: real production prompts that caused failures before. This is the main baseline. It loses because the set is cherry-picked, not because the prompts look fake.
- Alignment evals: scheming and reward hacking, for example impossible coding tasks.
- Benchmarks: SWE-Bench, AIME, and two internal ones (OpenAI-Proof QA, MonorepoBench).
Grading: GPT-5 Thinking judges labeled about 1.3M conversations. The paper checked its forecasts against real traffic after launch.
Open questions and caveats
- Judge reliability is barely covered. Neither source says much about how stable the "same outcome" or misbehavior judge is across repeated runs. Grader noise feeds straight into every rate.
- 5 to 10 traces is a sanity check, not a measurement. It catches outright breakage but not a failure rate going from 2% to 5%.
- "Same as before" is not "correct." Consistency checks against unverified traces lock in whatever the old behavior was, bugs included.
- Vendor framing. Claims like "one of the hardest problems in the world" and "first to market" are sales talk. The concept is old; the value is in the world-building quality.
- Contamination. Eval scenarios and methods are public and in training data, and anti-scheming training has been shown to raise test awareness. Real traffic is harder to learn as a pattern, but only while the traces stay real.
- Tail risks. Replay only covers behavior that current traffic can trigger. Rare or new failure modes need other methods.
- Trace completeness. Most trace stores (for example OpenTelemetry setups) do not capture everything needed to rebuild the branch point exactly.
References
- Predicting LLM Safety Before Release by Simulating Deployment (arXiv 2607.07184), Williams, Sheahan, Raymond, Korbak et al., OpenAI, July 2026
- OpenAI blog: Predicting model behavior before release by simulating deployment
- LessWrong discussion of the paper
- "Getting Started with Simulations," Ben Hylak (Raindrop), guest session in Hamel Husain's evals course
- Related prior work: Twitter Diffy (2015, traffic replay diffing); ToolEmu (Ruan et al., 2023, LLM-emulated tools); τ-bench (goal-driven simulated users)