Bryan Falkowski All writing

Agent Simulation via Trace Replay: Raindrop Approach + OpenAI Paper

Bryan Falkowski · September 23, 2026 · Based on Predicting LLM Safety Before Release by Simulating Deployment (OpenAI, 2026)

Simulation means replaying real production traces against a proposed change to an agent, then comparing what changed. You take a captured trace, cut it at a point, let the changed agent continue from there, and score the result with deterministic diffs and a judge.

The question it answers is "what does my change change?" The change can be a new model, an added or removed tool, a prompt edit, or a harness swap. Raindrop uses this as a pre-merge regression check. OpenAI uses the same mechanic to forecast how often misbehavior will happen after a model release.

Sources and how they relate

The two sources use the same replay mechanic for different purposes. Ben Hylak (Raindrop CTO) cites the OpenAI paper as precedent and borrows its tool-simulation and realism-check ideas.

Raindrop talk (Hamel Husain's evals course)OpenAI paper, July 2026
PurposeRegression check on each changeForecast misbehavior rates before release
What changesAnything: model, tool, prompt, harnessThe model
Judge asksIs the outcome the same? Did the known bug recur?Does this response show misbehavior X?
ReferenceOld trace (good, bad, or just current behavior)Misbehavior criteria, not the old response
Scale5 to 50+ traces, picked to touch the change~1.3M conversations graded
Validated againstReplaying main should equal mainReal production rates after launch

The idea itself is not new. It is traffic replay testing (for example Twitter's Diffy, 2015) and snapshot testing, adapted to agents. The agent-specific hard parts are simulating the environment and keeping the agent from noticing it is being tested.

The core loop

Capture a trace, cut it, let the changed agent continue, and compare. Raindrop's full cycle is log, detect issues, simulate the fix, ship, repeat.

flowchart LR
  A[Production traces] --> B[Select traces
that touch the change] B --> C[Cut at branch point] C --> D[Replay with change
in simulated world] D --> E[Diffs + judge
vs old trace] E --> F[Review differing traces] F --> G[Ship or fix]
  1. Capture. Keep full traces from production: user messages, system prompt, tool schemas, tool calls and results, retrieved content. Missing context here breaks the replay later.
  2. Cut. Pick the branch point, usually right before the agent turn your change affects. Everything before it is fixed input.
  3. Replay. Run the changed agent from that point. It can take as many internal steps (tool calls, sub-agents) as it needs within that one user turn.
  4. Run several times. Outputs vary run to run, so replay each trace a few times and look at rates.
  5. Compare. Cheap diffs first, then a judge where exact comparison does not work.
  6. Look at the traces. Read the ones that changed. Ben says this tells you more than the eval suite.

Choosing traces

Which traces you pick matters more than how many. A thousand replays tell you nothing about a removed tool if none of them called that tool.

ChangeTraces to pickRough count (Ben)
Remove or change one toolTraces where that tool was called5 to 10
Fix a known bugThe bad traces that showed the issue, plus a random sampleEnough to cover the failure
Prompt or model swapRandom sample plus targeted edge casesDozens
Harness rewriteBroad random sample50+

Building the world

The hardest part is recreating the environment the agent saw when the trace ran. Neither of the obvious shortcuts works.

What Raindrop does (the "Swiss cheese" model):

  1. Run real code as far as possible. They check out the customer's agent into a sandbox and run it unchanged. Deterministic parts, like an internal SQL dialect parser, run for real. They also spin up sandboxes if the agent uses them.
  2. Simulate at the network boundary. Past that point an LLM generates tool responses.
  3. Seed from the original trace. Each original tool result is a window into the world at that time. The simulator fills the holes between those windows so the answers stay consistent.
  4. Add real data when needed. For a brand-new data source, users can drop in JSON files or connect a read-only MCP (for example Supabase) so the world builder can pull real schemas and data.

The paper confirms how much this matters. Plain simulated tool outputs were picked as the real one only 11.6% of the time in a blind comparison. Giving the simulator richer context raised that to 49.5%, close to the 50% that means indistinguishable.

Evals over traces

An eval here is a pure function over traces: f(new trace, old trace) returns pass or fail. Once defined, it runs on any trace, unlike input/output pairs that have to be collected one by one.

Raindrop layers it this way: simulation is the base infrastructure, anomaly detection flags anything weird, and evals are the scores you define and track.

Start deterministic, then move up to judges.

LevelExamplesCost
Deterministic diffsFinal answer changed; new tool errors; different tools called; step count; cost; latencyNear zero
Threshold checksSteps within 20% of before; output length within 20%; cost did not riseNear zero
Cheap classifier or small judgeIs the final answer the same? Is it more verbose, yes or no?Low
Strong judgeIs it better? Is the writing good?High, and hard to define

In the paper, the judge checks each response against misbehavior criteria (deception, reward hacking and so on) rather than comparing to the old response.

Validating the simulator

The replay setup has to be tested before its results mean anything. There are three checks.

  1. Main equals main. Replay production traces with the current, unchanged code. The output should match production. Raindrop calls this the world-building phase and runs it before testing any PR.
  2. Blind discriminator. Show an LLM the original trace and the simulated one without labels, and ask which is simulated. Over many pairs, if it does better than chance, the simulation is giving itself away. Ben credits the OpenAI paper for this.
  3. Watch for eval awareness. Check whether the agent says or acts like it knows it is being tested. Examples Ben gives: "this isn't working because it's a simulation, so I'll answer like this," or "the user is probably testing my politeness." Either one invalidates the result.

Common giveaways:

Why it matters, from the paper: on hand-built alignment evals, models reasoned about being watched or graded 62 to 76% of the time, versus 0.12% in production. In simulated deployment, the rate of the model noticing it was being tested came within 2.6x of production.

Multi-turn and user simulation

Both sources simulate the environment, not the user. Replays stop after one user turn.

Kind of multi-turnWorks?Notes
Cutting deep into a conversation (20 messages of history, then the next turn)YesStill one new turn, just with long fixed history
Agent's own steps within a turn (tool calls, sub-agents)YesThis is what world simulation is for
Continuing past the cut, with the user respondingNot reliablyRequires predicting what the real user would say

A goal-driven simulated user (as in τ-bench) helps answer "can the agent still complete this task?" It is weak for predicting real user experience, for these reasons:

The paper also stops at one regenerated turn.

Pitfalls checklist

The first five come from Ben's list. The last three come from the paper or from gaps in the talk.

What the paper adds

The talk gives the practice. The paper gives evidence that replaying real traffic predicts production better than hand-built evals.

ResultSimulated deploymentHand-built evals
Predicted direction of change (misbehaviors moving 1.5x or more)92% correct54% correct
Predicted vs actual production ratePearson r = 0.91 (log scale)Not reported
Test awareness vs productionWithin 2.6x18 to 19x higher
Using public data (WildChat) instead of private trafficMean error under 3xNot compared

The hand-built baselines were:

Grading: GPT-5 Thinking judges labeled about 1.3M conversations. The paper checked its forecasts against real traffic after launch.

Open questions and caveats

References