Bryan Falkowski All writing

Is This Run Valid? Attributing Pipeline Failures Before Scoring LLM Evaluations

Bryan Falkowski · October 2026 · Code and data: github.com/bfalkowski/eval-run-validity

Abstract. An eval score only means something if the run that produced it worked. Most eval pipelines report a number either way, so a rate-limited API, a changed prompt or a scorer bug shows up as a model regression. I propose checking whether a run is valid before scoring it: fingerprint the run, run preflight canaries, trace every item with OpenTelemetry, assign each failed item a cause, and gate the run as valid (publish the score), degraded (publish the score on the remaining items and list what was excluded) or invalid (publish no score). I tested this on a small support-agent eval built on Jev, a small decision model from TypeSafe, with a fault service (faultd) that can break any part of the eval and a GitHub Actions pipeline that can break itself: 238 runs on the real API, for about $1.05.

Three results stand out. An older commit of the agent code scored at or above the best clean run, and only the fingerprint showed that the wrong code had been tested. When the agent silently swallowed failed calls, the plain score hid up to 23 points of damage, and the gate still caught every case from the traces. And asking the same LLM judge twice missed most of its mistakes: in this eval the judge was wrong on a quarter of items, but its two calls disagreed on only 3.5% of them.

Of 164 faulted runs, 72 (44%) scored low enough that a plain score comparison would have reported a model regression. Another 84 scored inside the clean range, and 36 of those had damaged items that only the gate found. The gate flagged every run whose faults crossed its thresholds. Attribution named the right cause for every affected item, but I wrote both the faults and the rules, so this shows the rules match the taxonomy, not that they would catch faults nobody planned for. The paper ends with the gate as a working CI pipeline.

In practice: add a validity gate between running an eval and publishing its score. Check a fingerprint of the run against the baseline, check that every item has a successful model and judge call, and check that results were collected once, from one run. Section 7 gives the stages.

1. The problem

While running the experiments for my previous paper, several things broke that had nothing to do with the models being tested. TypeSafe's API returned 503 on 26 of 40 calls in one pilot (TypeSafe makes Jev, a small model that answers typed questions, such as picking one option from a list or answering yes or no, with a confidence score). Claude Opus 5.5 rejected forced tool calls, and the Anthropic SDK version in use no longer accepted a temperature parameter. One task asked the agent to set an order to Shipped when it already was, so there was nothing correct to do. The scorer counted paging through a table as a false action. Each of these was found by hand, after the fact, by reading raw results and error messages; no check in the pipeline caught them. Scored the way many pipelines score, with a failed call counted as a wrong answer and no look at the errors, several of them would have lowered a model's score with no sign that the run itself had gone wrong.

This is the normal state of LLM evaluation pipelines. A score comes out whether the run worked or not. When the score drops, someone has to work out whether the model got worse, the judge broke, the dataset changed, or the infrastructure had a bad hour. That triage is slow, and when it is skipped, wrong conclusions ship.

This paper asks four questions:

  1. How often does a pipeline fault produce a false regression, or a false improvement, in a naive score?
  2. Can each failed item be attributed to the right cause automatically, from traces and run metadata alone?
  3. Where should a validity gate's threshold sit, trading bad runs caught against good runs wrongly rejected?
  4. Once invalid items are excluded, how close is the remaining score to a clean run's score?

2. Related work

Flaky tests. Software testing has studied this problem for a long time. Luo et al. (2014) built a root-cause taxonomy of flaky tests from fixed issues, and Google's testing blog (2016) described rerunning and quarantining flaky tests at scale. A 2023 multivocal review covers causes, detection and responses, and work on systemic flakiness shows that flaky failures often co-occur from one shared cause, which is close to the run-level view here. I borrow the taxonomy-first approach and the idea that a failure is not automatically a verdict on the code under test. The difference is that an eval produces a score, not a pass or fail, so a fault moves a number instead of turning a test red.

Infrastructure noise in LLM evals. Anthropic (2026) found that resource settings alone moved Terminal-Bench 2.0 scores by about 6 percentage points, with infrastructure error rates falling from 5.8% to 0.5% as limits were raised. That is direct evidence for the problem. It studies one kind of fault on one benchmark; this paper injects six kinds and tests attribution and a gate.

Reproducible evaluation. Biderman et al. (2024), from the lm-evaluation-harness team, catalog how small setup details change results and argue for sharing exact configurations. The run fingerprint here makes that comparison automatic and per run.

Benchmark auditing. BenchGuard (Tu et al., 2026) uses LLMs to audit agent benchmarks for defects in instructions, ground truth and evaluation scripts, and found 12 confirmed defects in ScienceAgentBench. Audits like this check a benchmark once, before use. This paper checks each run as it happens, and covers infrastructure and judge faults as well as dataset defects.

Statistics of eval scores. Miller (2024) sets out how to put error bars on eval scores and compare models properly. That work assumes the run is valid; this paper checks that first.

Tooling. The OpenTelemetry GenAI semantic conventions give standard attributes for model calls, which this paper uses and extends. The eval harnesses I looked at record logs and configurations. I did not find one that issues an explicit validity verdict per run, but I did not survey them systematically.

Consistency of LLM judges. Haldar and Hockenmaier (2025) show that LLM judges often give different scores to the same input across runs, which makes a single judge score unreliable. Section 6.9 looks at the reverse case: a judge that gives the same answer twice and is wrong both times. Ding (2026) finds that agreement among a model's samples correlates only weakly with correctness, and that the most self-consistent model in that study was among the worst calibrated. Verga et al. (2024) replace one large judge with a panel of smaller models from different families. A panel of different models is one way to get a check that does not come only from the judge; this paper does not test it.

3. A taxonomy of failures

Six causes. Only the last should count against the model. The right-hand column is what actually happened in my previous paper.

CauseWhat it looks likeSignal that detects itSeen in paper 1
InfrastructureTimeouts, restarts, rate limits, outages, auth failuresError status, HTTP codes and retries in traces503 on 26 of 40 pilot calls
Config driftModel, prompt, context, code or dependencies changed between runsRun fingerprint differs from the baselineForced tool calls rejected; SDK dropped temperature
Judge failureUnparseable verdicts, verdicts that change on repeatParse errors; disagreement between judge repeatsNot seen
Task or dataset defectThe expected answer is wrong, the record is missing, items are duplicatedExpected answer inconsistent with its fixture; lookup fails; duplicate IDsA task asked to set an already-shipped order to Shipped
Harness or scorer bugThe code that scores the output is wrongScorer's view disagrees with the trace; scorer fails known-answer casesPaging scored as a false action
Model failureThe model got a valid item wrong in a valid runNone of the aboveWhat we want to measure

A run where only some items finished is a state of the run, not a cause, and has its own rule in the gate.

4. The run validity framework

Preflight canary checks Fingerprint run manifest Traced run one span per call Attribution a cause per item Gate valid degraded invalid a failed preflight stops the run before it spends anything faultd (experiments only) breaks calls, data, scorer, config ground-truth log what was injected never read by attribution or the gate
Figure 1. The five steps around every eval run. The dashed parts exist only for the experiments in this paper.
  1. Preflight. Before any scored item: data files match their checksums, Jev answers a trivial canary question, the judge answers a trivial yes/no canary, and the scorer gets twelve known-answer cases right. A failed preflight stops the run.
  2. Fingerprint. A manifest records the model, hashes of the policy prompt, the decision and judge instructions, the context fields sent, the dataset, the agent code and the scorer code, plus package versions, lockfile hash and Python version. Fields are grouped by how much a change matters: a critical field that differs from the baseline without being declared makes the run invalid; a warning field makes it degraded.
  3. Traced run. Every item is an OpenTelemetry trace. Every Jev call and tool call is a span, written to disk as soon as it ends, so a run killed part way still leaves its traces.
  4. Attribution. Rules over each item's spans, its record and the run's fingerprint assign one cause. Items with any cause other than model failure are excluded from the score and listed.
  5. Gate. The run is invalid if preflight failed, a critical field drifted, results from another run are mixed in, fewer than 90% of items finished, or more than 15% of items were excluded. It is degraded for more than 2% excluded, a partial run above 90%, a warning-level drift, or a withheld judge score. Otherwise it is valid. These thresholds are starting values, frozen on a development split before the held-out runs are scored.

The three verdicts decide what gets published. A valid run publishes its score. A degraded run publishes the score on the items that were not excluded, with the excluded items and their causes listed beside it. An invalid run publishes no score. In the results, a run is flagged when it is degraded or invalid.

Each check answers a narrow question. Passing all of them does not make a run correct; it removes the failure causes the checks were built for.

CheckWhat it detectsWhat it does not establish
PreflightA dependency that is down, a scorer that fails known-answer cases, data files that changedThat the model and judge will keep working for the whole run, or that the scorer is right outside its twelve known cases
FingerprintAn undeclared change to the model name, prompts, context fields, dataset, code or environmentThat the model behind a name is unchanged: an alias such as jev-latest can point to a new model with the same name
IntegrityPartial runs, items counted twice, results from another run or configurationThat the items that did finish were handled correctly
CoverageItems with no successful model call or judge call, including failures the agent code swallowedThat a successful call returned a sensible answer
AttributionA cause for each failed item, from its spans, its record and the fingerprintThat an item left as a model failure really is the model's fault; a fault that leaves no signal in the trace is counted against the model
Judge repeatJudge answers that change between two callsThat the judge is right (Section 6.9)
Score on valid itemsThe score with the excluded items removedThat the remaining tickets are as hard as the full set; excluded tickets may not be a random sample (Section 6.5)

Table 1. What each check detects and what it leaves open.

Span attributes

AttributeOnUsed for
gen_ai.system, gen_ai.request.model, gen_ai.operation.name, gen_ai.usage.input_tokensJev callsStandard GenAI conventions; cost
http.response.status_code, error.typeJev and tool callsInfrastructure versus other causes
jev.attempts, jev.parse_ok, jev.latency_sJev callsRetries absorbed; unparseable answers
eval.roleJev callsSeparates the agent's decision from the judge
tool.name, tool.statusTool callsWhat the agent actually did, independent of the scorer
eval.item_index, eval.ticket_id, eval.run_idAllGrouping spans by item and run
eval.item.consistentItemExpected answer still matches the fixture

Nothing in a span says whether a failure was injected. The transport records a failed call even when the agent code swallows it, which is how silent fallbacks are caught (Section 5.4).

5. Experimental setup

5.1 The eval under test

A support agent works 60 customer tickets against a local orders service. For each ticket it looks up the order and the stock, asks Jev which action a five-rule support policy calls for (refund, reship, escalate, or reply only), takes that action and replies. The policy has edge cases on purpose: refunds over $500 must be escalated, damage reported more than 30 days after delivery is escalated, and a lost order is reshipped only if the item is in stock.

The first dataset was too easy. On 60 straightforward tickets Jev scored 60 of 60 on the real API. A perfect score leaves nothing for the gate to separate: no natural model failures, and no spread between clean runs. I wrote a second set of 60 tickets under the same policy, each hard in one deliberate way, and used it for every result below.

Kind of difficultyTicketsExample
Indirect wording or a distracting order number18"The box says webcam but inside is a ring light."
Customer asks for an action the policy does not give13A lost order, in stock, "just refund me": the policy says reship.
The $500 refund limit10Exactly $500 is refunded; an $899 lost order is reshipped, because the limit applies to refunds only.
Claim disagrees with the record9"Got it last week and it's broken" on an order delivered 45 days ago.
Dates exactly on a cutoff6Delivered exactly 30 days ago; shipped exactly 14 days ago.
Angry about an order that has not shipped4"Taking FOREVER, refund me now" on a Processing order.

Table 2. Dataset v2. Expected actions are 17 refund, 17 reply, 13 escalate and 13 reship, generated by a reference policy function.

After the first clean run on this set, I read every miss. Two of the eight were arguable readings of the policy text rather than model errors ("within the last 30 days" at exactly 30 days, and whether the "never arrived" rule covers a customer asking where the order is). I tightened the wording of those two rules rather than the labels, and every result below uses the tightened policy.

The score used throughout this paper

A ticket is correct when the side effects the agent caused in the orders service (refund, reship, escalate) are exactly those of the expected action. A reply-only ticket is correct when there are no side effects. Correctness is decided by a deterministic scorer; the judge has no part in it.

The plain score is the share of all 60 tickets that are correct. The score on valid items is the same share over the tickets left after the gate excludes items attributed to a cause other than the model. Every score, regression and band in this paper is one of these two, unless it is labeled as the judge score.

A judge asks Jev twice per ticket whether the resolution follows the policy. Its votes form a separate judge score, which is a diagnostic: it lets the judge-failure category be exercised, and it can be checked against the deterministic score. Jev answers through TypeSafe's typed choice and noul questions. Every decision in the system, the agent's and the judge's, is made by Jev. I chose one cheap model on purpose: the subject of this paper is the pipeline, not the model.

Ticket + order, stock Agent Jev picks the action Orders service side effects + reply Scorer deterministic Score primary result Judge Jev, asked twice Judge score diagnostic only compared (6.9)
Figure 2. How one ticket is scored. The deterministic scorer produces every score in the results. The judge runs alongside it and is used only as a diagnostic.

Paper 1's harness was built to be stable, so faults could not be switched on cleanly there. This repo builds its own agents and routes every point of failure through one fault API.

5.2 The fault API

faultd is a small HTTP service. Every component asks it, call by call, whether to misbehave: Jev calls, judge calls, each tool, the fixture loader, the dataset loader, the scorer and the run configuration. A fault has a kind, a target, a rate, an item range and a seed, so any faulted run can be replayed exactly. faultd writes what it injected to a ground-truth log. Attribution and the gate never read that log; a test fails if anything in the validity code imports faultd or names its log, and another test checks that a verdict is identical with the log deleted. Injected faults raise the same exceptions or return the same malformed bodies as real ones.

5.3 Faults inside a run

CauseFaults
InfrastructureJev 503, 429, timeout, connection refused, malformed body, out-of-range answer; tool 500; judge 503; Jev down for a window of tickets, down for the whole run, or flapping; slow responses
Config driftPolicy prompt edited (refund limit dropped), stock removed from the context, decision instructions reworded; each undeclared
Judge failureMalformed judge answers; judge verdicts flipped
Task or dataset defectStale fixture (order status changed), order record removed, ticket duplicated
Harness or scorer bugScorer compares against the next ticket's expected answer; service state not reset between tickets; an always-on scorer bug
ControlStock removed from the context, declared in the manifest. A real regression that the gate should let through.

Table 3. Item-level faults. Per-call faults run at two doses, 5% and 25% of calls.

5.4 Wrong key and silent fallbacks

One fault is real rather than simulated. For part of a run the agent sends an invalid key, and TypeSafe returns a real 401; then it switches to the correct key. This is run under two error policies. With a loud policy, the failed call raises and the item is marked errored. With a silent policy, the agent falls back to replying only and the judge votes "fail". Silent fallbacks are the realistic bug: the items look like ordinary wrong answers. The check that catches both is coverage. Every ticket must have a successful decision span and successful judge spans; a ticket without them is a missed eval, not a wrong answer. Comparing the traces of a real 401 with a faultd 401 also tests whether the simulated faults look like real ones.

5.5 Faults in the pipeline

CausePipeline faultHow it is injected
InfrastructureRunner killed mid-runA step kills the run process
InfrastructureJob timeoutThe run step is cut off after a few seconds
InfrastructurePartial artifact uploadTraces are dropped before upload
Config driftDependency driftOlder OpenTelemetry versions installed instead of the lockfile's
Config driftPython version driftRunner set up with a different Python
Config driftMissing environment variablePOLICY_VERSION unset, so the code falls back to a stale default prompt
Config driftWrong commitAn older commit of the agent code is checked out
Harness bugResults merged twiceThe collect step appends a run's items to itself
Harness bugRuns mixedThe collect step merges two runs into one folder

Table 4. Pipeline-level faults. These never appear in an agent trace; they test the run-level checks.

5.6 Runs

All paper runs execute in GitHub Actions on Python 3.11 with pinned dependencies, against the real Jev API, on October 2, 2026. Each run is about 180 Jev calls. The baseline fingerprint is a clean run on the final policy. A local stand-in for Jev is used only for the repo's tests.

Ground truth for each item is rebuilt from faultd's log alone. A Jev transport fault only counts when all three attempts of a call were hit; otherwise the retry absorbed it and the item was not affected. A scorer off-by-one only counts when the neighbouring expected answer differs, a missed state reset only when an earlier side effect carries over, and a stale fixture only when the changed order no longer fits the expected answer. A run's expected verdict uses the gate's own thresholds on the share of affected items, so the comparison asks whether the gate saw what really happened, not whether the thresholds are well chosen.

In total 238 runs called the real API: 3 baseline runs while the dataset and policy settled, 5 noise-floor runs, 86 development, 86 held-out, 34 silent and 24 pipeline. The 48 collect-step copies made no API calls. Spend was $1.03 for the 235 runs whose traces were kept (24.5 million input tokens), plus about $0.02 for the three runs whose traces were dropped on purpose by the lost-upload fault. There were no separate pilots on the real API; plumbing was tested against a local stand-in.

BatchRunsWhat was injectedUsed in RQ1 to RQ3Used in RQ4Plain score exists
Baseline3NothingThe last one is a clean referenceNoYes
Noise floor5NothingClean referenceNoYes
Development8641 fault plans, a clean run and the control, twice82 faulted runs, plus 2 clean, 2 control and 2 noise-floor clean68 runs with item-level faultsYes, except 4 stopped at preflight
Held-out86The same plans with new seeds, twiceSame as development68 runs with item-level faultsYes, except 4 stopped at preflight
Silent fallback3416 infrastructure, judge and wrong-key plans and a clean run, silent policy, twiceSection 6.6 only32 runs, in Table 10 onlyYes
Pipeline247 pipeline faults and none, three timesSection 6.7 onlyNoYes; killed and timed-out runs score only the items that finished
Collect copies48Each pipeline run merged twice, and mixed with another runSection 6.7 onlyNoYes, over the merged records; no API calls
Reference pipeline4One fault per runSection 7.1 onlyNoNot counted in the 238

Table 5. Where every run is used. RQ1 to RQ3 use the 82 faulted development runs and the 82 faulted held-out runs (164). RQ4 uses the runs whose faults act on items (infrastructure, judge, task and harness) and that produced a score on an unchanged dataset; Table 10 gives the exact counts.

5.7 Metrics

6. Results

The results draw on two kinds of evidence, and they support different claims. Injected faults come from faultd and the pipeline workflow, and each has an answer key; they test whether the checks catch the causes they were built for (Sections 6.2 to 6.7). Natural incidents happened without injection: Jev's run-to-run variation (6.1), a change in policy wording the gate was not built for (6.1), a rounding bug in the gate and a bug in my answer key found on held-out runs (6.3, 6.4), and the judge's own errors (6.9). The wrong-key 401 sits between the two: a real failure from TypeSafe's API, triggered on purpose.

6.1 Clean runs

Ten clean runs scored a mean of 83.0% (standard deviation 2.2 points, range 80.0% to 88.3%, or 48 to 53 tickets of 60). No clean run needed a single retry. Five tickets changed answer between clean runs: four sit near the 30-day cutoff and one is shipped exactly 14 days ago. Jev is not fully deterministic near a boundary, so a run-to-run swing of two or three tickets is normal, and a plain score comparison cannot see a fault that costs fewer.

The misses in the baseline run fall into three groups. Five are the customer's word winning over the record or the policy: reshipping an order the customer says is lost when tracking shows it shipped five days ago, or refunding a lost order the policy says to reship because the customer asked for money back. Four are orders delivered 31 to 35 days ago that Jev refunds instead of escalating. One is a $649 lost order, out of stock, that Jev refunds without applying the $500 limit.

The judge. Across the 600 items in the ten clean runs, the judge's first call disagreed with the deterministic score on 24.5% of items, but its two calls disagreed with each other on only 3.5%. Section 6.9 looks at what that means for checking a judge.

A natural config drift. Tightening the two policy rules is itself a test the gate was not built for. The first run after the change scored 50 of 60 against 52 before, a difference inside the noise floor. But eight tickets changed answer: three were fixed and five broke, mostly orders delivered 31 to 35 days ago that Jev now refunded instead of escalating. The gate marked the run invalid with "undeclared change from baseline in policy_hash". This is a run-level verdict. With an undeclared change the gate does not try to say which failures the change caused, and it labeled all ten failures as config drift, although comparing with the previous run only five were new. A plain score comparison would have called it noise.

6.2 How often faults look like regressions (RQ1)

A faulted run counts as a false regression when its plain score falls more than two standard deviations below the clean mean, below 78.6%. This cutoff is an operational choice for this study, not a significance test. The ten clean runs scored a mean of 83.0% with a sample standard deviation of 2.19 points, and one ticket is worth 1.67 points, so 83.0 − 2 × 2.19 = 78.6% means a run has to lose about three tickets more than an average clean run before it reads as a regression. The cutoff depends on this eval and these ten runs; with another eval or more clean runs it would move. Appendix A lists the clean scores.

A faulted run is not automatically a bad run. A Jev error that a retry absorbs leaves no affected item, and a fault that lands on one ticket of 60 stays under the gate's 2% threshold. Runs like these should pass, and the answer key (Section 5.6) says so. That is why the last column of Table 6 is below the number of faulted runs.

Injected causeRuns per batchFalse regressions, devFalse regressions, held-outFlagged by the gate, dev / held-out (after the fix)
Infrastructure46212032 / 24
Config drift6646 / 6
Task or dataset defect124510 / 11
Harness or scorer bug10578 / 9
Judge failure8005 / 4
All faulted runs8236 (44%)36 (44%)61 / 54

Table 6. No faulted run produced a false improvement. Judge faults never move the agent's score, which is why they never look like regressions. The held-out gate counts use the gate after the rounding fix described in Section 6.4. With the gate as frozen, 57 held-out runs were flagged: the same 54 plus 3 that should have passed.

Dot plot of every development and held-out run: plain score on the horizontal axis, injected cause on the vertical axis, each dot marked valid, degraded or invalid by the gate. Clean and control runs are all valid. Runs far below the clean band are all invalid. Runs inside the band are a mix, depending on whether the fault actually affected items.
Figure 3. Every development and held-out run (168 plotted; 8 more stopped at preflight and have no score). Shaded: the clean mean ± 2 standard deviations. Many faulted runs sit inside the clean band, where a plain score comparison sees nothing; the gate still marks most of them degraded. The control sits just below the band and passes, as a declared change should.

Across both batches, 72 of the 164 faulted runs (44%) scored below 78.6%. A team comparing plain scores would have read each of them as the model getting worse. Of the other 92, eight were stopped at preflight and never produced a score, and 84 scored inside the clean band, where a plain comparison reports nothing. That is not the same as being fine: the gate flagged 36 of those 84 because their faults had damaged items, and passed 48 where retries absorbed the faults or too few items were touched to cross the 2% threshold. So without a gate, 44% of faulted runs would have been given the wrong explanation, and another 22% (36 of 164) carried damage that no plain comparison could see.

At a 5% dose most Jev transport faults are absorbed by retries and the run is valid, which is why the gate flags 32 of 46 infrastructure runs on dev and 24 of 46 on held-out. Of the 72 false regressions, the gate marked 71 degraded or invalid. The two counts are consistent: a run should be flagged only when its faults affected more than 2% of items or changed the run itself, and the one false regression the gate passed was not in that group. That run (a scorer off-by-one on held-out) had a single affected item, under the 2% threshold, and still scored 79.7% on its valid items: that run was mostly Jev having a low run.

6.3 Attribution (RQ2)

Every cause in this table is an injected fault whose trace signal I designed. The 100% figures show that the rules are consistent with the taxonomy. They do not measure accuracy on faults nobody planned for.

CauseAffected items, devAffected items, held-outRecallPrecision
Infrastructure392372100%100%
Task or dataset defect7181100%100%
Harness or scorer bug4548100%100%
Judge failure (judge score)10813494% / 92%not computed

Table 7. The first three rows are attribution for the agent's score; config drift is judged per run, not per item. Unaffected items wrongly excluded from the agent's score: 0 of 4,194 (dev) and 0 of 4,217 (held-out, after the answer-key fix below). The item counts come from the 78 runs in each batch that scored items: every run except the six config-drift runs and the four stopped at preflight, so the clean and control runs add to the unaffected items. That gives 78 × 60 = 4,680 items, plus the extra copies in the duplicated-ticket runs, where each copy is scored as its own item (22 on dev, 38 on held-out), for 4,702 and 4,718 items in total. The judge row is attribution for the judge's score, recall on dev / held-out; precision is not computed because the gate also flags the judge's natural self-disagreement, which the answer key does not count as a fault.

On held-out runs scored exactly as frozen, one unaffected item was excluded as a harness bug. On inspection the gate was right and my answer key was wrong: two missed state resets in a row carried a side effect from two tickets back, and the ground truth only looked one ticket back. I fixed the answer key, not the gate.

For the judge score, injected judge failures were attributed correctly in 101 of 108 cases (dev) and 123 of 134 (held-out). Every miss was an item where both judge votes were flipped the same way, so the repeats agreed. That is the same blind spot as in Section 6.1.

The held-out batch does not change this. Its runs vary where faults land, not what kind of fault they are, and I wrote both the faults and the rules. Real incidents are messier: several causes at once, and signals that are weaker than the ones faultd leaves.

6.4 The gate (RQ3)

DevHeld-out, as frozenHeld-out, after fix
Runs that should be flagged that were flagged60 / 6054 / 5454 / 54
Runs that should pass that were wrongly flagged1 / 283 / 340 / 34
Verdict level matched exactly87 / 8885 / 8888 / 88

Table 8. Expected verdicts use the same 2% and 15% thresholds as the gate.

The held-out batch found a real bug. The judge exclusion rate was computed as one minus the share kept, so 9 of 60 came out as 0.15000000000000002 and tripped the "over 15%" rule. Three runs were marked degraded instead of valid. The fix computes the rate from counts. The single wrong flag in the dev batch is not a bug: a run with judge faults injected on 5% of calls also had the judge's natural self-disagreement, and together they crossed the threshold. The answer key counts only injected faults; the gate counts both, which is what it should do.

The thresholds. I expected the thresholds to trade bad runs caught against good runs rejected. They do not, because attribution never excluded an item in a run where no fault had affected anything. The group here is the same 168 runs as Table 10: dev, held-out and silent runs with item-level faults that produced a score on an unchanged dataset, which leaves out the config-drift runs, the eight stopped at preflight and the eight with duplicated tickets. In 116 of them a fault affected at least one item. In the other 52, no item in the agent's score was affected, either because retries absorbed every fault or because the fault only touched the judge. The share of items excluded behaves like this:

Flag a run when more than this share of items is excluded0%2%5%10%15%20%
Runs with affected items that are flagged100%90%66%59%47%39%
Runs with no affected items that are flagged0%0%0%0%0%0%

Table 9. Agent score only. The judge-score rule adds the one wrong flag discussed above.

So in this setup the thresholds are not about accuracy. They are a reporting policy: how many excluded items a team will accept before it stops publishing a score at all. At 2% a run with a single affected ticket out of 60 is still valid; at 0% it is marked degraded, and its report says so. The trade-off would appear in a pipeline where attribution makes mistakes, which this one, with faults whose signals I designed, does not.

6.5 Score on valid items (RQ4)

One caveat comes first. The score on valid items is computed on fewer tickets than a clean run. If the excluded tickets are not a random sample, the remaining set is easier or harder than the full one, and a score close to the clean mean does not make the two runs comparable. The task-defect row of Table 10 shows this happening.

Across the 136 dev and held-out runs with item-level faults, the plain score sat a median 3.0 points below the clean mean, and as far as 63 points below. After excluding the items the gate attributed to faults, the score on the remaining items sat a median 0.3 points above it, almost all of it inside the clean noise band. Measured as distance in either direction, the median gap fell from 3.0 points to 2.0 on dev and 1.6 on held-out, which is close to the clean runs' own standard deviation of 2.2. These medians and Figure 4 use the 136 dev and held-out runs. Table 10 adds the 32 silent-batch runs with item-level faults, 168 runs in all, so its means are over a slightly larger group.

Two rows of dots. Plain score: spread from 63 points below the clean mean up to a few points above, median 3.0 below. Score on valid items: clustered around the clean mean, median 0.3 above, with a few runs 10 to 20 points below.
Figure 4. Distance from the clean mean for runs with item-level faults, before and after excluding the items the gate attributed to faults. Black bars are medians; the shaded band is the clean runs ± 2 standard deviations. The runs still well below are ones where a third or more of the tickets were excluded, so the score is over a different, smaller subset: a wrong key on the last 20 tickets leaves the first 40, which happen to hold more of the hard cases. All of them were marked invalid, so no score was reported.
Injected causeRunsPlain score vs clean meanValid items vs clean mean
Infrastructure116−12.1 [−15.0, −9.5]−0.3 [−1.0, 0.3]
Harness or scorer bug16−7.3 [−10.2, −4.2]+0.5 [−0.6, 1.6]
Task or dataset defect16−5.6 [−8.3, −3.1]+2.0 [+0.9, +3.1]
Judge failure200.0 [−0.7, 0.6]0.0 [−0.7, 0.6]

Table 10. Mean points from the clean mean, with 95% bootstrap intervals over runs, pooled over the dev, held-out and silent batches. Runs with no score are left out: the four that stopped at preflight with Jev down (infrastructure: 46 + 46 + 28 − 4 = 116), the four with an always-on scorer bug, which preflight also stopped (harness: 10 + 10 − 4 = 16), and the eight duplicated-ticket runs, where the dataset itself changed (task defect: 12 + 12 − 8 = 16). Judge failure: 8 + 8 + 4 = 20.

For infrastructure and harness faults the valid-items score is indistinguishable from a clean run. For task defects it lands about two points above: the stale and missing records the gate excludes are not a random sample of tickets, so the tickets left over are slightly easier. Excluding bad items removes the fault's damage, but it can also change which tickets are being scored, and a report should say which were left out.

6.6 Silent fallbacks and the wrong key

With the silent error policy, a failed Jev call does not stop the item: the agent replies and takes no other action, and the judge votes "fail". In 34 silent runs the gate flagged all 30 that should have been flagged and none of the 4 that should not, with the exact verdict level matching on all 34. Attribution named the cause of all 381 affected items and wrongly excluded none of 1,659 unaffected ones. It can do this because the transport records the failed call in the trace even when the agent code swallows it, so a ticket with no successful decision call is a missed eval, not a wrong answer.

FaultPlain score, loudPlain score, silentGate, both policies
Jev returns 401 from ticket 30 on40.0%61.7%invalid
Tool 500 on 25% of calls26.7%50.0%invalid
Wrong key for the last 20 tickets48.8%63.3%invalid
Jev down for tickets 20 to 3957.9%70.8%invalid
Wrong key for the first 20 tickets57.5%58.3%invalid

Table 11. Mean plain score per policy (four loud runs, two silent). Clean runs score 83%.

Dumbbell chart for five faults. For each, the plain score with loud error handling is lower than with silent handling: tool 500, 27 to 50 percent; Jev 401 from ticket 30, 40 to 62; wrong key on the last 20 tickets, 49 to 63; Jev down for tickets 20 to 39, 58 to 71; wrong key on the first 20 tickets, 58 to 58. All are well below the clean band at 83 percent.
Figure 5. The same faults with loud and silent error handling. The silent fallback recovers up to 23 points of plain score without a single extra call succeeding. The gate marked every one of these runs invalid under both policies.

Silent fallbacks make a broken run look less broken. The default action, reply only, happens to be right for many tickets, so the plain score recovers up to 23 points without a single extra call succeeding. The three faults that cost a third of the decisions (Jev down for 20 tickets, a wrong key for the first or last 20) read as a 12 to 25 point drop under the silent policy, against 25 to 34 points under the loud one. Across the silent batch the median gap between the plain score and the clean mean was 8.8 points; on the items the gate kept it was 2.1.

Real and simulated failures. The wrong-key fault sends a real invalid key and gets a real 401 from TypeSafe; faultd's 401 is simulated. The two spans are identical in status, error type, HTTP status code, attempt count and events. The one difference is latency: about 150 ms for the real call and under 1 ms for the simulated one. Attribution does not use latency, so this does not change any result here, but it is a tell a reader could use, and a later version of faultd should add realistic delay.

6.7 Pipeline faults

Pipeline faultPlain scoreGateReason given
Runner killed mid-run67 to 82%invalid ×3partial run: 11 to 13 of 60 items finished
Job timeout70 to 80%invalid ×3partial run: 20 to 28 of 60 items finished
Traces lost in upload78 to 85%invalid ×3no traces found
Missing environment variable77 to 78%invalid ×3undeclared change in policy_hash
Older commit of the agent code88 to 90%invalid ×3undeclared change in policy_hash, agent_code_hash
Dependencies resolved to older versions82 to 87%degraded ×3environment differs in packages
Different Python version82 to 83%degraded ×3environment differs in python_version

Table 12. Pipeline faults on a clean eval. The ten clean runs score 83.0% (observed range 80.0 to 88.3%). Expected verdicts matched on all 21 runs.

None of these faults leaves a trace inside the agent, and several leave the plain score inside or near the clean range. Lost traces scored 78 to 85%, which looks like a normal run. Two are worth naming. A missing POLICY_VERSION made the code fall back to an older policy without the refund limit, and the run looked like an ordinary mediocre run. And checking out the previous commit of the agent code, which carries the older policy wording, scored 88 to 90%, at or above the best clean run (88.3%) and well above the clean mean. A plain score comparison would have reported an improvement. Only the fingerprint shows that the code under test was not the code anyone meant to test.

The collect step. The last two faults happen after the runs, when results are gathered. Merging a run's results twice leaves the plain score unchanged, because every ticket is counted twice, while the item count doubles and makes any interval look twice as tight as it should. Mixing two runs' results into one folder produces an average of two runs that never happened. The gate marked all 48 invalid: every item record carries its run ID and fingerprint hash, so a repeated item index or a foreign run ID is caught before any score is computed.

6.8 The control

The control removes stock levels from Jev's context and declares the change in the manifest. Its four runs scored 78.3%, 78.3%, 78.3% and 76.7%, about five points below the clean mean, and the gate marked all four valid: a real, declared regression goes through. The same change made without declaring it was marked invalid in all four runs, as were all twelve undeclared config-drift runs.

6.9 Repeated-judge self-consistency does not catch a consistently wrong judge

A cheap health check for an LLM judge is test-retest: call the same judge twice on the same item, with the same prompt, and see whether the two answers agree. That is the check tested here: two calls to one judge (Jev), not a panel of different judges or a larger sample. This eval has something most evals lack, a deterministic score for every item, so the check can be compared with a reference that does not come from the judge.

An example from the baseline run shows the failure. Ticket V005 is an order that shipped exactly 14 days ago and has not arrived. Rule 1 of the policy says to reply with the status and take no other action, and the agent did that, so the deterministic score is PASS. Both judge calls answered FAIL. The two calls agree, so a repeat check sees a stable judge, but the judge is wrong both times.

Clean runs, 600 itemsShare of items
Judge's first vote disagrees with the deterministic score24.5%
Judge's two votes disagree with each other3.5%
At least one vote is wrong26.3% (158 items)
Both votes wrong, in the same way22.8% (137 items)

Table 13. The judge against the deterministic score in clean runs.

The judge was wrong about one time in four. Of the 158 items where at least one of its two votes was wrong, the votes disagreed on 21 (13%), so a repeat check would have flagged about one in eight; on the other 137, both votes were wrong in the same way. Counted by the first vote alone, the judge was wrong on 147 items, and the second vote disagreed on only 10 of them (7%). The injected faults show the same thing from the other side. When faultd flipped judge votes, the gate caught every flip that left the two votes disagreeing, and missed every case where both votes were flipped the same way (7 items on dev, 11 on held-out).

Repeat agreement measures whether a judge is stable, not whether it is right. This matches what Ding (2026) reports for agreement among samples in general, and it is the other side of the inconsistency Haldar and Hockenmaier (2025) document: a judge can fail by changing its answer, or by giving the same wrong answer every time, and a repeat check only sees the first. These figures are for one judge, one prompt and one dataset. Another judge could be less consistent or more accurate, and the 3.5% and 24.5% should not be read as general rates. The claim is narrower: a two-call self-consistency check can pass while the judge is wrong on a quarter of items, so it cannot stand in for a reference. A judge used to make or check decisions needs a reference that does not come from the judge: a deterministic score where one exists, or a small set of human-labeled items scored on every run, with disagreement against that set tracked as its own metric.

7. Recommendation: a validity gate as a pipeline stage

The practical result of this paper is one stage to add to any eval pipeline. It sits between running the eval and publishing or comparing its score, and a score moves forward only if the run passes. Three of its checks need no model call and cost almost nothing, so they can be added to an existing CI job without changing the eval.

StageWhenWhat it checksWhat it caught here
PreflightBefore any paid callA canary question to the model and to the judge; known-answer cases for the scorer; checksums on the dataJev down for the whole run; an always-on scorer bug
FingerprintAt run startHashes of the prompt, policy, context fields, dataset, agent code and scorer; package and Python versions. Any undeclared difference from the baseline fails the run.A missing environment variable that loaded an old policy; an old commit that scored at or above the best clean run; every prompt and context change
CoverageAfter the runEvery item has a successful model call and a successful judge call in its trace, whatever the agent code did with the failureSilent fallbacks that hid up to 23 points of damage; a wrong key for part of the run
IntegrityWhen results are collectedCompleted items equal planned items, each item appears once, and every record carries its run ID and fingerprint hashKilled runners, timeouts, lost traces, results merged twice, two runs mixed together
ReportBefore publishingPublish the score on valid items with the list of excluded items and their causes, not the plain score. Compare two runs only if both passed.Brought faulted runs from a median 3.0 points off the clean mean to 1.6 to 2.0

Table 14. The gate as pipeline stages. Fingerprint, coverage and integrity need no model call.

One more recommendation applies to anyone using an LLM judge: do not treat agreement between two calls to the same judge as a health check on its own. In this eval the judge was wrong on about a quarter of items but its two calls disagreed on only 3.5% of them. A judge that is consistently wrong passes a repeat check every time. Catching it takes a reference: a deterministic score where one exists, or a small set of human-labeled items checked on every run.

Teams that cannot add code can still ask the same questions of each run before reading its score:

  1. Did preflight pass: are dependencies reachable, does the judge parse a known answer, does the scorer pass known cases, do the fixtures match their checksums?
  2. Does the run's fingerprint match the baseline's, and is every difference declared?
  3. Did every planned item finish, exactly once, from this run?
  4. Does every item have a successful model call and a successful judge call, or was something swallowed by a fallback?
  5. Does the scorer's view of each item agree with what the trace shows the agent did?
  6. What share of items failed for reasons other than the model, and is it under the threshold?

7.1 A reference pipeline

The repo includes this stage as a working GitHub Actions workflow, eval.yml, with four jobs. Each check is one command, python -m validity.stages <stage> runs/<id>, which writes a short section to the job summary and exits non-zero to stop the pipeline. Fingerprint checking happens in the first job, together with preflight, so a run with an undeclared change never makes a paid eval call.

preflight canaries, checksums fingerprint vs baseline run the eval, traced uploads the run folder validate integrity, coverage gate verdict report valid-items score exclusions, comparison stops before the eval runs undeclared change, canary or checksum fails stops before a score missing or repeated items, missed evals, invalid GitHub Actions jobs; each stage is python -m validity.stages <stage>
Figure 6. The reference pipeline (.github/workflows/eval.yml). A run only reaches the report if it passes preflight and validation.

I started the pipeline four times on the real API, each with one fault injected, to show where it stops. The runs are public.

Injected faultWhere it stoppedWhyRun
NonePassed all four jobsReport: 81.7% on all 60 items, within the clean band (mean ± 2 standard deviations of 13 clean runs: 79.2% to 86.8%)137049432289
Missing POLICY_VERSIONpreflight, after 6 secondsFingerprint: policy_hash changed and was not declared37049828906
Jev down for tickets 20 to 39, silent fallbackvalidateCoverage: 20 tickets with no successful model call. Gate: invalid, 20 of 60 items excluded (33%)37049951030
Judge votes flipped on 25% of callsPassed all four jobs, as degradedReport marked degraded: the judge score was withheld, and the agent's score, 83.3% on all 60 items, was published, within the clean range. Judge faults do not touch the agent's score.37050267138

Table 15. The reference pipeline on four real runs. A clean run takes about a minute across the four jobs.

GitHub Actions job graph for the silent-fallback run: preflight passed in 10 seconds, run passed in 1 minute 11 seconds, validate failed after 10 seconds, and report was skipped.
Figure 7. The silent-fallback run in GitHub Actions. The eval job succeeds and every ticket has an answer, but validate fails on coverage and the report never runs, so no score is published.
GitHub Actions job graph for the judge-flip run: preflight, run, validate and report all passed.
Figure 8. The judge-flip run passes all four jobs. Its report is marked degraded: the agent's score is published and the judge score is withheld.

The silent-fallback run is the case a plain pipeline gets wrong: the run job succeeds, every ticket has an answer, and the score would have been published. Here it stops at validation with the list of missed tickets. Every run in this paper can also be browsed in three linked views. A release check treats sets of runs as candidate releases and decides ship, hold or block against the current version, blocking any candidate whose runs failed the gate. Can I trust this eval run? gives each run a plain verdict, the evidence behind it and what to do next. An engineer view shows the fingerprint, the excluded tickets and the trace for any ticket. The pipeline is specific to this repo's run format (a manifest, item records and OpenTelemetry traces), but the stages only read those three files, so another eval that writes them can use the same checks.

8. Limitations

9. Reproducibility

Everything is in bfalkowski/eval-run-validity: the agents, the dataset, faultd and the fault catalog, the validity code, the workflows, and every run with its traces and ground truth on the data branch.

python run.py --run-id clean-01
python run.py --run-id wk-01 --fault-plan plans/wrong_key_first20.json --error-policy silent
python -m validity.gate runs/wk-01 --baseline baselines/manifest.json

analysis/analyze.py is the only code that reads the ground truth, and it runs after the fact on the collected data branch. Result files for each batch, and a record of what was frozen and when (analysis/FREEZE.md), are in the repo.

Not affiliated with TypeSafe. No employer data or code is used.

How this was made. I designed the study, chose the research questions, faults and claims, ran every batch in CI and reviewed the results. I worked with Claude throughout, in the Claude desktop app, on October 2 and 3, 2026: it wrote most of the code and drafts of the text with me, and ran the analysis scripts, which I checked against the raw runs. The session was configured for model ID claude-opus-5-5 (Claude Opus 5.5). The service can route individual turns to a different model, so that is the configured model, not a guarantee for every turn. Two reviews by other AI models and a manual review also shaped the final version.

Model versions. Every Jev call, for the agent and the judge, used the model alias jev-latest on TypeSafe's System One API on October 2, 2026. An alias can later point to a different model, so a rerun may not reproduce these exact scores; the run manifests record the alias and the date of every run.

Notes

  1. Two clean bands appear in this paper. All analysis uses the 10 clean runs listed in Appendix A (mean 83.0%, band 78.6% to 87.4%). The reference pipeline in Section 7.1 was built after the pipeline batch, so its report stage compares against 13 clean runs: the same 10 plus the three clean runs from the pipeline batch. Its stored baseline (mean 83.0%, standard deviation 1.9 points) gives a band of 79.2% to 86.8%. The bands differ by under a point at each end, and no run in this paper falls between them.

Appendix A. Clean run scores

RunBatchCorrect of 60Plain score
gh37012503737Baseline (final policy)5083.3%
gh37014038090 r1Noise floor5083.3%
gh37014038090 r2Noise floor4981.7%
gh37014038090 r3Noise floor5388.3%
gh37014038090 r4Noise floor5083.3%
gh37014038090 r5Noise floor4981.7%
gh37014574123 r1Development4880.0%
gh37014574123 r2Development5083.3%
gh37019134878 r3Held-out5083.3%
gh37019134878 r4Held-out4981.7%
Ten clean runs498 of 60083.0%, sd 2.19
gh37023643915 none r1Pipeline (13-run band only)5083.3%
gh37023643915 none r2Pipeline (13-run band only)4981.7%
gh37023643915 none r3Pipeline (13-run band only)5083.3%

Table 16. Every clean run on the final policy. The band used in the analysis is 83.0 ± 2 × 2.19 = 78.6% to 87.4%. The best clean run, 88.3%, is above that band, which is a reminder that the band is a reporting convention, not a hard limit on what a clean run can score. Raw items and traces for each run are on the data branch.

References