Bryan Falkowski Can I trust this eval run? · Engineer view · Paper

Release check

Every change to the automated reviewer (its prompt, the policy, the model, the code around it) is a release. Before it ships, the new version decides the same 60 gold-labeled cases several times, and this page answers two questions: can we trust the measurement, and is the new version at least as good as the current one? Each candidate below is a real set of runs from the study. Any run can be opened in Can I trust this eval run? to see its checks and evidence.

Candidates, newest first

The fingerprint is a short hash of everything that defines a build: model name, prompts, information sent to the model, dataset, code and environment. Same fingerprint, same build. All runs come from one day of the study, so the time ranges overlap; in a real pipeline each candidate would follow the last release.

How the decision is made
  1. Block if any run failed a validity check. An untrusted measurement can't approve or reject anything: declare the change or rerun.
  2. Hold if there are fewer than 3 runs, if the automated QA became unreliable, if accuracy is lower than the current version by more than run-to-run noise allows, if a critical error type rises by more than 5 points, or if any kind of case drops by more than chance allows for its number of decisions (and by at least 15 points).
  3. Ship otherwise. Environment differences are noted but don't hold a release on their own.
Quality is measured against the gold labels only, never against the automated QA judge.