Bryan Falkowski All writing

Earning the Next Rung

Bryan Falkowski · November 21, 2026 · Follows The Second Office's First User

Building58 is my self-building office, a team of Claude Code sessions that writes its own tickets, builds them, reviews each pull request and merges on Autopilot. Every pull request there still waits for a reviewer session to read the code. I want some kinds of change to merge with nobody reading them, but only when the evidence shows that is safe.

So the office is building a tool for itself, which I call Exhibit. For each pull request, Exhibit decides what evidence the change needs and checks it. Its first full run on Building58's history read 146 merged pull requests and traced 20 bugs that got through back to the pull requests that added them. In its first afternoon in shadow mode, it passed 11 commits that the reviewer sent back, so nothing merges on its verdict yet.

Three ideas guide the design. Exhibit learns what evidence each kind of change needs from the bugs that got through. It measures how strong that evidence is instead of trusting a green check. And each kind of change goes up the trust ladder on its own, when its own numbers say so. This post describes that design, which is the target the office is building toward, and the first real numbers.

Building58 is a toy project with one user, so the numbers say more about the method than about what would be safe at a company.

A cartoon of three small white robots on a tall wooden ladder in a loft office. The rungs read 0 Off, 1 Shadow, 2 Required check and 3 Merge without reading. One robot holds a long paper receipt labeled evidence.json, one checks a rung with a magnifying glass, and one sits on the top rung with a cup of coffee. A developer at the bottom holds the ladder steady with both hands.
Agents do the climbing, and a person still holds the ladder (made with Higgsfield).

Reading every diff does not scale

Lauren Tan describes a trust ladder for coding agents: agents get more as verification proves them, and near the top they merge their own pull requests while a person samples them after they land. I wrote about it in Who Owns the Process?

Building58 was most of the way up that ladder. As I described in The Office Builds Itself, a PM session writes tickets, Dev sessions build them, and Autopilot starts the tickets I approve and merges a pull request once a reviewer session gives it a merge verdict. But every pull request still waited for a model to read the diff, and that reviewer was the same model that wrote the code. It found real bugs and cost tokens and review rounds, and nothing measured what it missed.

I also had a reason outside the office. As I work more on Autopilot and try to translate it into actual enterprise-grade projects, I need to put more maturity around the trust ladder.

So I changed the question from whether the code is good to whether the evidence is strong enough that nobody has to read the code. The Claude Code team describes the same move in how it thinks about verification: the person stops reading every diff, says what done looks like, and reviews the evidence.

Where trust comes from, today and at rung 3 Today: a pull request goes to a reviewer session that reads the whole diff, and Autopilot merges on its verdict. The reviewer is the same model as the Dev, and nothing measures what it misses. At rung 3, the target, for one kind of change: the pull request goes to an Exhibit check of the evidence that kind needs, and Autopilot merges on the verdict. Its misses are counted for each kind of change, and I read a random sample of the pull requests after they land. Today: a reviewer session reads every diffPull requestfrom a DevReviewer sessionreads the whole diffMergeon its verdictSame model as the Dev.Nothing measures its misses.Rung 3, the target: measured checks decide, for one kind of changePull requestfrom a DevExhibit checkthe evidence the kind needsMergeon the verdictIts misses are counted,for each kind of change.I read a blind sampleafter the PRs land
Today, trust comes from a model reading the diff. At rung 3, it would come from checks whose misses are counted, plus a sample I read.

The office built it for itself

Building58 has matured and stabilized, and it can now recover on its own from many different failure modes. I am surprised how many there have been. So I wanted to give it a real project and see how it would do.

I wanted the end users to be developers, and I looked for a project that would be useful to Building58 but could also stand alone, one I could use in my own development. I also wanted the Building58 developers to iterate on it while they use it, so the product can grow and mature on its own. Exhibit was that project.

The building has several rooms, each a separate copy of the office with its own sessions and its own Board. Room C builds Exhibit. The plan is for the Building58 room to run it on every pull request to the office's own code, and for Room B to run it on edd-platform, an eval tool of mine.

The user rooms will install published releases only, so they hit the bugs any user would, and they will file issues when a session fights the tool. The first release, v0.1.0, came out on November 20.

Three rooms and one tool Room C builds the Exhibit repo and ships releases to two user rooms: the Building58 room, which uses Exhibit on every pull request to the office, and Room B, which uses it on edd-platform. The user rooms send issues and policy misses back to Room C. A hidden canary job opens pull requests with planted bugs in both user repos, and Exhibit must flag each one. The users, on published releases Room C builds the Exhibit repo Building58 room uses it on every office PR Room B uses it on edd-platform releases issues and policy misses Canary job: PRs with planted bugs
The plan: Room C builds Exhibit, the other two rooms use it, and a hidden job checks that it flags planted bugs.

Room C is building it from a spec in nine slices, and five were done by the third day.

Learn the rules from the bugs that got through

Exhibit runs as one loop. It learns a policy from history, checks each new pull request against it, and sends any bug that gets through back into the history.

One loop with two halves Top row, learn from history: merged pull requests from GitHub, find the fixes with fix signals, trace each fix back with SZZ, error analysis into codes and categories, and the policy in evidence.yml. The policy says what each kind of change needs. Bottom row, check each new pull request: a new PR goes to exhibit check, which gives a verdict and gaps, and then it merges. Sometimes a bug gets through and is fixed later. That escaped bug goes back into the history as a policy miss, and the loop runs again. Learn from historyMerged PRsfrom GitHubFind the fixesfix signalsTrace each fixback, with SZZError analysiscodes, categoriesPolicyevidence.ymlCheck each new PRNew PRfrom a Devexhibit checkverdict and gapsMergeto mainEscaped bugfound and fixed laterwhat each kind needsa policy miss goes back into the history
Each bug that gets past the check goes back into the history, so the policy improves from every miss.

The history starts with the fixes. Exhibit traces each fix back to the pull request that added the bug with SZZ, named for the authors of the 2005 paper behind it: run git blame on the lines the fix changed, and see which earlier change wrote them. A bug found that way is an escaped defect.

The first escaped defect Exhibit found in Building58 A timeline of pull requests on main. PR 27, titled Give the manager a glass office behind the back wall, wrote lines in office/baked/bake.json. Later, PR 30, titled Fix the manager's door and give him a keyboard, changed those lines. Git blame on the lines, just before the fix, points back to PR 27 for 12 of the 14 lines it counted, a confidence of 86 percent. PR 27 is marked as the change that added an escaped defect. mainPR #27PR #30"Give the manager a glass officebehind the back wall""Fix the manager's doorand give him a keyboard"office/baked/bake.jsonlines written hereoffice/baked/bake.jsonlines changed by the fixescaped defectgit blame, just before the fix: 12 of the 14 lines point back to #27 (86%)
The first escaped defect Exhibit found in Building58: git blame on the lines that fix PR 30 changed points back to PR 27.

A common mistake is to trust titles and labels to find the fixes. On Building58, a bug label, a title that starts with "fix", a fix/ branch and a revert together found 2 fixes in 146 merged pull requests, and one of them was a change of process, not a bug. The workers name a pull request for its symptom, such as "T-170: tasks.test.js can hang in CI". So the fixes are named by hand in evidence.yml, with signals such as pr:129. With a list of 33, the first full run found 34 fixes and traced 20 of them to the pull requests that added the bug. Most fixes with no link changed code from the office's first commit, which came before the repo used pull requests.

Exhibit's own repo has the same problem. Its titles start with a ticket number, such as "147: ...", and a number alone is not a ticket key. So none of the first 15 pull requests that Exhibit checked counted as a fix, and the check for a test that failed before never ran (ticket 233).

Then comes error analysis, the same method I teach for LLM evals. Claude drafts a short code for each defect, I keep or change it, and the codes group into categories. SZZ gets some links wrong, mostly where code was only moved, so I can also reject a link while I label.

A category becomes a rule only when it appears in at least two pull requests and its trigger can be read from a diff. Agents try to disprove it first, and I approve it in a pull request like any other change.

Example categoryRule in the policyKind of change
The restart path was never runA restart run, with its logrelay-core
Visible only in the browserA screenshot or a short clipoffice-visual
The test checked the wrong thingA floor on the mutation catch rateAll kinds
Worked with fakes, failed with the real gh or gitA contract test against the real toolticket-store
The manual was out of dateA link check and a plain-words checkmanual

Ask each kind of change for its own evidence

A docs fix and a relay change carry different risk, so they need different evidence. The rules live in the repo as evidence.yml. Each kind of change has a trigger read from the diff, the evidence it needs, and the escaped defects the rule came from, and Exhibit runs only the rules whose triggers match.

# evidence.yml, an example for Building58
kinds:
  relay-core:
    match: [relay/supervisor.js, relay/session-host.js]
    needs: [tests-before-after, restart-run, mutation: 0.85]
    merge_without_reading: never
    source: [D-004, D-011]
  ticket-store:
    match: ["relay/ticket-stores/**"]
    needs: [contract-test, mutation: 0.80]
    merge_without_reading: allowed
    source: [D-019, D-027]
  manual:
    match: ["docs/manual/**"]
    needs: [link-check, plain-words]
    merge_without_reading: allowed
    source: [D-031, D-040]
default:
  needs: [tests-before-after, mutation: 0.70]
Inside exhibit check A pull request is classified by its files into kinds of change with a risk. The policy, evidence.yml, read from the base commit, says what evidence those kinds need. Exhibit collects up to four kinds of evidence: tests on the base and the head, a test that failed before and passes now, mutation on the changed lines, and probes the policy names, such as a restart run, a screenshot or a link check. If every requirement is met, the verdict is meets policy. If not, it is needs a person, and the verdict names the gaps. Pull requestthe diffClassifyfiles, kind, riskevidence.ymlfrom the base commitTests on base and headwhat changedA test that failed beforeand passes nowMutation on changed linesare the tests strong?Probes the policy namesrestart, screenshot, linksMeetspolicy?✓ Meets policyyes✗ Needs a personnames the gapsno
The kind of change decides what evidence is required, and "needs a person" always names the evidence that is missing.

exhibit check gives each pull request one of two verdicts. "Meets policy" means every requirement was met. "Needs a person" lists each gap, the rule behind it and how to close it, so a reviewer can see why the pull request stopped without working it out from the diff. If Exhibit itself fails, the verdict is an error, and an error never passes.

The strongest evidence comes from running the change. Tests run on the base and the head, and a test that failed before and passes after shows that the change fixed something. For the relay, the policy asks for a real restart of the program, because two bugs that stalled the office were in the restart path. For the office page, it asks for a screenshot.

A pull request can match more than one kind. Then it needs the evidence of every kind it matches, and it can skip the reviewer only when every one of those kinds allows it. On Exhibit's own repo that is the usual case. Of the first 15 pull requests Exhibit checked there, only one matched a single kind, and 9 matched docs, source and tooling together, because a code change comes with its tests and the house rules ask it to update the spec in the same pull request. A docs kind at rung 3 would almost never apply there. Building58 is the same. In the first history run, with kinds drawn by folder, 140 of its 146 pull requests touched the relay and 108 touched the docs. So a kind as wide as a folder covers almost every pull request, and a useful kind has to be narrower, like relay-core in the example above.

Measure the tests, not only whether they pass

A weak test passes too, so a passing test is weak evidence on its own. Exhibit plants small bugs in the lines the pull request changed, such as > made into >=, and runs the tests again. The share of planted bugs the tests catch is the catch rate, the true positive rate of the tests for that change, and the same number I use to check an LLM judge in Can You Trust Your Judge's Score? Google has run mutation testing this way, on changed code during code review, for years.

How strong are the tests on this change? Four lines the pull request changed, each with a small planted bug: greater than made into greater than or equal, a line removed, true made into false, and a timeout made one longer. The tests fail on the first three, so those bugs are caught. They pass on the last one, so that bug survives. Across 25 planted bugs, the tests caught 23, a catch rate of 92 percent, and this kind of change needs 85 percent. The 2 that survived show that a test is missing. A line the PR changedThe same line with a planted bugDo the tests fail?if (elapsed > limit)if (elapsed >= limit)✓ caughtstop(task);// line removed✓ caughtreturn true;return false;✓ caughtsetTimeout(check, limit);setTimeout(check, limit + 1);✗ survived... and 21 more planted bugs on the changed lines92%23 of 25 caughtneeds 85%23 caught2 survived: a test is missing
In this example the tests catch 23 of 25 planted bugs, and the two that survive show where a test is missing.

The room's reviewer sessions already do this by hand. In pull request 252, a new test was meant to show that a guard for the repo "shop" leaves the repo "shopping" alone. In a scratch copy, the reviewer made the guard wrong in the way the ticket warned about, and the test still passed, so the test could not fail. Exhibit had said "meets policy" on that commit, because all 621 tests passed. Mutation on the changed lines makes this kind of check on every pull request, with no person.

Mutation runs only on the changed lines and stops at a time budget. When the budget runs out, Exhibit reports the catch rate with its interval, and too few mutants means "needs a person".

Exhibit also records who wrote each check. A test the Dev wrote can share the Dev's blind spots, so a hidden suite that the Dev cannot see or change counts for more.

Keep the rules out of reach of the change

A pull request must not be able to change the rules that judge it. Exhibit reads evidence.yml from the base commit, never from the pull request, and any change to the policy needs a person.

A verdict counts only for the commit it checked, and Autopilot merges with --match-head-commit, so a later push sends the pull request back. The verdict that counts comes from CI, because a session can skip or change a run on its own machine.

Where Exhibit runs Five lanes. Session: a Dev runs exhibit check locally, then opens the pull request. CI on the PR: exhibit check runs again and gives the verdict, the pull request comment and the exhibit on the data branch. CI on main: exhibit trace runs on each merge, which is not built yet. Weekly job: exhibit history, then exhibit backtest, which is not built yet, then the report. Me: I label the escaped defects and approve the policy, and I read the policy misses that trace finds, which is not built yet. SessionCI on the PRCI on mainWeekly jobMeexhibit check, localopen the PRpushexhibit checkverdict, comment, data branchexhibit traceon mergeexhibit historyexhibit backtestreportlabel, approve the policyread policy misses
The verdict that counts is made in CI, where a session cannot skip or change the run, and nothing needs a server. The dashed parts are not built yet.

CI also runs on every push to main, because two pull requests that each pass can still break main together. On the second day, the GraphQL change and the pr: signal merged eight minutes apart, and each had passed against the main it started from. Together they broke a test. The run on main caught it within minutes, and a fix merged 34 minutes later.

Prove the policy on history before trusting it

Exhibit tests the policy on pull requests it did not learn from. The backtest learns from the older pull requests and tests on the newer ones, and it counts how many escaped defects the policy would have caught.

The backtest A row of merged pull requests along a time line, split by date. The policy is learned from the older ones on the left and tested on the newer ones on the right. Taller red ticks are pull requests that added an escaped defect. On the test side, four of them are marked caught and one is marked missed. Small circles above four other pull requests on the test side mark false demands, where the policy asked for evidence on a pull request that was fine. Older PRs: learn the policyNewer PRs: test itsplit by date✓✓✓✗✓a PR that added an escaped defect✓caught✗missedfalse demand: evidence asked for on a PR that was fine
The policy learns from older pull requests and is tested on newer ones it never saw (an illustration, not the real counts).

The cost of a policy is its false demands, the evidence it asks for on pull requests that turn out fine. Exhibit measures them against clean pull requests of about the same size, because small changes pass easily and would make the policy look cheap.

Exhibit also counts what is lost at each step: the links SZZ found, the links I kept, the defects that became rules, and the defects check would have flagged. A canary job tests the policy live with pull requests that carry a planted bug, and Exhibit has to flag every one.

Start every gate in shadow mode

Each kind of change starts at the bottom of the ladder and moves up only when its own numbers say so. In shadow mode Exhibit gives its verdict and nothing waits on it, so I can read what it would have blocked before it blocks anything.

The trust ladder for one kind of change Four steps going up from left to right. Step 0, off: a reviewer session reads every pull request. Step 1, shadow mode: Exhibit reports and nothing waits on it. Step 2, required check: pull requests wait on the verdict, and a reviewer session still reads the code. Step 3, merge without reading: Autopilot merges on the verdict, and I read a blind sample. To go from 0 to 1, I turn it on. From 1 to 2, the verdict agrees with the reviews with few false demands. From 2 to 3, there are almost no misses for this kind, in the backtest and live. I take each step myself. 0 Off a reviewer reads every PR 1 Shadow mode Exhibit reports, nothing waits 2 Required check PRs wait, a reviewer still reads 3 Merge without reading Autopilot merges, I sample I turn it on for one kind Agrees with the reviews, with few false demands Almost no misses for this kind, in the backtest and live I take each step myself. A session never turns on a rung.
Each kind of change climbs on its own, and the step from 2 to 3 is the one that takes the reviewer out.

Exhibit began running in shadow mode on its own repo on November 20, and the first afternoon showed why this step comes first. Between 11:18 a.m. and 4:51 p.m. it checked 29 commits on 15 pull requests. Its policy then asked for the tests on the base and the head, and a link check on the docs. The table compares Exhibit's verdict on each commit with the room's review of the same commit. These values are real.

Exhibit's verdictCommitsReviewer said mergeReviewer said fixNo review
Meets policy2412111
Needs a person5131

The reviewer sent back 11 of the 23 reviewed commits that Exhibit passed, and 4 of those had a blocking finding. In pull request 249, all 616 tests passed, and the reviewer found five inputs where the new code read a quoted review as the comment's own verdict, which main had read correctly. None of the 5 "needs a person" verdicts found a real problem. Two came from early runs that matched no rule, two were changes to the policy, which always need a person, and one was a link check that failed because the script it runs was not on that branch yet.

So on that day, "the tests pass and the links work" was weaker evidence than the reviewer's reading, and nothing waited on Exhibit while that showed. Exhibit is not close to the step from 1 to 2, which needs its verdict to agree with the reviews. The run also showed what was missing: mutation was not built yet, and no fix signal matched the repo's own fixes. One afternoon on a repo that is still being built shows the gaps, not a rate.

The step from 2 to 3 needs a number I can defend. With no misses in n pull requests, the rule of three puts the 95% upper bound on the miss rate at about 3 divided by n, so 150 clean pull requests mean a miss rate that is probably below 2%. The bound does not prove the rate is zero, and I decide whether it is low enough for each kind.

The rule of three A line chart of the 95 percent upper bound on the miss rate against the number of pull requests of one kind with no misses, from 0 to 300. The bound falls fast at first and then slowly: about 10 percent at 30 pull requests, about 3 percent at 100, about 2 percent at 150 and about 1 percent at 300. 0%4%8%12%16%050100150200250300PRs of one kind with no misses95% upper bound on the miss rate30 PRs with no misses: the miss rate is below 9.5% with 95% confidence30 PRs: about 10%100 PRs with no misses: the miss rate is below 3.0% with 95% confidence100 PRs: about 3%150 PRs with no misses: the miss rate is below 2.0% with 95% confidence150 PRs: about 2%300 PRs with no misses: the miss rate is below 1.0% with 95% confidence300 PRs: about 1%
The bound falls fast at first, and after that each halving needs twice as many clean pull requests.
Kind of changePRsMisses95% upper boundFalse demandsRung
manual21201.4%4%3
ticket-store16101.9%9%3
default30532.8%10%2
office-visual13825.1%12%2
relay-core9715.6%21%2, never 3

The values in this table are examples of what the report gives. The relay is set to never, because a bad change there takes down every session, including the ones that could fix it.

Before any kind reaches rung 3, Room B will make the verdict required while the Building58 room stays in shadow mode. Then I will compare the two rooms on review rounds per merged pull request, tokens per merged pull request and stuck time.

Stuck time is the failure I care about most, and a gate that blocks can cause it. So in enforce mode, a blocked pull request can still merge with a recorded skip: a label, and a reason in its body. The verdict stays "needs a person", and every skip is counted for its kind of change, so overuse shows.

Autopilot's loop with Exhibit A ticket I approved goes to a Dev, who builds it, runs exhibit check and opens a pull request. Exhibit checks it in CI. If it meets the policy and its kind is at rung 3, the relay merges it with no reviewer. Otherwise a reviewer session reads the evidence and the gaps and gives a verdict, merge or fix, and a fix goes back to the Dev. After a merge, the relay pulls main and restarts if it needs to. Some merged pull requests are picked at random for a blind sample that I read. meets policy, rung 3 TicketI approved Dev buildsand runs the check Exhibit in CIverdict and gaps Mergechecked commit Pull mainand restart Reviewthe gaps first needs a person, or below rung 3 merge fix Blind sample I read random merged PRs
For a kind at rung 3, a pull request that meets the policy skips the reviewer.

At rung 3, Autopilot merges a pull request that meets the policy with no reviewer session, and it picks a random sample of those that I read without seeing the verdict. Only I turn on a rung, and Exhibit never merges anything itself.

Turn misses into rules, and remove rules that do not work

A policy miss is an escaped defect whose pull request Exhibit passed. When a fix merges, exhibit trace finds the pull request that added the bug, records its verdict, and sends a miss back into error analysis.

A rule that never catches anything but keeps asking for evidence costs every Dev time, so the backtest marks those rules and I remove them in a pull request.

Others are on the same ladder

Meta's RADAR decides which diffs can land with no human review, using eligibility rules, a learned risk score, an LLM review and deterministic checks. Meta reports more than 331,000 diffs landed with about a third of the revert rate of other diffs, though its paper describes no measure of test strength and no random sample of what landed.

The Autonomy Ladder guide Reading evidence instead of code covers mutation on changed files and moving classes of change up or down by evidence. The closest is the Claude Code team's post on verification, where several of Exhibit's rules came from. The spec's related-work section has the rest.

A toy project with one user

Meta's numbers come from production, and mine do not. Building58 and edd-platform are toy projects, and I am their only user. When a bug gets to main, I file a ticket. Nobody gets paged at 4 a.m., and no outage makes the news.

That is why I can turn on a rung after a few weeks of clean numbers, and it is also why my numbers do not carry over to a team with real users. What a 2% bound means depends on what a miss costs. At a company, the bound before rung 3 would have to be much smaller, each kind of change would need many more pull requests, and the misses would be counted next to reverts and incidents.

It was also a new project, and when I started I was not sure what the data would show. A lot of it was exploratory, to see whether it is useful at all.

What it does not do

Exhibit works only on GitHub. Every GitHub call goes through one forge module, so GitLab would be a second module behind the same methods, but nobody has built it. Exhibit also learns only from bugs that someone fixed, so the blind sample is my only way to find the rest.

A check is slow on a big test suite, because it runs the tests on the base and the head, and a third time for a fix. Building58's tests first ran one file at a time, and two runs took more than 72 minutes on a GitHub runner. With six files at a time, a real check took 17 minutes, and 24 minutes for a fix (pull request 239).

One model writes the code, the tests and the fixes, so the hidden suite and my sample are only a partial defense against shared blind spots. A policy belongs to one repo, which is why the field test planned two. The first run showed that edd-platform's history is too short to learn from: 112 pull requests in five days, 2 fixes and no linked bug. So Phase A uses Building58 alone for now. And what to build, the quality bar and the rungs themselves stay with me.

Exhibit is an MIT-licensed CLI and GitHub Action, and it needs no server. Claude only drafts what I confirm, and every command works without it.

Trying this on your own repo

  1. Run exhibit history in a clone of your repo and read the escaped-defects report. If it says the history is weak, count reverts as fixes or name a few fixes with pr:.
  2. Label the escaped defects, and make a rule only from a category that shows up in at least two pull requests.
  3. Run exhibit check in shadow mode on every pull request for a few weeks, and read what it would have blocked.
  4. Add mutation on the changed lines for one kind of change, with a tool such as Stryker, and compare its catch rate with what your reviewers find.
  5. Make the check required for that one kind, keep a blind sample, and skip the reviewer only when the bound fits what a miss costs you.

What's next

The constitution has two rungs left: a label that starts a ticket with nobody pressing Start, and a monitor that files tickets about what shipped. exhibit trace will do part of the second one. I also want to point Exhibit at a public repo with a long history and real users, where a miss costs more than a ticket.

References