Playbooks#Coding agent#Code review

How the Claude Code team verifies AI-written code: three test types, a 39-item checklist, and one rule that cannot move

Claude Code's team writes and reviews 90% of code with Claude. Its post: three test types, a verify skill, a 39-item checklist and checks the code can't edit.

On November 16 the Claude Code team published how it verifies work: roughly 90% of the team’s code is written and reviewed by Claude. That ratio does not rest on one-shot correctness. It rests on treating verification as its own system — with tests, a triggered checklist, an observe-before-enforce rollout, and an evaluation loop for the reviewers themselves. A Chinese summary of the post circulated quickly (and prompted this article), but the details are worth reading at the source.

Build the loop before scaling out

The team’s causal chain is blunt: without a way to verify output, you can only run one or two Claudes at a time, because you have to check everything yourself. With a verification loop, dozens or hundreds in parallel become meaningful. In internal testing on structured file generation, a separate grader checking Claude’s work against a rubric improved task success by up to 10 points over a standard prompting loop, with the largest gains on the hardest problems.

The cost side changed too. Tests, benchmarks and specs used to be expensive to produce; an agent now generates them in seconds, and once generated they run as software, so running them often costs few tokens. The results then feed back into the agent: if it keeps printing user data in logs, the team writes an explicit rule in CLAUDE.md or adds a hook so the next attempt is blocked.

Three test types worth the investment

End-to-end first. If you can only invest in one kind of test, pick E2E: it walks real user flows (add to cart, check out, land on confirmation), is hard to game, and has a definitive done criterion. Where a unit test reports “timeout after 30 seconds,” an E2E run can screenshot the flow and explain why the checkout button is disabled. The team gives Claude hands and eyes on the real system — a Chrome extension for web, simulators for iOS and Android, computer use for desktop, a locally running service for backend work — and uses the feature map (paths, shortcuts, selectors, expected results) as both guide and grading rubric.

Property tests target the edge cases people and agents both miss. Write a statement that must never fail, then use fast-check, Hypothesis or proptest to generate thousands of inputs trying to break it. Typical properties: encode-then-decode returns the original; an account balance can never go negative; running a migration twice equals running it once; optimized code produces the same output as the naive version. These random boundaries carry none of the original author’s bias, which is why they help most when a spec has drifted from production and rewrites need a different oracle.

Differential tests fit large migrations where behavior must not change: run the old and new systems on the same random inputs or a slice of production traffic, and any mismatch is a bug or a breaking change. Normalize noise such as whitespace, and keep an approved-differences list for cases where the new environment legitimately differs.

The team also names two failure modes. Flaky tests teach the agent to re-run until green instead of fixing code, and slow tests reduce how often the agent can check, which makes pinpointing problems harder.

Running the change is the evidence

For this team, verification starts with running the app, not with unit tests. The launch-and-check steps are codified in a skill called verify, placed under .claude/skills/ as close to the relevant code as possible; Claude knows to call it before committing. The skill itself gets investment: keep it fast, keep it reliable, stress-test it like a smoke-test harness.

Failures are handled with restraint. Claude tries to fix what fails; if a check is still failing after three rounds, it stops and brings in a human. The people who own the code write the checks and refine them until they catch what the owner would catch, and when a check pulls in a reviewer it must say why.

A checklist and rules that cannot move

The reliability checklist has 39 items, each with a trigger readable from a diff, a check, and the incidents or PRs behind it. One example: “--resume loads old transcripts.” When session storage or message types change, the question is whether a new field loads safely, since older transcripts will not have it. Claude reads the diff and activates only the three to ten items whose triggers match, so nobody memorizes past outages.

Reviews are not improved by stacking reviewers. Adding a second, more expensive reviewer on top of the primary is often worse than tuning the primary. A REVIEW.md states what to watch for in this code, and each round inherits context and past failures from earlier rounds instead of starting over.

Several lines do not move: a change cannot modify the check that judges it — REVIEW.md is read from the base or default branch, never from the pull request — reviews apply to commits, so a result on a commit that changed mid-review is worthless, and cheap checks run first, with tests, build and linter passing locally before a PR opens.

Observe, then enforce, then grade

Every commit carries a stamp for how it was verified, including “none,” so unverified commits can be queried later. The team built a commit gate with three modes: off, observe and enforce. Observe logs what would have been blocked so false positives can be reviewed; enforce blocks, with an explanation allowed in the message. If a session has been blocked twice, the third commit goes through so Claude does not get stuck.

The review system itself is graded. They track how many merged changes were reverted, how many maintainers rejected before merging, and a size-matched set of changes that stayed clean. In the formal evaluation, the repo is recreated at the state of a past incident, a one-sentence answer key names the bug, and only findings that identify it score; merged and stable changes act as a control against over-flagging. Findings are tagged Important, Nit or Pre-existing — the last does not hold the author responsible. Recall is measured separately for raw findings, post-verification and final review, so a single end-to-end score cannot hide bugs the system lost along the way. Recall is tuned only up to the point where the next gain would be overfitting to the eval.

Turn misses into rules

Two paths convert misses into rules. First, patterns in human review comments and PR reverts: an issue must appear in at least two PRs and be identifiable from a diff before it becomes a candidate rule; three agents then try to disprove it, and a human approves it in a PR. Second, replay three bug-finding tools against a PR tied to a known incident: if at least two find the bug on a small diff, the rule library has a gap and REVIEW.md gets a new entry.

Flaky tests get the same discipline: no skipping, no sleeps, no retrying until green. If the root cause cannot be found, the test is deleted with a written reason. Rule files, test suites and verification scripts are pruned on a routine so they stay worth running.

Summary

What transfers here is not a specific tool but three principles: maintain the verification layer like any other system; keep the rules out of reach of the changes they judge; and turn every miss into a rule, then prune the rules. Do that, and the job shifts from reading every diff to defining what “done” looks like and reviewing evidence — which is what makes running many Claudes at once possible. Teams that want a better review surface first can start with a structured workstation for plans and diffs such as Plannotator; for model-level context, see the Claude Fable 5.2 release.