TypeSafe after Claude Code: verify, then escalate

The interactive below runs the gate on changes shaped like the ones Claude Code produces for this site: a Safari modal fix with a test, a rename that quietly reformatted 39 files, a secret rotation that committed the secret. Watch the four checks land, see which policy line fires, then drag the two floors and watch outcomes re-sort with no new calls.

What you're seeing

The left window is Verify. The card at the top is the change: the commit message as its title, the brief it was supposed to satisfy, and a one-line diff summary with file counts and whether tests were added. The state block is what goes to the model: brief, message, diff, and test output. Then four rows, one per check. Each is a noul question, TypeSafe's yes/no type, and the answer is a single probability of yes. The row shows a yes or no chip, the probability, a margin, and the word "weakest" on whichever check is least decisive for this change.

The right window is the Router. The policy block is five lines, with the line that fired highlighted in red. The outcome counts chart shows how judged changes have split across auto-merge, confirm, review, and block. The recent list shows the newest changes with their outcome chip. The Tally across the top tracks changes checked, the auto-merge rate, how many still need a person, and the total verify cost, which after 25 changes is still under a thousandth of a dollar.

The four outcomes are four different amounts of your attention. Auto-merge is none: the change matched the brief, touched nothing protected, has tests, and the message is honest, all with margin to spare. Confirm is one click: the change is clean and decisive but it touches something protected (payments, auth, secrets, a destructive delete) or has no tests, so a person should see it go in. Review is a real look: at least one check came back torn. Block is a confident no: the diff does not do what the brief asked, or the message misdescribes it, and the model is sure.

A few things to try:

• Drag the review floor from 0.60 down to 0.30. Two changes leave review: the thumbnail fix that solved a hang by adding a 30 second sleep drops to block, and the Astro version bump rises to auto-merge. Both were sitting at margins around 0.4 to 0.5, decisive enough for a lower floor, not for the default.
• Drag the confirm floor from 0.85 down to 0.60. The dependency bump that pulled in a major version of an auth library moves from review to confirm. Its checks all lean the right way at around 0.85, which is short of the default bar for protected changes.
• Step to the rename that reformatted 39 files. on_brief comes back near 0.10 and msg_accurate near 0.12. The commit message says "rename"; the diff says otherwise. That is a block, and it is the change most likely to slip past a tired reviewer.
• Watch for the "weakest" tag. It moves between checks from change to change, and it is the only number the policy actually compares against the floors.
• Read the newsletter form. The model is torn on whether it matches the brief and whether the message is honest, and the protected check is a firm yes. Nothing about that should merge on its own.

How four checks become one decision

Each check is a noul question. The answer is a probability of yes, and there is no separate confidence field for a noul: a value near 0.5 means the model is torn, not that the answer is "medium". So the first thing code does is turn each probability into a margin, the distance from 0.5 doubled, which runs from 0 (torn) to 1 (certain either way). The change's margin is the smallest of its four. A change with three certain answers and one coin flip is a coin flip.

const margin = (p) => Math.abs(p - 0.5) * 2;
const checks = ["on_brief", "protected", "tested", "msg_accurate"];
const m = Math.min(...checks.map((k) => margin(a[k].noul)));

const failHard = a.on_brief.noul < 0.5 || a.msg_accurate.noul < 0.5;
if (failHard) return m >= REVIEW_FLOOR ? "block" : "review";
if (m < REVIEW_FLOOR) return "review";
const needsHuman = a.protected.noul >= 0.5 || a.tested.noul < 0.5;
if (!needsHuman) return "merge";
return m > CONFIRM_FLOOR ? "confirm" : "review";

The order encodes the priorities. A confident failure on the brief or the message is a block before anything else is considered. Anything torn goes to a person before the cheaper outcomes are even evaluated. Only then does the policy ask whether the change is the kind that can merge on its own. That structure follows the confidence-routing pattern in TypeSafe's docs, which suggests a floor around 0.6 below which every action goes to a human, and a higher bar, around 0.85, for actions with consequences.

The questions themselves are worth a look, because the whole gate rests on them. This is the request the demo follows, trimmed to two checks:

{
  "model": "jev-latest",
  "state": {
    "brief": "Escape should close the term modal in Safari",
    "message": "fix: close term modal on Escape in Safari",
    "diff": "<unified diff, 2 files>",
    "test_output": "<runner summary>"
  },
  "questions": {
    "on_brief": {
      "type": "noul",
      "instructions": "Does `diff` do what `brief` asked, and nothing else?",
      "criteria": {
        "true": "Every hunk serves the brief",
        "false": "Unrelated edits, reformatting, or a different fix than the brief describes"
      }
    },
    "msg_accurate": {
      "type": "noul",
      "instructions": "Does `message` describe what `diff` actually does?",
      "criteria": {
        "true": "A reviewer reading only the message would not be surprised by the diff",
        "false": "The message understates, overstates, or mislabels the change"
      }
    }
  }
}

Two things about this request. The backticked field names point the model at specific parts of the state, which is the convention the docs use for nested context. And the criteria are contrastive: they say what counts as false, not just what counts as true, which is what stops a reformatting-heavy diff from passing as "on brief".

Why the floors are the only knobs

Every answer the model returns is stored as a probability. The floors are comparisons against those probabilities. When you drag a floor, the demo re-applies the five lines to every stored answer, the counts re-sort, and the header above the chart reports "re-sorted N judged · 0 new calls · cost unchanged". Nothing was sent, nothing was spent.

This is the property that makes the gate reviewable. The questions and the two floors are the entire policy surface. TypeSafe's guidance for agent-written integrations is to keep exactly those constants in one place, because they are what a person should be editing, and agents are not especially good at writing questions on their own. Everything else in the gate is plumbing.

Where this fits after a Claude Code run

The natural home is a hook or a CI step that fires when Claude Code opens a pull request or finishes a task. The state is the brief you gave it, the commit message it wrote, the diff, and the test runner's output. The docs are clear about sending only the context a question needs, so a large diff should be trimmed to its hunks and file list rather than pasted whole, and the test output to its summary lines. One request, four questions, four probabilities, one of four outcomes.

What happens next is ordinary tooling. Merge means the PR merges. Confirm means a one-line message with a button. Review means the PR is assigned to a person with the four answers pasted into the description, which is a better starting point than a bare diff. Block means the PR is closed with the failing checks quoted and the agent is asked to try again from the brief. A human in the loop stays on the outcomes that need one, and the changes that were never going to need one stop waiting in the same queue.

Two cautions. The diff is content the model reads, and content that reaches a model is content that can try to instruct it. The gate's answer is that the model never acts, code does, and the worst a hostile diff can do is bias a probability that a floor then catches. See prompt injection for the general shape of that problem. And typed checks do not replace tests. The tested check asks whether tests exist that exercise the change; it does not run them. The runner does.

This is the third of three posts. The first puts a typed gate before Claude Code runs, and the second covers jobs that are decisions rather than code. For how a judge model compares with a set of typed checks, see LLM-as-judge explained. The changes in the demo are shaped after this site's own backlog.

Frequently asked questions

Key takeaways

• A yes/no answer near 0.5 is not "medium". It is the model saying it cannot tell, and the right response is to route, not to round.
• The weakest check decides. Three certain answers and one coin flip is a coin flip, which is why the policy takes the minimum margin and not the average.
• Protected changes never auto-merge, however confident the model is. The confirm outcome exists so a person spends one click on payments, auth, secrets, and deletes instead of a full review.
• The order of the policy lines is the safety story. Confident failures block first, torn answers go to a person second, and only then does anything merge on its own.
• Two floors are the whole tuning surface. Moving one re-sorts stored answers for free, which means you can calibrate the gate against last month's merges before you trust it with next month's.