Constrained decoding: forcing an LLM to emit valid JSON

The interactive below generates a small JSON document one token at a time, with a schema grammar standing between the model and the output. Watch the red sweep delete illegal candidates in the center histogram, then flip to Prompt Only, set the pressure to Hostile Logits, and watch the same distribution fall apart without the mask.

What does the demo show?

Three panels share one generation loop. The left panel is the output stream: a prompt asks for a user record as JSON, and the document below it grows one token at a time. A comment line under the prompt says whether a decoder mask is compiled or whether the model is just being asked nicely. Keys and values print in normal JSON colors, tokens that break syntax flash red, and tokens that parse but violate the schema get a red underline. The footer keeps a running tally of syntax and schema passes across runs, then prints a per-run verdict with the reason it failed, like an unquoted key or a value outside the enum.

The center histogram is the heart of the sim. Each bar is one candidate for the next token, sized by its probability, and the panel header cycles through the phases of a single step: predict, mask, renormalize, sample, commit. During the mask phase a red scanline sweeps across the bars and collapses every candidate the grammar forbids, leaving a dashed outline where its raw probability used to be. During renormalize, the surviving bars grow until they sum to 1 again, and the winning bar flashes bright before its token commits to the document. Every probability here is simulated from seeded logits, not a live model, and the panel is labeled that way.

The right panel is the compiled schema grammar: a state machine with one row per slot, from the opening brace through key, colon, value, and separator for each field, down to an explicit end token. The current slot glows red, finished slots dim, skipped slots get struck through, and violated slots pick up a flag. Below the rail, chips list the legal next tokens; in constrained mode the header reads enforced, in prompt-only mode it drops to advisory only. The stat block in the top right tracks the raw candidate count, how many the mask removed, how many survived, and the leftover probability mass being rescaled to 1.00, plus live JSON VALID and SCHEMA VALID readouts.

A few things to try in the demo:

• Switch to Prompt Only and pick Hostile Logits. The model keeps reaching for a prose preamble or a markdown fence, the parse-error strip lights up on the grammar rail, and JSON VALID flips to FAIL. Runs reseed each time, so some survive and many don't; the footer tally keeps score.
• Try Messy pressure in Prompt Only mode. Messy runs usually stay parseable but pick a wrong key or an enum value the schema never allowed, so the footer shows syntax passing while schema fails. That split is the sneakiest failure mode in production.
• Arm Inject Failure. In Prompt Only mode it forces the next step to sample the highest-probability illegal candidate, so you can watch exactly how one bad token derails a run, or quietly poisons it.
• Flip back to Schema Constrained and leave Hostile Logits on. The sweep deletes the junk on every single step and both verdicts read PASS no matter how bad the raw distribution gets. That asymmetry is the whole argument for the technique.
• Push the temperature slider to 2.00 in each mode. Constrained, the bars flatten but the output stays valid. Prompt-only, that flat distribution is exactly how low-probability junk gets sampled.

How does constrained decoding actually work?

Each step starts with the model scoring a handful of candidate continuations. A softmax with the current temperature turns those scores into the probability bars you see in the predict phase. In this sim the scores are hand-set and shifted by the pressure preset: cooperative boosts the legal candidates, messy boosts tokens that parse but violate the schema, and hostile boosts tokens that break JSON outright.

The schema is compiled ahead of time into the state machine on the right rail. For these three-field schemas that is fourteen rows: an opening brace, then key, colon, value, and separator for each field, then a closing brace and an end-of-sequence token. At any moment the grammar knows exactly which tokens are legal next, and at the value slots it also encodes the type: a quoted string for name, a bare number for age, one of three enum values for role.

In constrained mode, the mask runs between prediction and sampling. Every candidate outside the legal set for the current slot gets its probability set to exactly zero, and the survivors are divided by the mass that remains so they sum to 1 again. Sampling then happens only over the survivors. That is the entire trick: an illegal token is not discouraged or penalized, it is unsampleable. The renormalization also preserves the model's preferences, because the legal candidates keep the same ratios to each other they had before the mask.

In prompt-only mode the sample runs over the raw distribution, so failure is a matter of odds. The sim keeps two failure classes separate. Hard failures break JSON syntax: an unquoted key, a missing colon, a markdown fence, or a prose opener like "Sure". Once prose starts there is no legal recovery in this simple grammar, so the run derails into filler words until it hits a stop token. Soft failures are sneakier: "username" instead of "name", "superuser" when the enum allows admin, editor, or viewer, or a quoted "34" where a bare number belongs. Those runs finish, parse cleanly, and still fail schema validation.

Real systems play the same trick with more machinery. A production tokenizer emits subword pieces, so the engine has to track grammar legality across partial lexemes, and libraries typically compile a JSON Schema into a token-level automaton before generation begins. Masking during sampling is not the only option either: some stacks repair or re-ask after generation instead of intervening in the loop. This is the machinery behind the JSON and structured output modes that LLM APIs expose, and it is what makes reliable tool calls possible, since a tool's arguments are just JSON validated against a schema.

The guarantee stops at structure. A schema-valid record can still claim the wrong age or pick a plausible but wrong enum value, which the demo's caveats dialog spells out. Constrained decoding sits in the same sampling toolbox as top-k and top-p sampling and temperature, and it composes with them rather than replacing them. All of it rides on the token-by-token loop covered in our autoregressive generation demo.

Frequently asked questions

Key takeaways

• Constrained decoding does not persuade the model to behave. It zeroes the probability of every token the grammar forbids and renormalizes the survivors, so an illegal token has exactly zero chance of being sampled.
• Prompt-only failures come in two kinds, and retries only catch one. Hard failures (prose preambles, unquoted keys, markdown fences) break the parser and get noticed; soft failures (a wrong key, a string where a number belongs) parse cleanly and slip through.
• Temperature stops being dangerous once the mask is on. Raising it spreads probability across the legal candidates only, so you get more varied output without ever leaving the grammar.
• The guarantee is structural, not semantic. A schema-valid document can still carry a wrong or useless value, so constrained decoding replaces JSON repair code, not validation of meaning.
• Production grammar engines work on subword pieces, not whole lexemes like this sim, so they compile the schema into a token-level automaton that tracks legality across partial strings.