# Pro Trailblazer: full content > Practical guides with interactive demos covering LLMs, AI workflows, > and tooling for freelancers and small teams. Index: https://protrailblazer.com/llms.txt --- title: "TypeSafe after Claude Code: verify, then escalate" description: "Four typed checks on each change a coding agent makes, then code decides auto-merge, confirm, review, or block. Interactive demo with adjustable floors." published: 2026-09-17 updated: 2026-09-17 canonical: https://protrailblazer.com/posts/typesafe-after-claude-code/ tags: [typesafe, claude-code, demo, agents, code-review, verification, confidence-routing, system-one, developer-tools] --- # TypeSafe after Claude Code: verify, then escalate **Answer:** A verify gate runs after your coding agent finishes and before anything merges. Each change goes to TypeSafe's Jev model with four yes/no questions: does the diff do what the brief asked, does it touch anything protected, do the tests cover it, and does the commit message tell the truth. The answers come back as probabilities. Code turns the weakest one into a margin and applies a five-line policy: clean and confident merges on its own, protected or untested work gets a one-click confirm, torn answers go to a person, and confident failures are blocked. You review the doubtful fifth instead of all of it. The interactive below runs the gate on changes shaped like the ones Claude Code produces for this site: a Safari modal fix with a test, a rename that quietly reformatted 39 files, a secret rotation that committed the secret. Watch the four checks land, see which policy line fires, then drag the two floors and watch outcomes re-sort with no new calls. *Interactive demo: [open the typesafe verify demo](https://protrailblazer.com/demos/typesafe-verify/)* ## What you're seeing The left window is Verify. The card at the top is the change: the commit message as its title, the brief it was supposed to satisfy, and a one-line diff summary with file counts and whether tests were added. The state block is what goes to the model: brief, message, diff, and test output. Then four rows, one per check. Each is a noul question, TypeSafe's yes/no type, and the answer is a single probability of yes. The row shows a yes or no chip, the probability, a margin, and the word "weakest" on whichever check is least decisive for this change. The right window is the Router. The policy block is five lines, with the line that fired highlighted in red. The outcome counts chart shows how judged changes have split across auto-merge, confirm, review, and block. The recent list shows the newest changes with their outcome chip. The Tally across the top tracks changes checked, the auto-merge rate, how many still need a person, and the total verify cost, which after 25 changes is still under a thousandth of a dollar. The four outcomes are four different amounts of your attention. Auto-merge is none: the change matched the brief, touched nothing protected, has tests, and the message is honest, all with margin to spare. Confirm is one click: the change is clean and decisive but it touches something protected (payments, auth, secrets, a destructive delete) or has no tests, so a person should see it go in. Review is a real look: at least one check came back torn. Block is a confident no: the diff does not do what the brief asked, or the message misdescribes it, and the model is sure. A few things to try: - **Drag the review floor from 0.60 down to 0.30.** Two changes leave review: the thumbnail fix that solved a hang by adding a 30 second sleep drops to block, and the Astro version bump rises to auto-merge. Both were sitting at margins around 0.4 to 0.5, decisive enough for a lower floor, not for the default. - **Drag the confirm floor from 0.85 down to 0.60.** The dependency bump that pulled in a major version of an auth library moves from review to confirm. Its checks all lean the right way at around 0.85, which is short of the default bar for protected changes. - **Step to the rename that reformatted 39 files.** on_brief comes back near 0.10 and msg_accurate near 0.12. The commit message says "rename"; the diff says otherwise. That is a block, and it is the change most likely to slip past a tired reviewer. - **Watch for the "weakest" tag.** It moves between checks from change to change, and it is the only number the policy actually compares against the floors. - **Read the newsletter form.** The model is torn on whether it matches the brief and whether the message is honest, and the protected check is a firm yes. Nothing about that should merge on its own. ## How four checks become one decision Each check is a noul question. The answer is a probability of yes, and there is no separate confidence field for a noul: a value near 0.5 means the model is torn, not that the answer is "medium". So the first thing code does is turn each probability into a margin, the distance from 0.5 doubled, which runs from 0 (torn) to 1 (certain either way). The change's margin is the smallest of its four. A change with three certain answers and one coin flip is a coin flip. ``` const margin = (p) => Math.abs(p - 0.5) * 2; const checks = ["on_brief", "protected", "tested", "msg_accurate"]; const m = Math.min(...checks.map((k) => margin(a[k].noul))); const failHard = a.on_brief.noul < 0.5 || a.msg_accurate.noul < 0.5; if (failHard) return m >= REVIEW_FLOOR ? "block" : "review"; if (m < REVIEW_FLOOR) return "review"; const needsHuman = a.protected.noul >= 0.5 || a.tested.noul < 0.5; if (!needsHuman) return "merge"; return m > CONFIRM_FLOOR ? "confirm" : "review"; ``` The order encodes the priorities. A confident failure on the brief or the message is a block before anything else is considered. Anything torn goes to a person before the cheaper outcomes are even evaluated. Only then does the policy ask whether the change is the kind that can merge on its own. That structure follows the confidence-routing pattern in TypeSafe's docs, which suggests a floor around 0.6 below which every action goes to a human, and a higher bar, around 0.85, for actions with consequences. The questions themselves are worth a look, because the whole gate rests on them. This is the request the demo follows, trimmed to two checks: ``` { "model": "jev-latest", "state": { "brief": "Escape should close the term modal in Safari", "message": "fix: close term modal on Escape in Safari", "diff": "", "test_output": "" }, "questions": { "on_brief": { "type": "noul", "instructions": "Does `diff` do what `brief` asked, and nothing else?", "criteria": { "true": "Every hunk serves the brief", "false": "Unrelated edits, reformatting, or a different fix than the brief describes" } }, "msg_accurate": { "type": "noul", "instructions": "Does `message` describe what `diff` actually does?", "criteria": { "true": "A reviewer reading only the message would not be surprised by the diff", "false": "The message understates, overstates, or mislabels the change" } } } } ``` Two things about this request. The backticked field names point the model at specific parts of the state, which is the convention the docs use for nested context. And the criteria are contrastive: they say what counts as false, not just what counts as true, which is what stops a reformatting-heavy diff from passing as "on brief". ## Why the floors are the only knobs Every answer the model returns is stored as a probability. The floors are comparisons against those probabilities. When you drag a floor, the demo re-applies the five lines to every stored answer, the counts re-sort, and the header above the chart reports "re-sorted N judged · 0 new calls · cost unchanged". Nothing was sent, nothing was spent. This is the property that makes the gate reviewable. The questions and the two floors are the entire policy surface. TypeSafe's guidance for agent-written integrations is to keep exactly those constants in one place, because they are what a person should be editing, and agents are not especially good at writing questions on their own. Everything else in the gate is plumbing. ## Where this fits after a Claude Code run The natural home is a hook or a CI step that fires when Claude Code opens a pull request or finishes a task. The state is the brief you gave it, the commit message it wrote, the diff, and the test runner's output. The docs are clear about sending only the context a question needs, so a large diff should be trimmed to its hunks and file list rather than pasted whole, and the test output to its summary lines. One request, four questions, four probabilities, one of four outcomes. What happens next is ordinary tooling. Merge means the PR merges. Confirm means a one-line message with a button. Review means the PR is assigned to a person with the four answers pasted into the description, which is a better starting point than a bare diff. Block means the PR is closed with the failing checks quoted and the agent is asked to try again from the brief. A human in the loop stays on the outcomes that need one, and the changes that were never going to need one stop waiting in the same queue. Two cautions. The diff is content the model reads, and content that reaches a model is content that can try to instruct it. The gate's answer is that the model never acts, code does, and the worst a hostile diff can do is bias a probability that a floor then catches. See [prompt injection](https://protrailblazer.com/posts/prompt-injection/) for the general shape of that problem. And typed checks do not replace tests. The tested check asks whether tests exist that exercise the change; it does not run them. The runner does. This is the third of three posts. The [first](https://protrailblazer.com/posts/typesafe-before-claude-code/) puts a typed gate before Claude Code runs, and the [second](https://protrailblazer.com/posts/typesafe-instead-of-claude-code/) covers jobs that are decisions rather than code. For how a judge model compares with a set of typed checks, see [LLM-as-judge explained](https://protrailblazer.com/posts/llm-as-judge-explained/). The changes in the demo are shaped after this site's own backlog. > **info:** The request and answer shapes, the model name, the price, and the routing pattern come from the TypeSafe docs. The four probabilities for each of the 25 changes are drawn from a ground truth I wrote per change, with noise scaled by how clear the change is, so the outcome mix (roughly 36 percent auto-merge, 28 confirm, 20 review, 16 block at the default floors) is a property of the authored set, not a measurement. ## Frequently asked questions ### Why not have Claude Code review its own pull request? It can, and a second pass by a capable model (the LLM-as-judge pattern) catches real problems. The gate does something different: it produces four probabilities you can put a threshold on, every time, for a thousandth of a cent, in a third of a second. A prose review has to be read; a probability can be compared. Use both if you like, but the typed check is the one that can decide unattended. ### What does the margin actually measure? How far a yes/no probability sits from the coin-flip point, scaled to 0 to 1. It is computed in code, not returned by the model, because a noul answer has no separate confidence. Taking the smallest margin across the four checks means one torn answer sends the whole change to a person. ### Can a decisive answer still be wrong? Yes. Typed output guarantees the shape, not the truth. That is why confident failures block rather than auto-fix, why protected changes never auto-merge, and why the floors should be tuned against your own history of changes before the merge outcome runs without a person watching. ### What goes into state for a real diff? The brief, the commit message, the diff trimmed to its hunks and file list, and the test summary. The docs recommend sending only what the questions need and naming the parts so questions can point at them with backticked paths. A 4,000-line diff pasted whole is both expensive and worse for the model than its 200 relevant lines. ### Does this replace code review? It replaces the part of code review that was a checklist: did it do what was asked, did it touch something dangerous, is it tested, is the message honest. The judgment calls that remain are the ones routed to review, and they arrive with the four answers attached. ## Key takeaways - A yes/no answer near 0.5 is not "medium". It is the model saying it cannot tell, and the right response is to route, not to round. - The weakest check decides. Three certain answers and one coin flip is a coin flip, which is why the policy takes the minimum margin and not the average. - Protected changes never auto-merge, however confident the model is. The confirm outcome exists so a person spends one click on payments, auth, secrets, and deletes instead of a full review. - The order of the policy lines is the safety story. Confident failures block first, torn answers go to a person second, and only then does anything merge on its own. - Two floors are the whole tuning surface. Moving one re-sorts stored answers for free, which means you can calibrate the gate against last month's merges before you trust it with next month's. --- --- title: "TypeSafe before Claude Code: a typed preflight gate" description: "A preflight gate asks TypeSafe's Jev four typed questions about each dev request, then local code decides whether Claude Code runs at all. Interactive demo." published: 2026-09-17 updated: 2026-09-17 canonical: https://protrailblazer.com/posts/typesafe-before-claude-code/ tags: [typesafe, claude-code, demo, agents, triage, system-one, confidence-routing, developer-tools] --- # TypeSafe before Claude Code: a typed preflight gate **Answer:** A preflight gate is a small piece of code that runs before your coding agent does. It sends each incoming request to TypeSafe's Jev model with four typed questions (what kind of work is this, is it specified well enough, how risky is it, does it need code at all), gets back probabilities instead of prose, and routes the request with four lines of if-statements. Only requests that are ready, low-risk, and actually need code reach Claude Code. The rest bounce back for a clarifying question, get scoped by a person first, or get answered without an agent run. The interactive below runs the gate on requests pulled from this site's own backlog: a Safari modal bug, a Polar webhook returning 500, a Slack message that just says "make the site faster". Each one goes out as a single TypeSafe request with four questions. Watch the four answers fill in, watch which policy line fires, then drag the two thresholds and see every judged request re-sort without a new API call. *Interactive demo: [open the typesafe preflight demo](https://protrailblazer.com/demos/typesafe-preflight/)* ## What you're seeing The left window is the Preflight. At the top is the request card with its source (an issue, Slack, email, or my own todo list). Below it, the state block is the JSON that actually goes to the model: the request text, where it came from, and the repo. Then four rows, one per question. Each row shows the question id, its type, the criteria in plain words, and, once the answer lands, the typed value with a probability bar. The header of that section reports the modeled latency, the input tokens counted, and the cost of that one call. The right window is the Router. The policy block at the top is the entire decision logic, four lines of it, and the line that fired for the current request is highlighted in red. Below it, the route counts chart shows how the judged requests have split so far, and the recent list shows the newest ones with their route chip. The Tally across the top keeps the totals: requests judged, how many reached Claude Code, how many agent runs the gate avoided, and what all of that preflight cost. The four routes map to four things you would do by hand. Dispatch means the request is clear, low-risk, and needs code, so it goes to Claude Code with a scoped brief. Clarify first means the model is not confident the request is specified well enough, so the cheapest move is one question back to whoever asked. Human scopes it means the request is clear but risky (payments, auth, secrets, deleting data), or the model could not tell what kind of work it is, so a person writes the brief before any agent touches it. No code means the answer is a reply or a content task, and spinning up a coding agent for it would be waste. A few things to try: - **Drag the ready threshold up to 0.90.** Requests that were dispatching at 0.70 slide into clarify first. The counts chart re-sorts instantly, the header above it flashes "re-sorted N judged · 0 new calls", and the cost in the Tally does not move. - **Drop the confidence floor to 0.30.** Requests where the kind answer was torn between two options (bug or chore, question or unclear) stop bouncing to a human and start dispatching. Decide for yourself whether you would want that. - **Pause and use Step.** Each click judges one request and stops, so you can read all four answers and the fired policy line before the next one arrives. - **Watch the risk row on the Polar webhook and the secret rotation.** Both are clear and both need code, but the score lands near 2, and the third policy line catches them before they reach an agent. - **Read the state block on "Fix the thing we talked about yesterday".** The model has nothing to work with, and the ready probability says so. ## How the preflight actually works Every request is one HTTP call. The body has three parts: the state (the JSON you saw in the demo), the model name, and a map of questions. Here is the shape the demo follows, trimmed to two of the four questions: ``` { "model": "jev-latest", "state": { "request": "Polar webhook returns 500 on refund events", "source": "issue", "repo": "protrailblazer.com" }, "questions": { "kind": { "type": "choice", "instructions": "What kind of work is `request` asking for?", "criteria": { "bug": "Something that used to work is broken", "feature": "New behavior the site does not have", "chore": "Maintenance with no user-facing change", "question": "Wants an explanation, not a change", "unclear": "Not enough here to tell" } }, "risk": { "type": "score", "instructions": "How much damage could a wrong change do?", "criteria": [ "Cosmetic only", "Logic or build behavior", "Auth, payments, secrets, or data" ] } } } ``` The answers come back typed. A choice answer names the option with the highest probability and includes the whole distribution plus a confidence value that describes how peaked that distribution is. A noul answer (TypeSafe's name for a yes/no question) is a single probability of yes. A score answer is the probability-weighted position on the levels you defined, so a risk of 1.94 means almost all of the weight sat on "auth, payments, secrets, or data". Trimmed to the fields the policy reads: ``` { "answers": { "kind": { "type": "choice", "choice": "bug", "confidence": 0.79, "probabilities": { "bug": 0.86, "feature": 0.07, "chore": 0.04, "question": 0.02, "unclear": 0.01 } }, "ready": { "type": "noul", "noul": 0.91 }, "risk": { "type": "score", "score": 1.94, "probabilities": { "0": 0.01, "1": 0.04, "2": 0.95 } }, "needs_code": { "type": "noul", "noul": 0.97 } } } ``` The response also reports token usage. Jev 1.13 is priced at $0.042 per million input tokens and output tokens are free, so a 290-token preflight call costs roughly a thousandth of a cent. Then the policy, which lives in your code and not in the model: ``` if (a.needs_code.noul < 0.50) return "no-code"; if (a.ready.noul < READY_THRESHOLD) return "clarify"; if (a.risk.score >= 1.5 || a.kind.confidence < CONFIDENCE_FLOOR) return "human"; return "dispatch"; ``` Four questions go out in one request and run in parallel. They cannot see each other's answers, which is fine here because none of them depends on another. The docs put the per-request latency for a call like this in the 0.16 to 0.31 second band, and the demo draws its latency readout from that band. The order of the policy lines matters and is a choice: needs_code is checked first because a question should never be blocked for being risky. ## Why moving a threshold never re-runs inference The model returned probabilities, not decisions. The decision is a comparison your code makes against those probabilities, and the probabilities do not change when you change the comparison. So when you drag the ready threshold from 0.70 to 0.90, the demo loops over every stored answer and re-applies the four lines. No request goes out, no tokens are spent, and the Tally's cost stays put. The header above the counts chart says exactly that each time a slider moves. Compare that with the same gate built by prompting a language model to "triage this and be strict". Getting stricter means rewording the prompt and re-running every request, and you still have to parse whatever comes back. Keeping the judgment typed and the policy in code is what makes the threshold a knob instead of a rewrite. TypeSafe's own guidance is to keep the questions and thresholds in one place so a reviewer can look at those two things instead of the whole implementation. ## Where this fits in a Claude Code session In practice this is a script, a hook, or a small worker that runs when a request lands: a new issue, a Slack message, a line added to a todo file. It needs an API key from the TypeSafe console at [console.typesafe.ai](https://console.typesafe.ai/settings/keys), exported as TYPESAFE_API_KEY, and about thirty lines of code. If you want Claude Code to write those thirty lines, install the TypeSafe skill first: ``` claude plugin marketplace add typesafe-ai/skills claude plugin install typesafe@typesafe-ai ``` The dispatch route is where the answers pay off a second time. The kind and risk values go into the brief that Claude Code receives, so a risk-2 bug can be routed to a stronger model tier with tests required, and a risk-0 chore can run on the cheaper one. The clarify route is worth more than it looks: a request that bounces back with one question costs a thousandth of a cent, while a Claude Code session that starts on an ambiguous brief costs real minutes and usually ends in a rewrite anyway. Keeping a human in the loop for the risky ten percent is the part you would have done anyway. The gate just makes it the default instead of a habit. This is the first of three posts on where a System One model sits around an agentic workflow. The second, [TypeSafe instead of Claude Code](https://protrailblazer.com/posts/typesafe-instead-of-claude-code/), covers jobs that are decisions rather than code. The third, [TypeSafe after Claude Code](https://protrailblazer.com/posts/typesafe-after-claude-code/), puts the same kind of gate on the other end of a run to verify what the agent produced. For the wider picture of how agents chain tasks, see [agentic workflows](https://protrailblazer.com/posts/agentic-workflow-smb-guide/). > **info:** The request and answer shapes, the model name, the price, and the latency band come from the TypeSafe docs. The answers themselves are drawn from a ground truth I wrote for each of the 30 requests, with noise scaled by how clear the request is, so the routing shows the shape of the workflow rather than a measured accuracy. Confidence for the choice answer is modeled as the gap between the top two probabilities. ## Frequently asked questions ### What is a System One model? TypeSafe's term for a model trained to return typed judgments (a choice, a yes/no probability, or a score on levels you define) instead of generating text. Jev is the first one. You send state and questions; you get back probabilities your code can compare against a threshold without parsing anything. ### Why not just ask Claude Code to triage the request itself? It can, and for one request that is fine. The gate is for the stream. A typed call costs about a thousandth of a cent and returns in under a third of a second, so you can run it on every request before deciding whether the expensive session is worth starting. It also gives you probabilities, which a prose triage does not, and probabilities are what let you set a threshold instead of arguing with a paragraph. ### What happens when the model is wrong? The policy is built to fail toward a person. A torn kind answer or a low ready probability sends the request to clarify or to human scopes it, never straight to dispatch. Typed output guarantees the shape of the answer, not that it is right, so the thresholds should be checked against your own requests before you trust the dispatch route unattended. ### Does re-thresholding really cost nothing? Yes, as long as the questions and the state stay the same. The answers are stored probabilities and the threshold is a comparison in code. Changing a weight or a filter does not need a new inference call. New questions or new state do. ### How many questions can one request carry? As many independent ones as you need. They run in parallel inside one call and cannot see each other's answers, and the docs note that adding questions barely changes the response time. Speculative questions are fine too: ask the branch-specific question up front and only consume the answer if that branch is taken. ## Key takeaways - Preflight is cheap because the model never writes anything. A four-question call costs roughly a thousandth of a cent and returns a set of probabilities, which is a different economic unit from a session that reads files and writes code. - The policy is four lines of code, and it is the whole product. Reviewers should read the questions and the thresholds, not the integration. - Thresholds are knobs because answers are probabilities. Moving a threshold re-sorts stored answers; it never re-runs inference. A prompt that says "be stricter" cannot do that. - The order of the policy lines is a decision. Checking needs_code before risk means a question is never blocked for being dangerous, which is exactly the behavior you want and exactly the kind of thing a prompt gets wrong on a bad day. - The clarify route is the cheapest win. One question back costs nothing; a session started on an ambiguous brief costs the session. --- --- title: "TypeSafe instead of Claude Code: judgment, not generation" description: "The same 80 messages triaged by an LLM agent loop and by a System One model returning typed answers. Compare wall time, cost, and malformed outputs live." published: 2026-09-17 updated: 2026-09-17 canonical: https://protrailblazer.com/posts/typesafe-instead-of-claude-code/ tags: [typesafe, claude-code, demo, agents, structured-output, system-one, batch, developer-tools] --- # TypeSafe instead of Claude Code: judgment, not generation **Answer:** Some jobs you hand to a coding agent are not coding jobs. Labeling 80 messages, ranking candidates, flagging what needs a reply: each one is a decision, not a program. A System One model like TypeSafe's Jev answers those decisions directly as typed values with probabilities, so there is no text to generate and nothing to parse. The demo below races that against a generate-then-parse agent loop on the same batch and shows where the time, the cost, and the malformed outputs come from. The interactive below drops the same 80 inbound messages into two lanes. The agent lane prompts a language model, waits for it to write out an answer token by token, parses the result, and occasionally has to retry a malformed one. The System One lane sends each message with its questions and gets typed answers back. Both lanes run three workers. Watch the Scoreboard, then add questions per item and see which lane notices. *Interactive demo: [open the typesafe batch judgment demo](https://protrailblazer.com/demos/typesafe-batch-judgment/)* ## What you're seeing The top-left window is the agent loop. Each of its three rows is a worker holding one message, with the phase it is in (prompt, gen with a token count, parse) and a progress bar. When a worker hits a malformed output, the phase label turns red and reads "malformed, retry", and the worker goes back to generating. Below it, the System One window has the same three workers, but each one only has two phases: request and typed answer. On an ordinary run, that lane finishes the whole batch while the agent lane is still in its first quarter, then its title bar turns red and its rows read "done". The Scoreboard compares the two lanes on a shared scale per row: wall time so far, cost so far, and malformed outputs. The System One malformed count is zero by construction, because the answer arrives already in the schema you asked for. The Console on the right shows the most recent typed answer: the message as state, the questions that went with it, and each answer with its probability bar. The Tally keeps the running totals for both lanes side by side. The questions are the interesting knob. At one question per item, each message only gets a label (lead, support, billing, spam, other). At six, it also gets urgency, sentiment, whether it expects a reply, whether it mentions money, and a priority score. The agent lane has to write all of that out. The System One lane answers it in the same call. A few things to try: - **Slide questions per item from 2 to 6.** The agent lane's per-item time and cost roughly triple, because every extra field is more output to generate. The System One lane's request time moves by hundredths of a second, and the batch is done before you can read the Tally. - **Push the malformed rate to 15 percent.** Red retry labels appear in the agent lane and the malformed count climbs. Each retry is a full regenerate, so it costs time and output tokens twice. - **Set the malformed rate to zero.** Even a perfect parser does not close the gap. The gap is generation, not parsing. - **Read the Console while the System One lane runs.** The label answer carries the whole probability distribution and a confidence value. A message that is half lead and half billing shows up as a spread, not a guess. - **Press Replay when both lanes finish.** The batch restarts and the Scoreboard resets, so you can compare a second run with a different setting. ## How the two lanes are modeled Nothing in the demo calls a model. Both lanes are timing and cost models with their constants printed at the bottom of the Scoreboard, and the point is the shape of the difference, not the exact figures. - **Agent lane.** 0.7 seconds to the first token, then 55 tokens per second. Output is modeled as 24 tokens plus 26 per question (a JSON object with one field per question). Input is 320 tokens plus 40 per question. Priced at $3 per million input tokens and $15 per million output, which is a typical frontier-model price sheet rather than a specific vendor. A malformed output (4 percent by default) pays for a full second generation. - **System One lane.** 0.16 seconds per request plus 0.02 per extra question, inside the 0.16 to 0.31 second band TypeSafe reports for its cookbooks. Input is about 90 tokens plus 52 per question, since the criteria travel with the request. Priced at Jev 1.13's published $0.042 per million input tokens, with output tokens free. - **Both lanes.** Three concurrent workers, 80 messages, and a simulated clock that runs 2.5 times faster than real time so the agent lane finishes in about twenty seconds instead of a minute. Three workers keep the System One lane well inside the published rate limit of 1,200 requests per minute, which the docs note is adjusting dynamically under demand. At the defaults the batch costs a few cents in the agent lane and a few hundredths of a cent in the System One lane. The ratio is large, but the more useful number is per item: about a hundredth of a cent for typed answers to two questions, which is the scale at which you stop thinking about whether to ask. ## Why adding a question barely costs anything A language model produces an answer by writing it. Every field you ask for is more tokens to generate, and generation is the slow, expensive part of inference. A System One model does not write. It reads the state once and returns a probability for each question, and the questions are evaluated in parallel inside one request. That is why the docs say adding questions barely changes the response time, and why the demo's request time moves by hundredths of a second between one question and six. The malformed column is the other half of the story. When you ask a language model for JSON you get text that is usually JSON, and the usual fixes are a retry loop, a schema-repair pass, or [constrained decoding](https://protrailblazer.com/posts/constrained-decoding-json/) at the sampler. A typed answer skips the problem: the API defines the shape, so there is nothing to parse. What it does not skip is being wrong. TypeSafe's own docs put it plainly: typed output guarantees the interface, not the truth. The zero in that column means zero parse failures, not zero mistakes, and the probabilities exist so your code can route the uncertain ones instead of trusting them. ## When to reach for this instead of an agent The word "instead" is about the job, not the tool. Claude Code is the right tool when the output is code or prose, or when the work needs a sequence of tool calls that only makes sense after reading the previous result. A System One model is the right tool when the output is a decision over state you already have: a label, a score, a yes or no, a pick from a list. Most triage, routing, ranking, flagging, and extraction work is that second kind, and a lot of it ends up in an agent session only because the agent was already open. Reach for typed judgments when: - You have the state in hand and the answer is one of a defined set. Sorting messages, ranking candidates, deciding which handler runs, checking whether a field is present. - You would otherwise prompt-and-parse. If your code already has a regex around a model's output, that is the signal. - You need probabilities to act on. A threshold on a probability is a policy; a paragraph is not. - Volume matters. At a hundredth of a cent per item you can judge everything, including the things you used to sample. TypeSafe's suggested way to find these spots is to hand the search to the agent: ask Claude Code, with the TypeSafe skill installed, to explore the codebase for parsing and branching logic that a small judgment could replace, then run a few experiments against the API before proposing changes. The skill install is two commands (see the [preflight post](https://protrailblazer.com/posts/typesafe-before-claude-code/)). Human review then focuses on the questions and the thresholds, which the docs describe as the part agents are least good at writing. The [first post in this series](https://protrailblazer.com/posts/typesafe-before-claude-code/) puts a typed gate in front of Claude Code so only ready, low-risk work reaches it. The [third](https://protrailblazer.com/posts/typesafe-after-claude-code/) puts one after it to verify what came out. For how an LLM-as-judge differs from a typed check, see [LLM-as-judge explained](https://protrailblazer.com/posts/llm-as-judge-explained/). > **info:** Both lanes are timing and cost models. The System One numbers come from the TypeSafe docs (latency band, Jev 1.13 pricing, rate limit); the agent-lane numbers are illustrative and no vendor is named. The typed answers in the Console are drawn from a ground truth I wrote for each of the 24 messages in the batch, so they show the shape of the answers, not a measured accuracy. ## Frequently asked questions ### Is Jev a language model? TypeSafe calls it a System One model: it reads natural language state and returns typed answers with probabilities rather than generating text or reasoning. The docs describe it as trained for calibrated decisions, and the whole point of the design is that there is no output to parse. ### How accurate are typed answers? That depends on your questions and your domain, and the docs are explicit that you should validate on your own data. One published example: adding two TypeSafe calls to pick a skill for an agent turn cut wrong skill loads from 16.8 percent to 7.3 percent across 488 requests, at 0.25 to 0.43 seconds added per turn. Treat numbers like that as a shape to check, not a promise. ### Why is the agent lane's malformed count nonzero even at a low rate? Because over 80 items a 4 percent rate produces about three retries, and each retry regenerates the whole answer. The slider lets you set it to zero to see that parsing was never the main cost. ### Can I run the batch from Claude Code? Yes. With the TypeSafe skill installed and TYPESAFE_API_KEY set, Claude Code can write the script that sends each item with its questions. The docs recommend keeping the questions and thresholds in one file so they are easy to review and edit collaboratively. ### Where do I get an API key? Sign in at console.typesafe.ai and create a key under settings. Export it as TYPESAFE_API_KEY. The Python SDK installs with pip install typesafe-sdk, and the raw HTTP call is a single POST to /v1/systemone. ## Key takeaways - Generation is the cost, not parsing. Set the malformed rate to zero and the gap barely moves, because a language model still has to write the answer out. - Questions are nearly free on the System One side. They run in parallel inside one request, so the marginal question costs hundredths of a second and a few dozen input tokens. - Zero malformed means zero parse failures, not zero mistakes. The probabilities are there so code can route the uncertain answers instead of trusting all of them. - "Instead of Claude Code" is a statement about the job. If the output is a decision over state you already have, an agent session is the wrong shape for it. - Find the spots by asking the agent to find them. The exploration prompt in the TypeSafe skill is designed for exactly that search. --- --- title: "Why ChatGPT types one word at a time: streaming visualized" description: "Watch a model write tokens one at a time while streaming rushes each one to your screen and buffered mode holds the whole answer behind a gate." published: 2026-07-26 updated: 2026-07-26 canonical: https://protrailblazer.com/posts/why-chatgpt-types-one-word-at-a-time/ tags: [machine-learning, demo, streaming, llm, ttft, tokens, inference, chatgpt] --- # Why ChatGPT types one word at a time: streaming visualized **Answer:** ChatGPT types one word at a time because the model underneath genuinely writes its answer one token at a time, and streaming pushes each token to your screen the moment it exists instead of holding the finished answer back. The typing effect is honest delivery, not a loading animation. The demo below runs the same generation through a streaming lane and a buffered lane side by side, so you can watch one start painting almost immediately while the other shows nothing until the very last token is done. The interactive below runs a full token delivery pipeline on a shared clock. Pick a prompt preset or dial in your own parameters, then switch between streaming, buffered, and compare modes to see what actually changes between them. *Interactive demo: [open the llm streaming demo](https://protrailblazer.com/demos/llm-streaming/)* ## What you're seeing The scene runs left to right through four stages. On the left, a MODEL block produces tokens on a clock, one at a time, with a live WRITING readout and a grid of produced-token chips filling in as generation advances. The tokens travel rightward through one wire (streaming mode) or two wires (compare mode). In STREAMING, each chunk departs as soon as its last token exists and lands immediately in the UI pane, painting text character by character. In BUFFERED, the same tokens pile up behind a gate and arrive as one block the moment the model finishes. A latency waterfall strip at the bottom anchors both timelines to the same generation span. The HUD across the top tracks six live numbers: time to first token (TTFT), tokens per second, network delay, tokens generated, tokens displayed, and total time. In compare mode, each lane gets its own FIRST/TOTAL stopwatch. The token boundary underlines on delivered text show where one token ends and the next begins. Punctuation, sub-word fragments like "stream" and "ing", and standalone commas each arrive as their own unit, which is why the typing rhythm is uneven. A few things to try: - **Switch to COMPARE mode and press play.** Both lanes generate from the same clock, so the token grid fills at identical speed. Watch the streaming pane paint text long before the buffered pane shows anything. The waterfall strip at the bottom makes the timing difference concrete. - **Try the LONG EXPLANATION preset.** The hatched PREFILL span in the waterfall grows noticeably. A long prompt forces the model to read your whole input before writing the first token, stretching TTFT for both lanes even when the network never changes. - **Drag the network latency slider to 400 ms, then enable SIMULATE STALL.** Generated tokens queue up behind the departure point, the wire flashes STALL and the NET DELAY stat spikes, then all queued tokens burst out at once. This is what happens when your connection hiccups mid-response. - **Change the chunk size from 1 to 8 tokens per packet.** The streaming pane now paints in visible bursts instead of one token at a time, even though the model is still generating at the same rate. Many providers batch tokens before sending. - **Use STEP TOKEN to advance one event at a time.** You can watch each token write to the grid, depart on the wire, and land in the UI pane as three distinct moments. ## How LLM streaming actually works When you send a prompt to a chat model, three things happen before you see any text. First, the model runs a forward pass over your entire prompt to build up internal context, a phase called prefill. This is the part that scales with prompt length: a 3,000-token prompt takes longer to prefill than a 50-token one, regardless of how fast the model can generate afterward. After prefill, the model generates one token per forward pass. It samples from the distribution over its vocabulary, writes the chosen token, appends it to the context, and runs another forward pass to pick the next one. This repeats until the model writes a stop token or hits the generation limit. The KV cache stores the key and value tensors from each previous token so the model does not recompute them on every pass; without it, generating token 500 would be almost 500 times slower than generating token 1. Streaming means the API does not wait for this loop to finish. The moment a token is sampled, the server wraps it in a Server-Sent Events packet and sends it over the open HTTP connection. Your client receives and renders it immediately. Buffered mode is the same loop running on the same schedule, but the server accumulates all tokens before sending the first byte of the response body. The practical result: streaming does not finish the answer any faster. Both modes complete generation at the same instant. Streaming only moves the first paint earlier. For a 200-token answer at 30 tokens per second, the buffered user waits roughly 6.7 seconds before seeing anything, while the streaming user sees the first token in under a second. The answer finishes at the same moment, but the experience of waiting is completely different. The size of the context window affects both prefill time and KV cache memory. A long conversation history or a large system prompt all have to be prefilled before the first generated token arrives, which is why TTFT often grows noticeably in long chat threads. For more on how the generation loop chooses which token to write, see the [next-token generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text/). For how compute budget shapes latency and quality, see [inference-time compute](https://protrailblazer.com/posts/inference-time-compute-explained/). ## Why apparent typing speed is not model intelligence The speed at which text appears on your screen is determined almost entirely by delivery settings, not by the quality or depth of the model's reasoning. A slower typing effect usually means higher network latency, larger chunk sizes batched before send, or a provider-side rate limit, not a smarter model thinking harder. You can verify this in the demo: drag the generation speed slider to its minimum and maximum and confirm the answer still finishes in exactly the same number of tokens. Real providers add further variation. Safety classifiers may scan chunks before release. Request batching means your generation shares GPU time with other users. Variable SSE packet framing means chunk sizes are not uniform across a response. The caveats panel in the demo covers these gaps. What the demo models accurately is the structural split between prefill, autoregressive generation, and network delivery, and the fundamental reason why streaming and buffered mode have the same completion time but different first-paint times. ## Frequently asked questions ### Does streaming make ChatGPT finish its answer faster? No. Streaming and buffered mode complete generation at the same instant because the model runs the same autoregressive loop either way. Streaming only moves the first painted character earlier. The total time from submit to last token is identical; the perceived wait time is much shorter because you are reading while the model is still writing. ### What is time to first token and why does it matter? TTFT is the gap between sending your prompt and receiving the first byte of the response. It is dominated by prefill (reading your prompt) plus network round-trip. TTFT is the metric that makes a chat interface feel responsive or sluggish, because it controls how long the screen shows nothing. ### Why does a longer prompt slow down the response? The model has to run a forward pass over every token in your prompt before it can write the first reply token (the prefill phase). A 3,000-token prompt has 60 times as many tokens to prefill as a 50-token one, so TTFT scales roughly linearly with prompt length at fixed hardware and batch conditions. ### Is a token the same as a word? Not always. Common short words often map to a single token, but longer or less common words split across multiple tokens. Punctuation and whitespace are usually separate tokens. The demo uses hand-authored token arrays with explicit sub-word splits ("stream" + "ing", standalone commas) to illustrate this. The boundary underlines on delivered text show exactly where each token ends. ### What causes the burst of text after a stall? The model never stops generating during a network stall. Tokens pile up at the departure point behind the delivery bottleneck. When the connection recovers, all queued tokens send in rapid succession, producing the burst effect. The sim captures this with the SIMULATE STALL toggle, which blocks departures for a window then releases the queue at once. ## Key takeaways - The typing effect is not a loading animation. The model really does produce one token at a time, and streaming delivers each token as soon as it exists instead of waiting for the last one. - Streaming does not finish the answer any faster. Both lanes complete generation at the same instant; streaming only moves the first paint earlier, and that earlier first paint is the entire reason chat feels responsive. - Time to first token has a hidden component: prefill. The model must read your whole prompt before it can write anything, so a long prompt stretches TTFT for both lanes even when the network never changes. - A token is not a word. Punctuation, sub-word fragments, and code symbols each ride the stream as their own token, which is why the typing rhythm looks uneven. - Apparent typing speed measures delivery settings, not model intelligence. Chunk size, network latency, and stalls shape what you see; a stalled connection makes any model look frozen, then dump a burst of queued text. --- --- title: "What does 7B mean? LLM parameter counts explained" description: "The B in 7B counts the learned weights stored in a model. See how count, precision, and architecture determine file size in this interactive scale tunnel." published: 2026-07-24 updated: 2026-07-24 canonical: https://protrailblazer.com/posts/what-does-7b-mean/ tags: [machine-learning, demo, llm, parameters, quantization, mixture-of-experts, model-size, fp16, int4] --- # What does 7B mean? LLM parameter counts explained **Answer:** The B in 7B stands for billion, and it counts the learned weights stored inside the model, not facts, tokens processed, or neurons in the biological sense. A 7B model holds exactly 7,000,000,000 floating-point numbers that were adjusted during training to minimize prediction error. Those numbers take up memory proportional to how many bits each one is stored with: at FP16 (16 bits per weight) a 7B model occupies about 14 GB just for the weights, and switching to INT4 (4 bits) drops that to roughly 3.5 GB without changing the count at all. Parameter count is a measure of size, not capability. The interactive below draws six models from 1B to 400B parameters as tile fields on a shared canvas, where tile area is exactly proportional to parameter count. Pick a model, change the precision, switch to MoE, or run the story presets to see how count, bits, and architecture combine into a weight file size. *Interactive demo: [open the llm parameter counts demo](https://protrailblazer.com/demos/llm-parameter-counts/)* ## What you're seeing The canvas places all six model sizes on a shared horizontal baseline. Each model is a rectangular tile field whose area is proportional to its parameter count, so 7B is literally 7 times the area of 1B. Same-size individual tiles hold the same number of weights wherever they appear in the scene. A count sweep ticks through the selected model's weights in real time, and the number in the HUD top-left (TOTAL PARAMS) shows the exact integer, not a rounded approximation. Below the tile fields runs a memory ruler. It converts the selected model's parameter count and the current precision setting into an exact weight file size using the formula: params x bits per weight / 8. Reference ticks for consumer and server hardware appear along the ruler as labeled guides. The formula is exact (1 GB = 10^9 bytes, as stated on the ruler). Running a model needs more memory than the weight file alone (the KV cache, activations, and buffers all add on top), but the weight file is the anchor. The HUD top-right tracks seven live values: total parameters, active parameters per token (which differs from total only in MoE mode), bits per weight, weight file size, relative scale versus 1B, dense or MoE architecture, and the count sweep progress. The NOT strip in the bottom-left panel crosses out FACTS, TOKENS, NEURONS, and LINES OF CODE to make explicit what the number does not measure. A few things to try: - **Run the "Quantize 7B" story preset.** Watch TOTAL PARAMS stay fixed at 7,000,000,000 while WEIGHT FILE collapses from 14 GB (FP16) down to 3.5 GB (INT4). The tile field does not change size. Only the bits column shrinks. - **Switch to "MoE 400B" and compare TOTAL PARAMS with ACTIVE / TOKEN.** The 400B MoE stores all expert weights but routes each token through roughly 32.5% of them (8 experts, top-2 routing, illustrative). The tile field shows the full 400B footprint; the HUD shows how much of it fires per forward pass. - **Run "70B vs 7B" and use Fit All to see both models at once.** The 70B tile field is 10 times the area of 7B. Then change the 70B to INT4 and the 7B to FP32 and read the memory ruler: the 70B at INT4 (35 GB) is smaller than the 7B at FP32 (28 GB). File size is not the same as parameter count. - **Toggle the Training Memory band on.** The x3 to x8 range above the weight file shows the rough envelope of training memory (gradients, optimizer states, activations). This band is labeled ILLUSTRATIVE because the real multiplier depends heavily on batch size, optimizer, and distributed sharding strategy. - **Use the zoom controls or drag the canvas to read tile counts at different abstraction levels.** Zooming in shifts the granularity so each tile represents fewer parameters. At the finest zoom, the matrix microscope at the tunnel's far end resolves into individual stored numbers with values labeled ILLUSTRATIVE (they re-encode when precision changes, and the count stays unchanged). ## How parameter counts actually work A language model is, at its core, a large collection of transformer layers, each containing matrices of floating-point numbers. During pretraining, those numbers are updated repeatedly via backpropagation to minimize the error the model makes when predicting the next token. The total count of those numbers is the parameter count. The count is fixed after training. Nothing about running the model (the inputs you give it, the length of the conversation, the temperature setting) changes the number of weights stored in the file. What changes at inference is how those weights are read: during a forward pass, each weight is loaded from memory, multiplied with an activation, and discarded from the compute unit. The memory pressure during inference comes from holding all the weights in VRAM at once, not from repeatedly writing new ones. The file size formula is arithmetic: weight file size in bytes = parameter count x bits per weight / 8. A 7B model at FP16 (16 bits) uses 7,000,000,000 x 16 / 8 = 14,000,000,000 bytes = 14 GB. At INT4 (4 bits) the same model uses 3.5 GB. At FP32 (32 bits) it uses 28 GB. The number 7,000,000,000 appears in all three equations. Only the bits column changes. For a walkthrough of what happens to those weights once the model starts generating text, see [how LLMs generate text](https://protrailblazer.com/posts/how-llms-generate-text/). ## Why MoE models report two different numbers A mixture-of-experts architecture replaces each dense feed-forward layer with a set of expert sub-networks and a routing mechanism. The router selects a small subset of experts to process each token. The headline parameter count reflects all experts stored in the file. The active parameter count reflects only the experts that fire on any given token. The demo models this with an illustrative 8-expert, top-2 routing configuration plus a shared expert layer. At 10% of parameters shared and the top-2 of 8 specialized experts active, roughly 32.5% of the total parameter count is used per token. This is labeled ILLUSTRATIVE because real MoE models vary in expert count, top-k, and share ratio by architecture and layer. The practical consequence is that sparse activation lets a MoE model store far more total capacity than a dense model of the same compute budget. A 400B MoE may activate only as many weights per token as a 130B dense model, while having access to the full 400B of trained knowledge across the expert pool. But you still need enough VRAM to hold all 400B weights if the model is not split across multiple devices. The [mixture-of-experts explainer](https://protrailblazer.com/posts/mixture-of-experts-explained/) covers routing strategies and real-world tradeoffs in more depth. ## Frequently asked questions ### Does a 70B model know 10 times as many facts as a 7B model? Not in any direct sense. Parameter count is a capacity measure, not a knowledge inventory. A larger model has more degrees of freedom to fit training data, but what it retains depends on the training data itself, the architecture, the learning rate schedule, and how the model was aligned or fine-tuned. Two models with identical counts can differ substantially in what they know or can do. ### Why does the parameter count stay the same when I change precision? Precision determines how many bits each parameter is stored with, not how many parameters there are. Switching from FP16 to INT4 changes the number from 16 bits wide to 4 bits wide, but there are still 7,000,000,000 of them. The file gets smaller because each one takes less space, not because any are removed. ### What is the difference between total parameters and active parameters per token? In a dense model they are the same: every weight participates in every forward pass. In a mixture-of-experts model, the architecture stores multiple expert sub-networks but only routes each token through a subset at inference time. The total count is what you store; the active count is what you compute with per token. A 400B MoE might activate only 130B of its parameters per token depending on the routing configuration. ### How much memory does it actually take to run a model? The weight file is the floor, not the ceiling. Running a model also requires the KV cache (grows with context length), intermediate activations, and framework overhead. A rough rule for inference is 1.2 to 1.5 times the weight file for short contexts. Longer contexts expand the KV cache further. Training adds optimizer states and gradients on top, which is why the demo's training band spans x3 to x8 of the weight file. ### Why do some model cards list two parameter counts? Usually because the model is a mixture-of-experts architecture. The first number is total parameters (everything stored in the file). The second number is active parameters per forward pass (what actually computes during inference). If you see something like "400B total, 52B active," the model stores 400B weights but only uses about 52B per token. ## Key takeaways - Parameter count and weight file size are linked by arithmetic, not heuristics: multiply the count by bits per weight, divide by 8, and you have the exact weight storage in bytes. A 7B FP16 model is 14 GB; a 70B INT4 model is 35 GB. No estimation needed. - Quantization changes bits per weight and therefore file size, but the parameter count printed on the model card never moves. The HUD in the demo shows this directly: TOTAL PARAMS stays fixed while WEIGHT FILE collapses as you drop from FP16 to INT4. - MoE models report total parameters across all experts but activate only a fraction per token. A 400B MoE with 8 experts and top-2 routing computes with roughly 32.5% of its weights per token, which is why a 400B MoE can run faster than a 70B dense model at similar quality points despite the headline count. - The parameter count on the model card says nothing about what the model can do. Two models with identical counts can differ wildly in capability depending on architecture, training data, fine-tuning, and the precision of activations and the KV cache at inference. - Running a model costs more memory than storing its weights. The KV cache, activations, and optimizer states during training all add on top. The demo's training band (an illustrative x3 to x8 multiplier) shows how far training memory can exceed the raw weight file. --- --- title: "Self-attention: how a token decides which tokens to read" description: "See how self-attention works. Tap any word to make it the query and watch scaled dot-product attention weight every other word in the sentence." published: 2026-07-21 updated: 2026-07-21 canonical: https://protrailblazer.com/posts/self-attention-explained/ tags: [machine-learning, demo, attention, transformers, self-attention] --- # Self-attention: how a token decides which tokens to read **Answer:** Self-attention is how a language model lets every word in a sentence look at every other word and decide which ones matter for its own meaning. Each token sends out a query, every token answers with a key, and the match scores become weights that blend the other tokens' values into a new, context-aware version of that token. It is the step that lets a model work out which noun the word "it" is standing in for. The interactive below runs real scaled dot-product self-attention over a sentence you can edit. Tap any word to make it the query, and watch weighted arcs fan out to every other word, thick where the attention is strong and faint where it is weak. Add words from the bank, switch between one head and two, and open the panels to see the raw numbers behind each weight. *Interactive demo: [open the self attention demo](https://protrailblazer.com/demos/self-attention/)* ## What you're seeing The sentence starts as "the cat sat on the warm mat," drawn as a row of tappable chips. Each chip is one token. Tap one to make it the query, the word currently doing the looking. Downward arcs stretch from the query to every other token, and the thickness and brightness of each arc is that token's attention weight. A pulse of light travels the strongest arc, so the current focus is always visible. The readout at the top tracks four numbers: which token is the query, which token it attends to most strongly, the size of that peak weight, and the entropy of the whole distribution in bits. Low entropy means the query is locked onto one or two words. High entropy means its attention is spread thin across many. You are not limited to the default sentence. Type your own, or tap from the bank of 48 words that carry hand-built meaning vectors. Words outside that bank still work, but they show up as dashed chips with a question mark, because their meaning vector is a neutral placeholder. Their arcs are still drawn, they just are not meaningful. A few things to try: - **Tap different words as the query.** Make "mat" the query, then "cat," and watch which words each one leans on. - **Edit the sentence.** Type your own or pull words from the bank, then re-tap a query to see the weights recompute. - **Turn on the second head.** Switch from one head to two and compare what each one attends to. - **Open The numbers.** Expand it to see the raw dot products, the same scores after dividing by the square root of the dimension, and the softmax weights. - **Open Show the math.** Watch the live values get substituted into softmax(QKᵀ/√d)V one step at a time. ## How self-attention actually works Self-attention gives every token three vectors derived from its embedding: a query, a key, and a value (the query, key, value framework). The query is what this token is looking for, the key is what each token offers, and the value is the information it hands over if it is attended to. The score between the query and one key is their dot product, a single number for how well they match. The query is dotted against every key in the sentence, producing one raw score per token. Those scores are then divided by the square root of the key dimension, which keeps them from growing too large as the vectors get wider. A softmax turns the scaled scores into weights that are all positive and sum to 1. The output for the query token is the weighted sum of every token's value vector, using those weights. So the new representation of "it" is literally a blend of the other words, weighted by how much it attends to each. Every token is processed this way at once, in parallel. ## Why run two heads? One set of query, key, and value vectors can only capture one kind of relationship. In the demo, the first head is semantic: it scores words by meaning, so "cat" and "mat" attend to each other and "warm" leans toward "mat." The second head is positional: it is a previous-token head, where each word looks one step back regardless of meaning. Multi-head attention is exactly this idea scaled up. A real model runs many heads in parallel, each with its own query, key, and value projections, and concatenates their outputs. One head can track subject-verb agreement while another tracks position and a third tracks something we do not have a clean name for. Notice that the positional head needs a position signal to work at all. Attention on its own is order-blind: shuffle the words and the raw content scores come out the same. That is why models add position information to the tokens before attention ever runs. Self-attention is one instance of a more general pattern. To see the same score, normalize, allocate loop show up in search ranking and social feeds, read [how attention works across search, feeds, and LLMs](https://protrailblazer.com/posts/how-attention-works-across-systems/). And to see where these weights end up, the blended output feeds [next-token prediction](https://protrailblazer.com/posts/how-llms-generate-text/). ## Frequently asked questions ### What is self-attention in simple terms? It is the step where each word in a sentence looks at every other word and builds a new version of itself, blended from the words it finds relevant. A model uses it to work out things like which noun a pronoun refers to, or which adjective modifies which object. ### What are the query, key, and value? Three vectors each token produces from its embedding. The query is what a token is looking for, the key is what a token offers to matches, and the value is the content it contributes when attended to. Scores come from queries dotted with keys, and the output is a weighted sum of values. ### Why divide the scores by the square root of the dimension? As vectors get wider, dot products grow larger on average, which pushes softmax toward putting almost all its weight on a single token and flattening the gradients. Dividing by the square root of the key dimension keeps the scores in a range where softmax stays smooth and training remains stable. ### What is multi-head attention? Running several attention operations in parallel, each with its own query, key, and value projections, then combining their outputs. Each head can specialize, one on meaning and another on position, so the model captures several kinds of relationship at once. ### How does self-attention relate to the context window and KV cache? The keys and values every token computes are what a model caches during generation so it does not recompute them for each new token. That cache grows with the length of the [context window](https://protrailblazer.com/posts/context-window-explained/), which is a big part of why long contexts cost more memory and time. ## Key takeaways - Attention is a weighted average, not a lookup. Every token contributes to every other token, just by different amounts, and the weights for one token always sum to one. - Dividing the scores by the square root of the key dimension is not cosmetic. Without it, large dot products push softmax into a corner where one weight is nearly 1 and the gradients flatten out, so training stalls. - A single head can only track one kind of relationship at a time. Real models run many heads in parallel so one can follow meaning while another follows position. - Attention itself is order-blind. The raw mechanism would score the same words the same way in any order, which is why a separate position signal has to be added before the scores are ever computed. - The keys and values a token produces here are exactly what a KV cache stores during generation, which is why longer contexts cost more memory and time. --- --- title: "RLHF and DPO: how models learn what people prefer" description: "Label response pairs and watch two preference-optimization pipelines, RLHF with a reward model and DPO's direct update, reshape a conceptual policy landscape." published: 2026-07-16 updated: 2026-07-16 canonical: https://protrailblazer.com/posts/rlhf-vs-dpo/ tags: [machine-learning, demo, rlhf, dpo, preference-optimization, reward-model, llm-alignment, kl-divergence] --- # RLHF and DPO: how models learn what people prefer **Answer:** RLHF and DPO are the two main recipes for teaching an LLM which of two answers people prefer. RLHF first fits a separate reward model to the preference labels, then updates the policy to chase that reward while a KL leash holds it near the original model. DPO skips the reward model and turns each labeled pair into a direct policy update. The demo below lets you label pairs yourself and replay the exact same labels through both pipelines. The interactive below is a preference-optimization training studio. Pick a prompt, choose a labeler persona, then tap the response you prefer. Each labeled pair flows into either the RLHF pipeline or the DPO pipeline and deforms a conceptual policy landscape in real time. Use the mode toggle to replay your exact labels through the other pipeline and see how the two approaches diverge from the same data. *Interactive demo: [open the rlhf dpo demo](https://protrailblazer.com/demos/rlhf-dpo/)* ## What you're seeing The screen is divided into three panels. On the left, the pipeline diagram shows the active training stages lighting up in sequence as you label: in RLHF mode the reward model fitting step activates first, then the policy update stage; in DPO mode only the direct policy update fires. A math panel behind the "show math" button displays the simplified objective for whichever mode is active. The center panel is the response studio. Four prompt presets (explanation, refusal, writing edit, factual) and four labeler personas (helpful, concise, formal, cautious) let you explore different preference shapes. Tap response A or B to label a pair, or let the auto-labeler run. The undo button removes the last pair; reset clears the dataset entirely. The right panel is the conceptual policy landscape, a 2-axis contour map drawn on canvas (concise to verbose on one axis, cautious to assertive on the other). Six response archetypes sit as fixed points in that space. The white ring is the frozen reference policy; the glowing dot is the current policy. As training steps accumulate, the dot drifts toward the style the persona prefers. The HUD at the top tracks labeled pairs, label agreement with the persona, the policy's probability on the preferred style, KL divergence from the base policy, total training steps, and the current loss. A few things to try: - **Switch the mode toggle from RLHF to DPO after labeling ten pairs.** The landscape replays your labels through the other pipeline from scratch. Any difference in where the policy dot ends up comes from the training machinery, not the data. - **Turn on label noise, then crank training strength to 2.0 or higher.** Watch the KL readout climb and the landscape warn that the policy is collapsing toward a single archetype. The KL leash in RLHF mode limits exactly this kind of overshoot. - **Swap personas mid-run.** Both pipelines drag the policy toward the new persona's preferred style immediately. The AGREEMENT stat measures consistency with the current persona's taste, not with any ground-truth notion of answer quality. - **Set batch size to 16 and run the auto-labeler.** The policy landscape moves in larger jumps: the batch accumulates 16 pairs before each update, so you see fewer, coarser steps rather than the fine-grained drift of batch size 1. - **Watch the RM LOSS versus DPO LOSS readout in the HUD.** In RLHF mode the loss is the reward model's cross-entropy on the labeled pairs; in DPO mode it is the DPO contrastive objective directly on the policy. They measure different things, so comparing raw numbers across modes is not meaningful. ## How RLHF and DPO actually work Before either method runs, the base model has already been through pretraining on a large text corpus. That gives it broad language ability. Preference optimization is a second stage that shapes the model's style and safety behavior without retraining from scratch. RLHF works in two steps. First, human annotators (or, in many current pipelines, a combination of human and AI raters) label response pairs: given prompt P, is response A or B better? Those labels train a separate reward model, typically a smaller transformer with a scalar head, to predict the preferred response. Then the policy model is updated using the reward model's scores as the training signal. A KL penalty keeps the updated policy from drifting too far from the original, preventing the model from gaming a reward that is itself imperfect. DPO, introduced by Rafailov et al. in 2023, collapses the two RLHF steps into one. It derives a closed-form reparameterization showing that the optimal policy for any reward can be written directly in terms of the policy's own log-probabilities and the reference policy. Training then uses a loss function that directly increases the relative probability of the preferred response versus the rejected one, with no reward model in the middle. The KL constraint is implicit in the formulation rather than a separate penalty term. Because DPO updates the policy directly on each pair, it only moves probability mass between the two responses in that pair. RLHF's reward model, by contrast, is a smooth generalizer: it fits a score surface over the entire style space and can push probability toward styles that never appeared in the labeled pairs, as long as the reward model extrapolates correctly. The demo makes this visible, but note that both the reward model and the policy landscape are conceptual simulations, not real model parameters. ## Why the KL leash matters Both RLHF and DPO anchor the updated policy to a frozen copy of the starting model called the reference policy. In RLHF the anchor is explicit: the training objective adds a KL penalty proportional to how far the new policy has moved from the reference. In DPO the same constraint is baked into the loss derivation. Without it, the optimizer would find reward-maximizing or pair-winning answers that look nothing like coherent language output. The KL readout in the HUD measures KL divergence in nats. When it rises sharply, the policy is collapsing onto one region of the style space, a sign that training strength is too high or the dataset is too noisy. Real implementations clip or anneal the training signal at this point. In the demo, turning on label noise and pushing hyperparameters like training strength high enough will trigger a collapse warning on the landscape panel. Understanding how these updates move the policy is connected to how the base model generates text in the first place. The [next-token generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text/) covers how logits and softmax combine to produce a probability distribution over tokens, which is the exact distribution preference training reshapes. And for the optimization mechanics underlying both RLHF and DPO, the [gradient descent demo](https://protrailblazer.com/posts/gradient-descent-demo/) shows how gradient descent drives weight updates toward lower loss. ## Frequently asked questions ### What is the difference between RLHF and DPO? RLHF trains a separate reward model on preference labels and then uses that model's scores to update the policy under a KL penalty. DPO skips the reward model and turns each labeled pair directly into a policy update using a contrastive objective. Both can train on identical preference data and produce similar results, but RLHF requires the extra reward model training step while DPO is a single-stage fine-tune. ### Why do RLHF and DPO produce different landscapes in the demo even with the same labels? The reward model in RLHF generalizes across the style space, fitting a smooth score surface that can influence styles beyond those in the labeled pairs. DPO's update is more local, directly adjusting the relative probability of the two responses in each pair. The conceptual landscape in the demo shows this divergence: the RLHF dot can drift toward styles that only the reward model extrapolates to, while the DPO dot tracks the labeled pairs more directly. Both landscapes are simulated and illustrative. ### What is the KL leash and why does preference training need one? KL divergence measures how far the updated policy has moved from the reference (starting) policy. Left unconstrained, a reward-maximizing optimizer or a contrastive loss will push the model toward extreme responses that score high on the training objective but have collapsed diversity or incoherent language. The KL penalty (in RLHF) or the implicit KL term in the DPO loss caps that drift at a tunable level. ### What does the labeler persona change in the simulation? Each persona has a preferred response archetype built into the auto-labeler. When you switch from "helpful" to "concise", the auto-labeler starts picking shorter, denser responses as the winner in each pair. Both pipelines then steer the policy toward that archetype. The AGREEMENT stat in the HUD counts how many of the labeled pairs agreed with the active persona's taste. Changing persona mid-run does not reset the dataset, so you can see two training regimes tug the policy in different directions. ### Is anything in this demo running a real model? No. All values, including the policy landscape, the reward model loss, the KL divergence, and the response text, are conceptual simulations running in the browser with no model weights. The sim is designed to make the algorithmic logic visible, not to reproduce empirically accurate numbers. Every stat is labeled as conceptual or simulated within the demo. ## Key takeaways - RLHF and DPO can train on the exact same preference pairs. The mode toggle replays your labels through the other pipeline, so any difference you see in the landscape comes from the training machinery, not the data. - A reward model is a generalizer. Because it fits one smooth score over the whole style space, RLHF can shift probability toward styles that never won a single comparison, while DPO's update only moves the two responses in each labeled pair. - More preference training is not automatically better. Push the training strength up and the KL readout climbs until the landscape warns that the policy is collapsing onto one style. The KL leash and the frozen reference exist to stop exactly that. - Preference data encodes the labeler, not the truth. Swap personas mid-run and both pipelines drag the same policy toward the new target, and the AGREEMENT stat measures consistency with the persona's taste, not answer quality. --- --- title: "Reranking: why top-K retrieval isn't good enough" description: "A fast retriever pulls top-N candidates, then a cross-encoder style reranker reads query and document together and reorders them. Traps fall, answers rise." published: 2026-07-13 updated: 2026-07-13 canonical: https://protrailblazer.com/posts/retrieval-reranking/ tags: [machine-learning, demo, reranking, retrieval, rag, cross-encoder, semantic-search] --- # Reranking: why top-K retrieval isn't good enough **Answer:** Reranking is a second scoring pass that reads query and document together, catching relevance signals that a fast first-stage retriever misses by scoring them separately. A bi-encoder or keyword index retrieves a broad candidate set at speed, then a cross-encoder style reranker reads each (query, document) pair jointly and reorders the list. Documents that looked relevant on surface overlap fall; documents with precise intent or phrase alignment rise. The demo lets you watch that reorder happen candidate by candidate, with a feature breakdown showing exactly why each document moved. The interactive below runs a two-stage retrieval pipeline over a 36-document corpus. Pick a query preset, drag the Retrieve N and Rerank depth sliders, toggle the reranker on and off, and tap any candidate row to read the full feature breakdown that drove its score. *Interactive demo: [open the reranking demo](https://protrailblazer.com/demos/reranking/)* ## What you're seeing The scene runs left to right in two stages. On the left, the stage-1 panel streams the full corpus through a fast retriever that scores each document in isolation using token overlap and a hand-authored semantic feature. The top-N documents by that score become the candidate list, shown as a vertical ranked column with each document's title, snippet, and initial rank. When the animation reaches the reranking pass, the selected candidates advance to stage 2. A cross-encoder style scorer reads query and document together and computes pairwise features: intent alignment, phrase match, field weight, and negation detection. Rank-change arrows appear between the two columns as documents reshuffle. A document that matched on keywords but triggered a negation feature drops; a document with low surface overlap but strong intent alignment rises. The final top-K winners land in the answer context tray on the right. The HUD across the top tracks how many candidates were retrieved, how many the reranker read, the simulated pipeline latency and estimated token count (both labeled SIMULATED), and REL@K, the share of ideal hand-labeled relevance that made it into the final top K. The candidate inspector (tap any row) shows the exact feature breakdown and a note when a special feature such as negation fired. A few things to try: - **Switch to SUPPORT query and watch the negation trap.** In balanced mode, "Delete your account and erase all data" ranks highly after stage 1 on keyword overlap with account-management docs. After reranking it falls sharply. Tap its row at each stage to compare the negation feature before and after the joint read. - **Toggle the reranker off with Reranker On/Off.** The final tray fills from the stage-1 list unchanged. Compare REL@K with the reranker on versus off to see how much precision the second pass adds for the same N. - **Switch between SPEED, BALANCED, and QUALITY mode presets.** SPEED turns the reranker off entirely (RERANKED 0, TOP-5 Δ 0). QUALITY reranks all 30 retrieved candidates at higher simulated latency and token cost. Watch the LATENCY and TOKENS READ stats update to see the cost of a deeper rerank. - **Drag Rerank depth down to match Final K.** Setting depth to 5 with K=5 means the reranker reads only the documents it must return. REL@K often stays near the full-depth value because the most relevant documents tend to cluster near the top of the stage-1 list. - **Try the COMPARE and POLICY presets for different failure modes.** COMPARE has queries where the key signal is a field match (pricing tier, feature table), not keyword density. POLICY queries have low surface overlap with the most relevant answer, so intent alignment does most of the heavy lifting in stage 2. ## How two-stage retrieval actually works The problem reranking solves is a speed-quality tradeoff baked into the first stage. A dense retriever encodes the query into a vector, encodes every document separately, and finds nearest neighbors by dot product or cosine distance. This is fast because the document embeddings are precomputed and the query encoding is a single forward pass. But scoring is done in isolation: the model never sees query and document at the same time, so fine-grained signals like intent, negation, or phrase alignment cannot influence the score. A cross-encoder reranker fixes this by concatenating the query and document into a single input and running a joint scoring pass. It can see that 'delete account' and 'manage account' share most of their tokens but have opposite intent. The price is linear cost: scoring N candidates requires N separate forward passes through the reranker. That is acceptable for a shortlist of 10 to 30 documents; it would be prohibitive for the full corpus. This is why two stages exist. Stage 1 is a recall gate: fast, approximate, designed to ensure the correct documents are somewhere in the top-N candidates. Stage 2 is a precision pass: slower, exact, designed to put the best documents at the top of the final K returned to the user or the LLM. The demo's scoring model is a deterministic teaching version, not a real neural network. It computes hand-authored features for token overlap, a semantic category match, an intent alignment score, a phrase match bonus, a field weight (metadata like pricing or API docs), and a negation penalty when the query contains a negation word paired with a matching concept. These features mimic the kinds of signals a real cross-encoder would learn from training data, labeled clearly as SIMULATED in the UI. ## Why rerank depth is a design choice, not a fixed setting Reranking depth is the number of stage-1 candidates the reranker reads. Setting depth equal to final K is the minimum: the reranker scores exactly the documents it will return, at the lowest possible token cost. Setting depth higher increases the chance the reranker will promote a document that stage 1 ranked below K but that has strong pairwise signals. In the demo you can drag depth from K up to N and watch TOKENS READ rise proportionally. REL@K typically plateaus well before full depth because the very best documents tend to land near the top of the stage-1 list anyway, and the marginal gain from reading further diminishes. The practical sweet spot in production systems is usually 2x to 4x K, which recovers most of the quality improvement at a fraction of the full-depth cost. The same tradeoff applies to latency. A cross-encoder running on GPU can score 10 to 30 documents quickly enough that the added latency is small compared to the LLM generation step that follows. That math changes for very long documents, where the context window cost of concatenating query and document grows with document length. This is one reason late-interaction models like ColBERT, which precompute token-level embeddings rather than full text, are appealing for deep reranking over long documents. To see how the documents that survive reranking get consumed by an LLM, the [retrieval-augmented generation walkthrough](https://protrailblazer.com/posts/retrieval-augmented-generation/) traces the full RAG pipeline. And for the vector space that the stage-1 retriever navigates, see [embeddings in 3D](https://protrailblazer.com/posts/embeddings-in-3d/). ## Frequently asked questions ### What is the difference between a bi-encoder and a cross-encoder? A bi-encoder encodes query and document independently and compares the resulting vectors, usually with dot product or cosine distance. It is fast because document vectors can be precomputed. A cross-encoder concatenates query and document into one input and scores the pair jointly, giving it access to fine-grained interaction signals at the cost of requiring one forward pass per candidate. Most production systems use a bi-encoder for stage 1 and a cross-encoder for reranking. ### What is REL@K and why does it matter more than the ranked list order? REL@K measures how much of the ideal hand-labeled relevant set appears in the final top K results, expressed as a percentage. A pipeline can produce a nicely ordered top-5 list while missing one of the two genuinely relevant documents entirely, which would show as REL@5 of 50 percent rather than 100 percent. Rank order within the top K matters for presentation but REL@K tells you whether the retrieval found the right things at all. ### Does reranking help keyword search as well as vector search? Yes. Two-stage pipelines pair a reranker with BM25 or TF-IDF just as often as with a dense vector retriever. The stage-1 method changes; the stage-2 reranker does not care how the candidates were retrieved. Hybrid systems that combine keyword and dense retrieval in stage 1, then rerank in stage 2, are common in production because they capture recall advantages from both approaches. ### What happens when I toggle the reranker off in the demo? With the reranker off, the final top-K documents are taken directly from the stage-1 ranking by score. The demo is in SPEED mode and the RERANKED stat shows 0. You can compare REL@K with the reranker on versus off at the same N and K to see how much precision the second pass adds. For simple queries where the stage-1 retriever already surfaces the right documents in the right order, the gap is small. For queries with negation or intent ambiguity, it can be large. ### Are the latency and token numbers in the demo realistic? No, they are simulated to illustrate relative cost, not absolute performance. The HUD labels them SIMULATED. Real reranker latency depends on the model architecture, hardware, batching strategy, and document length. A dedicated cross-encoder on GPU can rerank 20 short documents in under 50ms; a large LLM used as a reranker on CPU could take seconds. The demo uses the numbers to show that depth and N are levers on cost, not to make claims about any specific system. ## Key takeaways - Stage-1 retrievers score query and document independently, which makes them fast enough to scan thousands of documents but blind to intent signals that only appear when both texts are read together. - A document can rank highly on keyword overlap while being semantically wrong: a negation like 'delete your account' shares tokens with 'manage your account' and climbs stage 1, but the cross-encoder style reranker catches the negation feature and drops it. - Reranking depth is a latency knob, not a quality absolute: reranking only the top 10 of 20 candidates costs half the token budget of reranking all 20, and the quality gap narrows quickly because the worst candidates rarely hold genuinely relevant documents. - REL@K measures what actually matters: how much of the ideal relevance set survived into the final top K. Simulated latency and token counts show the cost side of the same tradeoff. - Turning the reranker off and switching to SPEED mode shows that the stage-1 list is not random noise: it already contains most relevant documents, just in the wrong order. Reranking is a precision move, not a recall move. --- --- title: "RAG vs fine-tuning: which one your project needs" description: "Run the same question through RAG and a fine-tuned model side by side. Push the source date past the cutoff and watch freshness, citations, and style trade off." published: 2026-07-09 updated: 2026-07-09 canonical: https://protrailblazer.com/posts/rag-vs-fine-tuning/ tags: [machine-learning, demo, rag, fine-tuning, llm, retrieval-augmented-generation, training-cutoff] --- # RAG vs fine-tuning: which one your project needs **Answer:** RAG and fine-tuning solve different problems. RAG bolts a search step onto an unchanged model so it can read current documents and cite them at answer time; fine-tuning adjusts the model's weights so a voice, format, or behavior comes out by default, frozen at whatever the training data knew. The demo below runs the same question through both pipelines side by side, so you can watch freshness, citations, and style fit trade off as you push the source date past the training cutoff. The decision bench below sends the same question through a RAG pipeline and a fine-tuned model simultaneously. Pick a scenario preset, slide the source date past the JAN 2026 training cutoff, degrade the corpus or retriever quality, and watch every bar on the center spine respond in real time. *Interactive demo: [open the rag vs fine tuning demo](https://protrailblazer.com/demos/rag-vs-fine-tuning/)* ## What you're seeing The canvas splits into two pipeline lanes. On the left, the RAG pipeline flows from query through a retriever, into a context packet fed to a base model, and out to a cited answer. On the right, the fine-tuned pipeline flows from query into a model whose weights were updated during training on data frozen at the JAN 2026 cutoff, producing an uncited answer in the target voice or format. Between them, a center spine compares the two lanes on seven dimensions: freshness, citability, style fit, retrieval hit, training age, latency (RAG path versus fine-tuned path, both labeled SIMULATED), and upkeep load. The source date slider runs from OCT 2025 to JUL 2026 and crosses the training cutoff. Before the cutoff both lanes can know the fact; after it, only the RAG lane can retrieve it. The five scenario presets (TODAY'S POLICY, BRAND VOICE, PRIVATE MANUALS, NEW PRODUCT, OUTPUT FORMAT) each carry a distinct teaching beat so you can see how the trade-offs shift by use case. All spine bar scores and latency figures are deterministic teaching rules, labeled illustrative or SIMULATED throughout the UI. Tap any bar, tag, chip, doc, or training strip to open a plain-language explanation. A few things to try: - **Slide the source date past JAN 2026 on TODAY'S POLICY.** Watch the RAG lane keep its freshness bar high while the fine-tuned lane's freshness drops and the answer chip shows the stale fact stated with full confidence. - **Switch to BRAND VOICE with the date before the cutoff.** Both lanes know the underlying fact, but the RAG lane can only quote the style guide it finds; the fine-tuned lane applies the tone by default. Style fit belongs in weights, not in a retrieval index. - **Try PRIVATE MANUALS and drop corpus quality below 45%.** The retriever fails to find the needed document. RAG responds with a confident wrong citation; the fine-tuned lane can recall the procedure but cannot cite the section it came from. - **Switch to BOTH mode on OUTPUT FORMAT.** The combined overlay runs a narrow third lane. The verdict reaches BEST TOGETHER, showing when a fine-tune supplies the output schema while RAG supplies current values. Then degrade retrieval: the combined lane inherits RAG's failure, tagged RETRIEVAL MISSED / WEIGHTS FALLBACK. - **Read the caveat panel.** It covers prompting, tool calls, structured outputs, and base-model cutoffs. In many real cases one of those three beats building either pipeline. ## How RAG and fine-tuning actually work RAG keeps the base model's weights entirely unchanged. At query time a retriever converts the query into a dense vector, searches an index of document embeddings for the nearest chunks, and pastes those chunks into the model's context window as grounding text. The model reads and cites the retrieved text on every call. If the index is kept up to date, the model can answer questions about documents published yesterday; the knowledge lives outside the weights in an updatable store. Fine-tuning continues pretraining from a different starting point: an already-trained base model. The process exposes the model to a curated set of examples and updates the weights to shift the model's behavior toward the target voice, format, or task. Those updates are frozen at the end of training. A fact in the training data can be recalled at inference time without any retrieval step, but the model cannot reach past the training cutoff and cannot point to a specific source the way a retriever can. The center spine separates the two on dimensions that reflect real engineering decisions. Freshness measures how current each side's facts can be. Citability measures whether the answer can point at its evidence. Style fit measures how reliably output matches a target voice or format. Upkeep measures the ongoing maintenance load: re-indexing versus retraining. These are the dimensions that actually determine which approach is cheaper to maintain as the world changes. ## Why the cutoff matters more than it looks A fine-tuned model states old facts with the same confidence it states current ones. The training cutoff is invisible in the output. When a policy changes or a product price updates, the fine-tuned model still returns the old answer, correctly spelled, fluently phrased, and indistinguishable from a fresh one. This is different from a retrieval failure, where a careful reader might notice the citation is missing. Staleness without any signal of staleness is the harder problem. RAG moves the knowledge out of the weights and into an index so facts can be updated without retraining. But it shifts the failure mode: if the retriever returns the wrong document, a stale revision, or nothing at all, the model will cite that miss as confidently as a hit. The PRIVATE MANUALS scenario in the demo shows this directly: a retrieval miss with a citation attached looks more trustworthy than an honest "I don't know." Combining both pipelines is a pairing decision, not a safety net. The OUTPUT FORMAT preset ends in a BEST TOGETHER verdict because the fine-tune provides a reliable output schema (behavior from weights) while RAG provides the current values (facts from the index). But when retrieval misses, the combined pipeline falls back to the weights, inheriting the fine-tune's cutoff. Combining does not eliminate either failure mode; it adds a new one: the failure is now harder to trace. Before building either pipeline, it is worth asking whether a structured prompt with tool calls or a few worked examples would achieve the same result. The caveat panel in the demo covers this. For many business cases, retrieval and fine-tuning are both over-engineered answers to a problem that structured outputs or prompting already solves. See the [RAG pipeline deep dive](https://protrailblazer.com/posts/retrieval-augmented-generation/) for the retrieval mechanics in detail, and the [embeddings explainer](https://protrailblazer.com/posts/embeddings-in-3d/) for how the dense vectors that drive retrieval are actually built. ## Frequently asked questions ### When should I use RAG instead of fine-tuning? Use RAG when the facts you need change frequently, when you need the model to cite its sources, or when your knowledge base is too large to fit in a context window reliably. Use fine-tuning when you need a consistent output format, a specific voice, or a behavior pattern that prompting alone cannot lock in. The two dimensions rarely overlap: freshness and citation push toward RAG; style and behavior push toward fine-tuning. ### Can I use both RAG and fine-tuning together? Yes, and it is sometimes the right call. The OUTPUT FORMAT scenario in the demo shows a case where a fine-tune provides the output schema while RAG provides the current values. The cost is that the combined system inherits both failure modes: a retrieval miss falls back to the fine-tune's training data, and a style failure is harder to diagnose when retrieval is also in the loop. Combine when the two serve genuinely different jobs; do not combine to make either one more robust. ### Why does a fine-tuned model state stale facts so confidently? The model learned that the old fact was true from its training data. At inference time it has no awareness of a training cutoff; it does not know that time has passed. The fact was true when the training examples were written, so the model returns it with the same confidence it returns any other fact it learned. No prompting instruction can make the model reliably flag its own cutoff for information it genuinely believes is current. ### What are the main failure modes of a RAG pipeline? Three are common. First, retrieval miss: the needed document is not in the index or is not returned by the retriever, and the model produces a fluent answer without valid grounding. Second, stale index: the document exists but a newer version superseded it and the index was not updated. Third, citation laundering: the model cites a retrieved passage that is plausible but not directly relevant, giving a confident wrong answer the appearance of sourced authority. The PRIVATE MANUALS scenario in the demo exercises all three. ### How many tokens does RAG add to each request? It depends on the retriever configuration: how many chunks are retrieved and how long each is. A typical RAG setup retrieves three to five chunks of 256 to 512 tokens each, adding 800 to 2,500 tokens per request before the query. That cost is per call and adds to inference latency. The LATENCY stat in the demo reflects this as a simulated teaching value, not a measurement. ## Key takeaways - RAG and fine-tuning change different things: retrieval changes what the model can read at answer time, while a fine-tune changes how the model behaves by default. Facts that move belong in an index; voice and format belong in weights. - Staleness never sounds unsure. The old fact was true when the model was trained, so a fine-tuned model states it confidently, and no answer-time setting can reach past the cutoff. Only retraining moves it. - A fine-tune is not a document database. It compresses patterns from training data, so it can recall a procedure but cannot cite the page it came from, and low-quality data blurs similar procedures into one confident wrong answer. - Most RAG failures are retrieval failures. A missing document or a superseded revision still produces a fluent answer with a citation attached, which makes a wrong answer look more trustworthy, not less. - Combining both is a pairing, not a safety net. When retrieval misses, the combined pipeline falls back to the weights, staleness included, and prompting or structured output modes often beat building either one. --- --- title: "Prompt injection: when data becomes instructions" description: "Watch untrusted content compete with the real task inside an agent pipeline, then test which layered defenses actually hold." published: 2026-07-07 updated: 2026-07-07 canonical: https://protrailblazer.com/posts/prompt-injection/ tags: [machine-learning, demo, prompt-injection, llm-security, agent-security, agentic-ai, ai-safety, tool-use] --- # Prompt injection: when data becomes instructions **Answer:** Prompt injection is what happens when untrusted content inside a retrieved document or tool result contains text that reads like an instruction, and the model follows it instead of (or alongside) the original task. The attack does not break into the model from outside. It rides in through the data the model was asked to read. The interactive below puts you at the controls of a six-stage agent security checkpoint. Pick an attack vector, toggle defenses one at a time, and watch the pipeline report its verdict. Use the Step button to move through stages manually, or let it run and compare verdicts as you adjust controls. *Interactive demo: [open the prompt injection demo](https://protrailblazer.com/demos/prompt-injection/)* ## What you're seeing The pipeline runs left to right through six stages: INGEST, ASSEMBLE, INTERPRET, TOOL GATE, OUTPUT CHECK, and VERDICT. On the left, three input sources feed the agent: a SYSTEM GOAL (the operator's standing instructions), a USER TASK (the immediate request), and a RETRIEVED PAGE or TOOL RESULT (external content the agent fetched to do its job). The injected payload, shown as a red shard, rides inside that third source. Trusted content arrives as white-bordered packets; the injection glows red. At the center, the model assembles all three inputs into its context window and interprets them together. The HUD stat labeled ATTACK REACHED MODEL shows whether the injected directive cleared the model's interpretation step. After the model, a TOOL GATE shows which tools are exposed and whether any of the active defenses block the tool call before it fires. An OUTPUT CHECK scanner sits between the tool gate and the action terminal. The VERDICT panel on the far right shows the final status (SAFE, BLOCKED, FLAGGED, SUSPECT, REDUCED, or HIJACKED), the severity, and two residual lines that explain what remained true even after defenses fired. The CONCEPTUAL MODEL caveat on the defense panel is there for a reason. This is a deterministic rule model designed to teach the shape of the attack surface, not a live LLM inference. The influence scores and factor multipliers are illustrative. Treat the outcomes as a map of how each defense covers (or fails to cover) each attack path, not as a precision estimate of real-world success rates. A few things to try: - **Enable only "Content delimiters," then switch to Indirect injection.** Notice that delimiters barely reduce the influence (factor 0.95) because the attack arrives inside second-hop content the agent fetched on its own. The model sees it inside the data section, not flagged as untrusted. - **Switch to Fake tool result and enable only "Allowlist tools."** The tool gate blocks the send_email call because it is not on the task allowlist. The BLOCKED verdict appears even though the model was still influenced. Then remove the allowlist and add only "Confirm actions" to see the same block come from a different checkpoint. - **Try Data exfil request with Full access and no defenses.** Then add only "Least privilege" and watch the severity drop from HIGH to MODERATE. The tool still executes, but scoped credentials mean only project notes are exposed, not the full note store. Then add "Output check" and watch the exfil link get caught before delivery. - **Toggle "Trust boundaries" off to hide the trust-tier overlay.** This is closer to what a model sees: all three input sources are present in one assembled context with no structural separator it can rely on. - **Enable all six defenses against each attack preset.** The strongest stack never reaches HIJACKED, but the residual line always notes that the model was still influenced. No stack reads "secure," only "reduced" or "blocked." ## How prompt injection actually works A language model processes its entire context as one flat sequence of tokens. The system prompt, the user message, and the retrieved document all get the same treatment: the model predicts the next token based on everything that came before. There is no hardware separation between "instructions I should follow" and "data I should read." That boundary is a convention, and the model was trained to observe it, but training is not a hard wall. Prompt injection exploits that flatness. An attacker places instruction-shaped text inside content the model was asked to process, such as a retrieved web page, a search result, or a tool call response. If the injected text is plausible enough given the model's training distribution (which includes vast amounts of instruction-following text), the model may weight it alongside the legitimate system instructions when deciding what to do next. The demo models this as an influence score. Each attack type has a base influence (how instruction-like the injected text appears) and each defense applies a multiplicative factor. "Trust labels" and "Content delimiters" reduce influence by making the injected text look more like data. If the reduced influence clears a threshold, the model acts on the hijack. If it does not, the intended task runs instead. For the indirect injection preset, delimiters barely reduce influence (factor 0.95) because the attack arrives inside a page the agent fetched autonomously, which sits in the data tier where delimiters are already expected. Once the model decides to follow the injected directive, the checkpoints outside the model are what matter. The tool gate checks whether the tool the attack requires is in the agent's current scope. The allowlist check compares the required tool against the task-specific allowed set. The confirmation gate holds sensitive tools (send_email, post_web) for human review before firing. The output scanner checks the final response against known payload patterns. Least privilege reduces what the tool can do even if the call goes through. ## Why the attack surface grows with autonomy A prompt-only chat session has a small injection surface: the model can produce bad output, but it cannot send email or post to a public board without tools. The surface expands as you add agentic capabilities: read tools bring untrusted content in, write tools let the model act on what it read, and network tools let it exfiltrate data or interact with third-party systems. Each tool added to the agent's scope is a potential blast radius expansion if an injection succeeds. The autonomy preset in the demo (Read only / Limited tools / Full access) shows this directly. With Read only, the agent has one tool: read_page. Attacks that require send_email or post_web degrade to output-only vectors, which are less severe. With Full access, every attack that clears the model hits a tool that can fire. The demo marks those results HIJACKED at HIGH severity when no other defense is present. The indirect injection variant is worth examining separately. In the demo, it arrives via a second-hop fetch: the agent reads a partner page that links to another page, and the injection is on that linked page. The agent fetched it autonomously as part of the task. From the model's perspective, the content came from a source the system prompt implicitly authorized by giving the agent a browse tool. This is why indirect injection is the hardest to detect: the retrieval itself was legitimate. For a broader picture of how agents are structured and what controls apply to them, the [model context protocol post](https://protrailblazer.com/posts/model-context-protocol-explained/) covers how tool exposure and server scoping fit into a standardized agent architecture. And for how these risks show up in practical deployments, see [the agentic workflow guide](https://protrailblazer.com/posts/agentic-workflow-smb-guide/), which walks through concrete workflow design choices that limit tool scope from the start. ## Frequently asked questions ### What is the difference between direct and indirect prompt injection? Direct injection puts the malicious text somewhere the user controls, like the chat input itself. Indirect injection puts it in external content the agent fetches during a task: a retrieved web page, a search result, or a tool response. Indirect injection is harder to defend because the content source is at least partly outside the operator's control. ### Can you train a model to ignore prompt injection? Training and fine-tuning can reduce susceptibility, and some models are hardened to be less likely to follow instruction-shaped text in the data tier. But training alone is not a complete defense. The attack surface is too broad and the injection patterns too varied for any trained heuristic to cover reliably. Defense-in-depth outside the model is what the current security posture relies on. ### Do content delimiters actually help? Yes, for some attack types and not at all for others. Delimiters help the model recognize when it is processing untrusted content, which reduces the probability it treats that content as an instruction. But for indirect injection arriving in second-hop fetches, the content lands inside the data section where delimiters are already present, so they provide almost no additional signal. The demo models this with a factor of 0.95 for the indirect attack versus 0.80 for simpler vectors. ### What does the output check actually do in the demo? It pattern-matches the model's proposed output or tool call against a set of known payload signatures before delivery. In the demo, some payloads are marked detectable (the access phrase reveal, the collect.example exfil link) and some are not (the indirect fallback note). The scanner catches detectable payloads and redacts them, resulting in a FLAGGED verdict instead of HIJACKED. It cannot catch patterns it does not know about, and it cannot undo a tool call that already fired. ### Why does the residual line always show something even when the attack is blocked? Because blocking the action is not the same as removing the cause. When the tool gate or output check blocks a hijack attempt, the model was still influenced by the injected directive. The next agent turn, a different task, or a change in tool scope could let the same influence vector succeed. The residual line is a reminder that defenses reduce impact but do not eliminate the underlying condition: untrusted content reached the model's context. ## Key takeaways - Defenses live outside the model, not inside it. The model itself has no reliable way to distinguish a malicious instruction embedded in a retrieved page from a legitimate one in the system prompt. That means the meaningful checkpoints are the tool gate, the output scanner, and the autonomy scope, not the model's own judgment. - Layering defenses reduces impact but no single layer is complete. Content delimiters barely slow indirect injection because the attack arrives in second-hop content the model fetches autonomously. Allowlisting tools stops tool-seeking attacks but leaves output-only vectors unchecked. A strong stack cuts severity at every stage without eliminating the underlying risk. - Least privilege is the highest-leverage single control. An agent that can only read the page it was asked to read cannot send email or post to a public board, no matter what the injected text instructs. Reducing the tool surface shrinks the blast radius before any other defense fires. - Output checks can catch what the tool gate missed, but they cannot undo an action that already fired. If an injected directive triggers a tool call and the tool executes, scanning the output afterward is damage assessment, not prevention. - The indirect injection variant is the hardest to detect because the malicious content is one hop removed from anything the operator put in the context. The agent fetches a linked page autonomously and the injection arrives inside content that the delimiters treat as data tier because the fetch itself was task-sanctioned. --- --- title: "Lost in the middle: where context windows go blind" description: "Drag a needle fact through a long context and run seeded trials to watch recall dip in the middle, plus the strategies that flatten the U." published: 2026-07-04 updated: 2026-07-04 canonical: https://protrailblazer.com/posts/lost-in-the-middle/ tags: [machine-learning, demo, context-window, long-context, positional-bias, llm-recall, retrieval, prompt-ordering] --- # Lost in the middle: where context windows go blind **Answer:** Lost in the middle is the tendency of LLMs to recall a fact placed at the start or end of a long context more reliably than the same fact buried in the middle, so recall plotted against position forms a U shape. The lab below lets you drag a needle fact through a simulated context, run seeded trials until the U-curve emerges, and test the strategies that flatten it: moving the evidence up front, repeating it in an end summary, or retrieving only the relevant chunks. The interactive below puts a single needle fact inside a simulated token ribbon and lets you run seeded recall trials, sweep the needle through every position, and toggle mitigation strategies. Move the needle to the middle, run 100 trials, then compare that recall rate against the same needle placed at the start or end. *Interactive demo: [open the lost in the middle demo](https://protrailblazer.com/demos/lost-in-the-middle/)* ## What you're seeing The main canvas shows a horizontal token ribbon representing the full context window. One bright red cell is the needle fact, every darker cell is a distractor. Above the ribbon a heatmap accumulates recall rates by position across trials, and a U-shaped trend curve builds from those results. The right panel shows the current simulated model answer, whether it matched the needle, and a confidence bar. The HUD tracks context size in tokens, needle depth as a percentage of context, trial count, recall rate at the current position, distractor count, and the position penalty versus the best edge performance. Every recall value is labeled SIMULATED, and the caveat panel notes that the pattern is conceptual, varies by model and eval, and does not mean middle tokens are invisible to the model. A few things to try: - **Place the needle at 50% depth and run 100 trials.** Then move to start and run another 100. The heatmap and penalty stat make the edge advantage visible immediately. - **Hit Play on the position sweep.** The needle moves through every position preset automatically and the trend curve fills in, showing the full U shape in one pass. - **Scale the context length from 8K to 128K with the same needle position.** Watch the middle penalty grow as the context lengthens, because the needle is now further from both edges in relative terms. - **Raise distractor density and switch query specificity from Exact to Multi-hop.** Each change deepens the dip independently, and together they can push mid-position recall near zero. - **Toggle the strategy chips: Move evidence, Repeat summary, and Retrieve top chunks.** Each flattens the U by a different route, and the panel notes the tradeoff that comes with each one. ## How position-dependent recall actually works When an LLM processes a long input, every token can theoretically attend to every other token. In practice, empirical needle-in-a-haystack evaluations have shown that models tend to weight tokens near the start and end of the context more than tokens buried in the middle, a pattern first described and named in research by Liu et al. (2023). The exact cause is debated, but training data, positional encoding schemes, and the way attention interacts with very long sequences all likely contribute. The lab models this as a Bernoulli recall probability that peaks near position 0% and 100% and dips in between, scaled by context length, distractor count, and query specificity. This is a teaching abstraction, not a measurement of any real model, and the sim labels it as such. What the abstraction captures correctly is the qualitative shape and the directional effect of each variable: longer contexts tend to worsen the dip, more distractors tend to worsen it, and vaguer queries tend to worsen it. One important nuance: a middle miss does not usually look like an explicit failure. In the sim you will notice that failed trials often return a plausible-sounding distractor value rather than a blank or an "I don't know." This matches how real position failures manifest in LLM outputs, which makes them easy to misread as hallucinations rather than positioning problems. ## What actually flattens the U The three mitigations in the lab each address the problem from a different angle. Move evidence up front places the needle fact near the beginning of the context before querying. This works well when you know in advance which fact the model will need, but it requires knowing that in advance. If you are filling a context with many documents and do not know which one the query will hit, you cannot reliably move the right one to the front. Repeat summary appends a condensed version of the key evidence at the end of the context. The model now sees the fact near a strong position (the end) as well as wherever it originally sat. This is a second chance rather than a fix: the original middle copy is still less likely to be recalled, and the summary copy may introduce slight paraphrasing that interacts with query specificity. Retrieve top chunks trades the positional problem for a retriever-accuracy problem. Instead of putting all documents in a single long context, a retriever (like the RAG pattern described in [retrieval-augmented generation](https://protrailblazer.com/posts/retrieval-augmented-generation/)) selects the most relevant chunks and places them near the top of a shorter context. If the retriever finds the right chunk, the model sees the fact at a strong position in a shorter window. If the retriever misses, the mitigation fails entirely, a different failure mode than a soft positional dip. Newer models have narrowed the dip significantly compared to early evaluations, and the magnitude of the effect varies by model, task, and eval setup. For a broader view of how the model processes the context it does attend to, see the [context window explainer](https://protrailblazer.com/posts/context-window-explained/). ## Frequently asked questions ### What is the lost-in-the-middle problem? It is the observation that LLMs tend to recall facts placed near the beginning or end of a long context more reliably than facts placed in the middle, so recall rates plotted against position form a rough U shape. The effect is not universal or fixed in magnitude, but it has been reproduced across multiple models and evaluation setups. ### Does this mean models cannot see middle tokens? No. The model attends to all tokens in the context. The issue is that recall reliability is lower for middle positions under certain conditions, not that middle content is invisible. A single model can recall middle facts perfectly in some trials and miss them in others, with the miss rate increasing as context length, distractor count, and query ambiguity grow. ### Why does making the context longer worsen the dip? At longer context lengths, the middle positions are further from both edges in absolute token distance. The same fact at 50% depth sits thousands of tokens from either boundary at 128K tokens versus hundreds at 8K. Whatever mechanisms bias attention toward boundaries, the middle is geometrically and representationally more isolated at greater lengths. ### Is retrieval-augmented generation a reliable fix? It routes around the positional problem rather than fixing it. A retriever selects relevant chunks and places them near the start of a shorter context, bypassing the middle-position risk. But if the retriever fails to surface the right chunk, the model never sees the fact at all. The failure mode changes from a soft positional dip to a hard retrieval miss. ### How does this interact with the KV cache? The KV cache stores key and value representations for all tokens in the context across generation steps. Position-dependent recall is a property of how attention weights those cached representations at query time, not of how the cache stores them. Quantizing the KV cache to save memory at long context can interact with which facts are reliably surfaced, but that is a separate effect on top of the positional one. ## Key takeaways - Position is a real lever, not a formatting detail. The same fact with the same query can succeed near the edges of the context and fail in the middle, so where evidence sits in the prompt changes whether the model uses it. - A middle miss rarely announces itself. In the lab, most failed trials confidently return a nearby distractor fact instead of saying not found, which is why position problems often get misdiagnosed as hallucinations. - Everything that makes the task harder deepens the dip. Longer contexts, more distractor facts, and vaguer queries each grow the middle penalty, so a fact that survives mid-position at 8K can start failing at 128K. - Mitigations route around the curve instead of fixing it. Moving evidence to the top only works when you already know which fact matters, a repeated summary is a second chance rather than a guarantee, and retrieval trades the position problem for a retriever-accuracy problem. - The U shape is a tendency, not a law. It varies with model, task, length, and eval setup, newer models narrow the dip, and a middle miss does not mean the model cannot see middle tokens. --- --- title: "LoRA fine-tuning: adapting a model without rewriting it" description: "See how LoRA trains a task update in two thin matrices while the base model stays frozen, and why rank sets a hard floor on what the adapter can learn." published: 2026-06-29 updated: 2026-06-29 canonical: https://protrailblazer.com/posts/lora-fine-tuning/ tags: [machine-learning, demo, lora, fine-tuning, peft, adapters, frozen-weights, low-rank] --- # LoRA fine-tuning: adapting a model without rewriting it **Answer:** LoRA (low-rank adaptation) fine-tunes a model by freezing all of its original weights and training a small add-on instead. Two thin matrices, A and B, learn an update that gets added on top of a frozen weight matrix, so the base model never changes and only a tiny fraction of parameters train. The demo below trains a real adapter live: watch the loss fall, sweep the rank, and see why an adapter with too little rank hits a floor it can never train past. The interactive below runs a real low-rank adapter against a seeded 12x12 target weight matrix. Hit train, sweep the rank slider, swap task presets, and tap the merged or adapter-off toggles to see the base model snap back instantly. *Interactive demo: [open the lora fine tuning demo](https://protrailblazer.com/demos/lora-fine-tuning/)* ## What you're seeing The canvas is split into three columns. On the left, a 12x12 grid shows the frozen base matrix W in gray, exactly as it was after pretraining. The base cells never change color or value during training. In the center, the low-rank adapter overlays W as a translucent red sheet (ΔW = (α/r)·B·A). Each step that sheet brightens or shifts as the adapter learns. On the right, output bars compare three signals against white target ticks: base only, adapter output, and the white seeded target. The closer the adapter bars land to the ticks, the lower the loss. The HUD in the top-right tracks rank, training step, current loss, and a white rank-floor line that shows the minimum loss achievable at the current rank. When the live loss curve flattens onto that white line, the adapter has learned everything it can. Below the loss display, the parameter panel lists exact counts for the demo (the tiny 12x12 matrix) alongside an illustrative 7B-scale estimate that updates when you switch the target-modules preset between query, value, or all-attention layers. A few things to try: - **Drop the rank to 1 and watch where the loss curve flattens.** The white rank-floor line is above zero, meaning no amount of extra training steps can close the gap. The update you are trying to learn has more structure than a rank-1 matrix can hold. - **Raise rank to 4 or 8 on the same task and compare.** The floor drops toward zero and the adapter converges faster. More rank means more capacity to represent the target update, not just a dial that makes things better in general. - **Switch between the task presets (tone, domain vocab, formatting) at the same rank.** Each preset seeds a different target ΔW, so the same rank sits at a different floor depending on how structured the task update actually is. - **Toggle "merged view" while training.** The display folds ΔW into W and redraws as a single combined matrix (W' = W + ΔW). This is what serving looks like after deployment: one matmul, zero extra overhead. Toggle it off and the two matrices separate again. - **Hit "adapter off" mid-run.** The output bars snap back to the base-only values instantly, showing that the base model is unchanged and the adapter is a removable layer, not an edit. ## How LoRA actually works A standard weight matrix in a transformer has dimensions d×d, so its full update during fine-tuning would require training d×d parameters per layer. For a 4096-dimensional model that is roughly 16 million parameters per matrix. LoRA replaces that full update with a product of two thin matrices: A with shape r×d (down-projection) and B with shape d×r (up-projection), where r is far smaller than d. The update ΔW = (α/r)·B·A, and the adapted output becomes (W + ΔW)x. During training, W is completely frozen. Only A and B receive gradients via backpropagation. Because r is small, the number of trainable parameters collapses. At rank 4 on a 4096-dimensional query matrix, you train 4×4096 + 4096×4 = 32,768 parameters instead of 16 million. That is about 0.2 percent of the original parameter count. The demo's 12x12 matrix makes the ratio more vivid: at rank 2 the adapter is two rows and two columns, a third of the base parameter count on this toy scale, but the same recipe on the illustrative 7B estimate trains around 0.007 percent. B is initialized to all zeros. This means ΔW = 0 at step zero, so fine-tuning begins from the exact base model behavior and does not need to unlearn random noise first. A is initialized from a random Gaussian. The alpha (α) hyperparameter controls the scale of the adapter's contribution relative to its rank, functioning as a stable learning-rate multiplier that does not need retuning when you change r. The loss function the demo minimizes is the Frobenius distance between the adapter output (ΔW·x) and the target update (ΔW*·x) for a batch of random inputs. Real LoRA training minimizes a task loss (cross-entropy on next-token prediction, for example) through the same two matrices. The gradient descent update reaches A and B through the chain rule exactly as in full fine-tuning, just with far fewer parameters to update. ## Why rank is capacity, not quality A rank-r matrix can only represent vectors that lie in an r-dimensional subspace. If the true update ΔW* has a higher rank than r, there is no setting of A and B that can reproduce it exactly. The best you can do is the closest rank-r approximation, which leaves a residual that the training loop can never eliminate. That is the white floor line in the demo. This is not a bug. Most task-specific updates to a pretrained model are low-rank in practice, which is why LoRA works. A language model already knows grammar, facts, and how to follow instructions; fine-tuning on a narrow task (summarizing legal contracts, for example) shifts the model in a direction that lives in a small subspace of the full weight space. Rank 4 to 16 captures that shift in most published experiments, though the right value depends on the model, the task, and how different the target distribution is from the original training mix. LoRA is most often applied to the query and value projection matrices inside attention layers (see the target-modules preset in the demo). The demo's 7B illustrative estimate updates when you switch between query-only, value-only, and all-attention modes to show how much the trainable count changes with scope. The QKV framework is also where you can see why targeting different projections might favor different tasks, though the full theory of which layers to adapt and why is still an active area of research. For the mechanics of what these weights do during inference, the [next-token generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text/) traces the full path from input token through the weight matrices to a sampled output. And for the optimizer that actually trains A and B step by step, see the [gradient descent demo](https://protrailblazer.com/posts/gradient-descent-demo/). ## Frequently asked questions ### Does LoRA change the base model's weights at all? No. The base matrix W receives no gradients and never updates. Only the adapter matrices A and B are trained. At serving time you can either pass the input through W and ΔW separately (two matmuls) or merge them into a single W' = W + ΔW (one matmul, same result, zero latency overhead). The separation is permanent: you can always subtract ΔW back out and recover the original base model. ### What is the difference between rank and alpha in LoRA? Rank (r) sets the dimension of A and B, which determines how much of the update space the adapter can express. Alpha (α) scales the adapter's output by α/r before it is added to the base output. In practice α is often set to a fixed multiple of r (commonly α = r or α = 2r) so that the effective scale stays stable when you change r. Rank controls capacity; alpha controls how strongly the adapter's update is weighted against the frozen base. ### Why does the loss plateau and not keep falling? The white rank-floor line marks the closest approximation a rank-r matrix can reach to the target update. If the target has a true rank higher than r, no training run of any length can cross that floor. The demo shows this directly: at rank 1 on tasks with a richer target, the loss curve flattens onto the floor line and stays there even if you run for many more steps. ### How many parameters does LoRA actually train versus full fine-tuning? On the demo's 12x12 matrix at rank 2, the adapter has 2×12 + 12×2 = 48 parameters versus 144 in the full matrix. The illustrative 7B estimate in the parameter panel shows the real-world version: targeting query and value projections across 32 transformer layers trains roughly 0.007 percent of the base parameter count at rank 8, and the adapter fits in about 1 MB at FP16. These illustrative figures are approximate and depend on architecture; the demo labels them as such. ### Can LoRA be applied to layers other than attention? Yes. LoRA can wrap any linear layer, including the feed-forward blocks, embedding layers, and output projection. In practice most published results apply it to the query and value projections because those are where task-specific behavior tends to concentrate, but the math is the same regardless of which layer you target. The target-modules preset in the demo only affects the 7B illustrative parameter count, not the actual training, since the demo matrix represents a generic weight update. ## Key takeaways - LoRA never edits the base model. The frozen weights and the adapter are separate objects, which is why you can merge the update into the weights for zero-overhead serving and still subtract it back out later. - Rank is capacity, not quality. A rank-r adapter can only express a rank-r update, so when the task's true update is richer, training plateaus at a hard floor that no extra steps or patience can cross. - The trainable fraction collapses at scale. In the tiny 12x12 demo a rank-2 adapter is a third of the base parameters, but the same recipe on the illustrative 7B estimate trains about 0.007 percent and fits the adapter in roughly a megabyte. - B starts at zero on purpose. That standard init makes the update exactly zero at step 0, so fine-tuning starts from the base model's exact behavior instead of from random noise it must first unlearn. --- --- title: "Can my computer run an LLM? VRAM requirements visualized" description: "A live VRAM calculator: watch model weights, a growing KV cache, and runtime overhead fill a GPU memory bar, spill into system RAM, and compare GPUs." published: 2026-06-27 updated: 2026-06-27 canonical: https://protrailblazer.com/posts/llm-vram-requirements/ tags: [machine-learning, demo, vram, kv-cache, quantization, gpu-memory, llm, inference] --- # Can my computer run an LLM? VRAM requirements visualized **Answer:** Whether your computer can run an LLM comes down to one comparison: the model's total memory demand versus your GPU's VRAM. That demand has three parts, the weights (fixed once the model loads), the KV cache (which grows with every token of context), and runtime overhead. The calculator below adds all three live, so you can watch a model that fits at short context spill past the capacity edge as the session gets longer. The interactive below lets you set a model size, weight precision, context length, batch size, and GPU capacity, then press play to watch the KV cache grow token by token until it either fits, spills into system RAM, or overflows entirely. Use the scenario presets to jump straight to the situations that catch people off guard. *Interactive demo: [open the llm vram calculator demo](https://protrailblazer.com/demos/llm-vram-calculator/)* ## What you're seeing The main element is a horizontal GPU memory bar divided into three colored segments. The red segment on the left is weight memory, fixed at load time. The amber-tinted segment in the middle is the KV cache, which grows rightward with each token as the play session runs. A smaller segment at the right edge represents runtime overhead, controlled by the overhead slider. A vertical white line marks the GPU's VRAM capacity. When the total demand exceeds the capacity line, the overflow drains downward into an OFFLOAD tray labeled slow / illustrative (when the CPU offload toggle is on) or the bar edge turns red and flags DOES NOT FIT (when offload is off). A second GPU compare bar can appear below the main bar on the same GB scale, letting you test whether a larger card absorbs the same demand. The HUD across the top (or the bottom sheet on narrow screens) tracks weight memory, KV cache, overhead, total estimate, headroom, fit status, offloaded GB, and current context tokens in real time. Every number in the demo is derived from the formulas shown in the collapsible how-calculated panel: weights = params x effective bits / 8 (INT4 at about 4.85 effective bits; INT8 at about 8.5); KV cache = 2 x layers x (hidden / GQA divisor) x bytes-per-value x context x batch. These are planning estimates, not benchmarks. Real requirements vary by framework, kernel, quantization layout, KV paging, and model-specific architecture details. A few things to try: - **Load the "128K context" preset.** The 7B model's weights take about 7 GB at INT4, but the KV cache alone grows past 16 GB by the time the context fills. Watch a card that looks fine at short context get crushed as the session extends. - **Try "70B on 24 GB" and enable the compare bar with an 80 GB card.** At FP16 a 70B model's weights alone exceed 130 GB. Even at INT4 (about 39.5 GB for weights) the 24 GB card cannot hold just the weights, while the 80 GB card fits the full session with room left over. - **Load "serve batch 8" and watch the KV cache.** Batch serving stores eight separate KV caches against one shared copy of the weights. The weight bar does not grow, but the cache bar multiplies by eight, which is why serving hardware needs far more VRAM than a single-user chat setup. - **Toggle CPU offload on versus off during an overflow.** With offload on, the overflow drains into the amber tray and the session continues, flagged as slow. With offload off the bar turns red and stops. This illustrates the real tradeoff: offload lets you run something that does not fit, but every offloaded gigabyte must cross the PCIe bus at inference time. - **Open the how-calculated panel and change the model to CUSTOM.** Edit params, layers, hidden size, and the GQA divisor directly to match an architecture you care about. The bar and HUD recalculate live. ## How VRAM demand is actually calculated Weight memory is the simplest part. Each parameter is stored at a given precision: FP16 uses 2 bytes, INT8 uses about 1.06 bytes (including scale metadata), and INT4 uses about 0.61 bytes (4.85 effective bits including group-scale overhead). Multiply parameters by bytes per parameter and you have the weight footprint. A 7B model at INT4 needs roughly 4.3 GB for weights; at FP16 it needs 14 GB. The KV cache is where the surprises happen. For each token in the context window, every transformer layer stores a key vector and a value vector. The formula is: 2 (for keys and values) x layers x (hidden size / GQA divisor) x bytes per value x context length x batch. At FP16 with a 7B model (32 layers, hidden 4096, GQA divisor 1, 2K context, batch 1) the cache is about 0.5 GB. At 128K context the same model needs over 16 GB for the cache alone. That is why a card that handles a short session can fail later in a long conversation. Runtime overhead covers the activations, framework buffers, CUDA context, and workspace tensors that live in VRAM during inference but are not part of the weights or cache. The demo models this as an explicit slider from 0.5 to 6 GB, labeled conceptual because the real number depends on framework, kernel implementation, batch size, and whether you are doing speculative decoding or multimodal preprocessing. The slider is not a formula; it is an honest acknowledgment that something is there. GQA (grouped-query attention) is worth understanding here. Standard multi-head attention stores one KV head per attention head, so the cache scales with the full head count. Many modern models use GQA, where multiple query heads share a single KV head. The GQA divisor in the advanced panel controls this ratio: a divisor of 8 means 8 query heads share one KV pair, cutting the KV cache to one-eighth of what standard MHA would need. The default model shapes in the demo use GQA divisors that loosely reflect current open-weight families. ## Why fit status changes during a session When a model loads, only the weights and overhead are in memory. The KV cache starts empty and grows with every token generated. This means fit status is not a static property of a model-and-GPU pairing: it is a function of how far into a conversation you are. The calculator shows four fit states. FITS means total demand is below VRAM with at least 10 percent headroom. TIGHT FIT means less than 10 percent remains, flagged in amber as a warning that a longer session or a heavier batch will overflow. RUNS SLOW / OFFLOAD means the model technically runs but some memory lives in system RAM, crossing the PCIe bus at every token. DOES NOT FIT means offload is off and there is no path to run the model at the current settings. The practical implication: if you are choosing a GPU for local inference, do not size for just the weights. Size for the weights plus the KV cache at the context length you actually plan to use, plus a buffer for overhead. The 128K context scenario in the demo demonstrates why a 12 GB card that runs a 7B model at short context cannot finish a long document analysis session. For a deeper look at what happens inside the model during each step of generation, including how the KV cache feeds into each [autoregressive token step](https://protrailblazer.com/posts/how-llms-generate-text/), see the generation walkthrough. For how inference-time strategies like best-of-N and chain-of-thought amplify the compute and memory demand per request, see the [inference-time compute explainer](https://protrailblazer.com/posts/inference-time-compute-explained/). ## Frequently asked questions ### What is the minimum VRAM needed to run a 7B model? At INT4 precision, a 7B model's weights need roughly 4 to 5 GB. Add 0.5 to 1 GB of runtime overhead and you can load and run at short context on an 8 GB card. The problem is that the KV cache can easily grow another 4 to 16 GB depending on context length and batch size, so 8 GB is a tight fit for anything beyond short sessions. A 12 GB card gives substantially more headroom. ### Does quantization fix the KV cache problem? Quantization shrinks the weights, not the KV cache by default. You can quantize the KV cache separately (FP8 or INT8 KV is increasingly common), but that is a separate toggle from weight precision. The demo lets you set KV cache precision independently (FP16 vs FP8/INT8) to show the difference. At long context the cache often exceeds the weight footprint, so quantizing only the weights while leaving the cache at FP16 does not help as much as people expect. ### Why does batch size multiply the KV cache but not the weights? The weights are shared across all sequences in a batch: every request reads the same weight values in parallel. But each request has its own sequence of tokens, and each sequence needs its own key and value vectors cached at every layer. So running 8 concurrent requests stores 8 separate KV caches against a single shared copy of the model. The demo's serve batch 8 preset makes this visible: the weight bar stays fixed while the KV bar multiplies. ### What does CPU offload actually mean for performance? When the total memory demand exceeds VRAM, some tensors are moved to system RAM. At inference time, the model reads those tensors back over the PCIe bus, which is roughly 10 to 50 times slower than reading from VRAM depending on the bus generation and what is being transferred. Offload lets you run a model that would otherwise not fit, but generation speed drops substantially, often to single-digit tokens per second or worse. The demo labels the offload tray as slow / illustrative to make this clear. ### Why are the numbers in the demo labeled as estimates? The formulas in the demo are planning estimates based on parameter counts, precision widths, and architecture shapes. Real inference adds framework memory (PyTorch, llama.cpp, vLLM each have different overheads), CUDA context, KV paging structures, optional speculative decoding buffers, multimodal preprocessing tensors, and OS memory pressure. The demo's how-calculated panel lists these caveats explicitly. Use the estimates to understand which factor dominates and to size hardware; do not treat the numbers as exact measurements for a specific model and framework. ## Key takeaways - VRAM demand is not one number. The weights are fixed at load time, but the KV cache grows with every token of context, so a model that loads fine can overflow the same card later in a long session. - Quantization shrinks the weights, not the KV cache. In the 128K context scenario the 7B model's weights take about 7 GB while the cache alone grows to 16 GB, so at long context the cache is what sinks you. - Quantized weights cost more than their nominal width. Counting scale factors and metadata, INT4 is about 4.85 effective bits per weight, which is why a '4-bit' 70B model still needs about 39.5 GB for weights before a single token is processed. - Batch multiplies the KV cache, not the weights. Serving eight concurrent sequences stores eight separate caches against one shared copy of the model, which is why serving hardware needs far more memory than a single chat. - CPU offload turns 'does not fit' into 'runs slow', not into 'fits fine'. Every offloaded gigabyte has to cross the PCIe link during generation, so treat the offload tray as a warning, not a solution. --- --- title: "LLM quantization explained: fewer bits, less memory" description: "See how quantization shrinks a model's weight memory by storing FP16 values on a coarser grid of levels, and where the error actually lands." published: 2026-06-23 updated: 2026-06-23 canonical: https://protrailblazer.com/posts/llm-quantization-explained/ tags: [machine-learning, demo, quantization, int8, int4, model-compression, llm] --- # LLM quantization explained: fewer bits, less memory **Answer:** Quantization shrinks an LLM by storing each weight with fewer bits, rounding the original FP16 values onto a small grid of representable levels. INT8 keeps 255 levels and halves the weight memory; INT4 keeps 15 levels and quarters it. The demo below quantizes a real 64x64 weight tensor live, so you can watch the memory collapse instantly, the rounding error creep in gradually, and a simulated next-token pick flip at the 2-bit extreme. The interactive below runs real symmetric quantization over a seeded 64x64 weight tensor. Pick a precision, choose per-tensor or per-group scales, drag the morph scrub, and tap any cell in the matrix to read the exact original and quantized values side by side. *Interactive demo: [open the quantization demo](https://protrailblazer.com/demos/quantization/)* ## What you're seeing The scene runs left to right through four stages. On the left, a seeded FP16 weight field draws the original distribution as a histogram with sigma bands and a dashed mean line. Next, a quantizer gate shows the bin boundaries for the chosen precision, and particle tracers stream through it as the morph plays. To the right of that, a collapsing bit tower fills from the bottom as bits per weight drops, and a memory readout shows GB for 1B, 7B, and 70B weight sizes (labeled illustrative). The far-right panel is the simulated downstream head, a fixed 5-token prediction panel clearly marked SIMULATED / NOT A REAL LLM. Below the main scene is a 64x64 tensor tile. The default view zooms in on a 12x12 patch where each cell is color-coded by rounding error from white at zero to deep red at maximum deviation. Switch to the full 64x64 view to see the global error pattern, then tap any cell to read the FP16 original, the quantized value, and the absolute difference. The HUD across the top tracks bits per weight, unique representable levels, weight memory, compression ratio vs FP16, mean absolute error, and cosine similarity between the original and quantized tensor in real time. A few things to try: - **Switch the weight distribution to OUTLIERS with per-tensor INT4.** Watch mean absolute error jump versus the BALANCED preset, because a handful of large weights stretch the grid spacing and everything near zero rounds coarsely. - **Keep OUTLIERS and switch from per-tensor to per-group.** The error drops visibly as each group gets its own scale. Check the HUD: effective bits per weight rises slightly (for example 4.13 at group size 128) because the scale values cost metadata. - **Push precision to 2-bit and watch the downstream panel.** Symmetric 2-bit has only 3 representable levels. The top-1 token pick in the simulated head flips at this extreme, even though cosine similarity can still read near 1.0. - **Drag the morph scrub slowly at INT4.** The bit tower collapses the moment you change precision, but MAE and cosine similarity move gradually with the morph, so the visual cost of each level of rounding emerges step by step. - **Tap individual cells in the matrix.** The heaviest error cells are not random: they cluster where the original value sat between two widely spaced quantization levels. ## How quantization actually works Every weight in an LLM is stored as a floating-point number, usually FP16 (16 bits, ~65,000 possible values per weight). Quantization replaces that with a lower-bit integer by computing a scale factor, dividing every weight by that scale, rounding to the nearest integer, and clamping to the integer range. For symmetric quantization (what the demo uses), the scale is the largest absolute value in the tensor divided by the maximum representable integer. Every weight is divided by that scale, rounded to the nearest integer, and stored. At inference the stored integer is multiplied back by the scale to recover an approximation of the original. The error is whatever the nearest integer could not represent exactly. INT8 gives 255 unique levels (from -127 to 127 in symmetric form) and halves weight memory relative to FP16. INT4 gives 15 levels and quarters memory. The 2-bit concept in the demo has only 3 levels (-1, 0, 1 in symmetric form), which is enough to illustrate breakdown but not a production format. During a model's forward pass, the quantized weights are dequantized on the fly before each matrix multiply: the integer is multiplied by the stored scale and used as if it were the original FP16 value. Memory savings come from storage and transfer; compute precision depends on the kernel and hardware. ## Why per-group scales matter With a single per-tensor scale, the largest weight in the entire tensor sets the grid spacing. If a few weights are five to nine times larger than the rest, the levels spread apart and everything clustered near zero rounds coarsely, wasting most of the 15 available INT4 levels on a range that has almost no weights in it. Per-group quantization divides the weights into small blocks (commonly 128 weights) and gives each block its own FP16 scale. Each block's scale is sized for that neighborhood, so the grid fits the local range and rounding error drops. This is what makes INT4 usable in practice: methods like GPTQ, AWQ, and NF4 all exploit per-group (or per-channel) scales. The cost is metadata. At group size 128, each group needs one FP16 scale stored alongside its 128 INT4 weights. That adds roughly 0.13 bits per weight, pushing effective storage from 4.00 to about 4.13 bits per weight. You can see this in the HUD when you enable per-group mode. The memory savings are still large, just not quite as clean as the raw bit count suggests. Quantization also affects what the KV cache costs at inference time. Some deployments quantize the KV cache separately from weights, because the two have different sensitivity profiles. Reducing KV cache precision from FP16 to INT8 can nearly halve the memory required for long-context generation, though the tradeoffs differ by layer and model. To see how the quantized weights feed into prediction, the [next-token generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text/) traces the full path from weights through logits to a sampled token. And for the tradeoff between precision and generation speed at inference, see [inference-time compute](https://protrailblazer.com/posts/inference-time-compute-explained/). ## Frequently asked questions ### What does quantization actually do to a model's weights? It replaces each FP16 weight (roughly 65,000 representable values) with a lower-bit integer (255 for INT8, 15 for INT4) by dividing by a scale factor and rounding to the nearest integer. At inference, the integer is multiplied back by the scale. The rounding error is the price; the memory saving is the reward. ### Does INT4 quantization make a model meaningfully worse? Often very little for INT4 with good per-group scales and careful calibration, but "INT4" is not a single fixed quality level. The same label covers different quality depending on architecture, calibration data, method (GPTQ, AWQ, NF4, GGUF), the precision kept for activations and the KV cache, and what you are evaluating. Bit count alone does not predict usability. ### What is the difference between per-tensor and per-group quantization? Per-tensor uses one scale for the entire weight matrix, sized to the largest absolute value. Per-group divides weights into small blocks (commonly 128 per block) and gives each block its own scale. Per-group dramatically reduces error when a few outlier weights would otherwise stretch the grid spacing and waste most of the available quantization levels. ### Why does the demo's cosine similarity stay near 1.0 even when the token pick flips? Cosine similarity measures the angle between the original and quantized tensor as vectors. Even at 2-bit, most weights move a little and the overall direction barely shifts, so cosine can read above 0.95. But the downstream softmax picks the token with the highest logit, and logit gaps between competing tokens can be tiny. A small absolute error in the right weight is enough to flip the pick even when the tensor as a whole still points roughly the same direction. ### How does quantization interact with the context window? The context window length determines how many key and value vectors the model caches during generation. At FP16 those vectors are large; quantizing the KV cache to INT8 or INT4 can roughly halve or quarter that memory cost, which is one reason large-context deployments often quantize weights and KV cache separately. ## Key takeaways - The memory win is arithmetic, not statistical. Dropping FP16 to INT4 cuts a 7B model's weight memory from 14 GB to 3.5 GB before you know anything about accuracy, which is why quantization decisions start with the bit count and end with the error measurements. - One outlier weight can poison thousands of others. With a single per-tensor scale, the largest absolute value sets the grid spacing, so a few weights 5 to 9 times bigger than the rest stretch the levels apart and everything huddled near zero rounds coarsely. - Per-group scales buy back fidelity with metadata, not magic. Each block of 16 to 256 weights gets its own FP16 scale, so effective bits per weight rise slightly (4.13 instead of 4.00 at group size 128) in exchange for a grid that fits each neighborhood. - Cosine similarity can sit near 1.0 while behavior still shifts. In the demo's simulated head, the top-1 pick flips at the 2-bit concept setting even though most weights barely moved, because downstream decisions ride on small logit gaps. - Bit count alone does not predict usability. The same INT4 label covers wildly different quality depending on architecture, calibration, method (GPTQ, AWQ, NF4), kernels, and what precision the activations and KV cache keep. --- --- title: "How AI detectors work, and why they are wrong so often" description: "Score text with the signals AI detectors use: predictability, burstiness, repetition, and style. Formulaic humans get flagged, edited model text slips by." published: 2026-06-20 updated: 2026-06-20 canonical: https://protrailblazer.com/posts/how-ai-detectors-work/ tags: [machine-learning, demo, ai-detectors, ai-detection, perplexity, burstiness, false-positives, tokens, llm] --- # How AI detectors work, and why they are wrong so often **Answer:** AI detectors never see who wrote a text. They measure statistical fingerprints, how predictable each token is, how uniform the sentence rhythm is, how much phrasing repeats, and how template-heavy the style is, then compare a fused score to a threshold. The demo below runs those same four signals over five specimens so you can watch a formulaic human essay get flagged while lightly edited model text slips under the line. The interactive below runs four heuristic signals over any text you paste or pick from five presets, fuses them into a model-like score, and lets you slide the decision threshold and run a labeled 24-sample dataset to watch the confusion matrix fill in. Try all five presets before reading on. *Interactive demo: [open the ai detectors demo](https://protrailblazer.com/demos/ai-detectors/)* ## What you're seeing The specimen panel at the top shows the text being analyzed, each word backed by a predictability strip: taller and redder means the word landed in a high-frequency, easy-to-predict position. Below that, a rhythm chart plots sentence lengths as bars, with a dashed mean line and a one-sigma band so you can see whether the rhythm is machine-flat or human-varied. The detector panel on the right fuses four toggleable signals into a MODEL-LIKE SCORE dial with a gauge needle and a shaded uncertainty band. The HUD tracks tokens analyzed, average predictability, sentence-length variation (as a coefficient of variation), n-gram repetition rate, and the final score out of 100. At the bottom, a threshold slider (default 65) draws a line across a scatter plot of the 24-sample dataset. Dots above the line are predicted AI; dots below are predicted human. Color tells you whether the prediction was right. The confusion matrix accumulates as you run the dataset, and the false-positive and false-negative chips update so the precision/recall tradeoff is visible, not asserted. The scoring uses transparent heuristic proxies: word-frequency tiers stand in for token probability, sentence-length coefficient of variation stands in for burstiness, repeated bigrams and trigrams stand in for repetition, and type-token ratio plus template connectives plus punctuation spread plus contractions stand in for style. The UI labels these HEURISTIC PROXIES and the uncertainty band ILLUSTRATIVE. No model runs and no text leaves the page. A few things to try: - **Load FORMULAIC HUMAN and run the scan.** The score lands at 69, above the default threshold, and a FALSE POSITIVE chip appears. Notice that the rhythm chart is as flat as a model's: five-sentence paragraphs, identical openers, a template structure a teacher drilled in. - **Switch to POLISHED MODEL.** It scores 68, just under the FORMULAIC HUMAN. The model output is actually less flaggable than the formulaic essay at default settings. - **Try NON-NATIVE ENGLISH.** It scores 70. The restricted vocabulary and simple sentence structures push predictability up; the detector has no way to distinguish a learner from a machine. - **With POLISHED MODEL loaded, run COPYEDIT then ADD VARIATION.** Watch the score fall from 68 to 58, below the threshold, triggering a FALSE NEGATIVE. Casual edits, not adversarial tricks, are enough to slip past. - **Slide the threshold from 30 to 90 while the dataset is running.** The false-positive and false-negative counts trade against each other continuously. No threshold zeroes out both on this 24-sample set. ## How AI detectors actually work A real AI detector is a classifier trained on a mix of human and model text. Its features are statistical properties of the text, and the most diagnostic ones fall into four buckets. Predictability (often called perplexity in the academic literature) measures how surprised a language model would be by each word or logit prediction in the sequence. A model generating text prefers high-probability continuations, so model output tends to have lower perplexity than human writing. The catch is that "high probability" and "formulaic" overlap heavily: the same words that make model output predictable also make student template essays and technical documentation predictable. Burstiness measures sentence-length variation. Human writers mix short punchy sentences with longer elaborated ones. Early language models produced very uniform rhythm. Most modern models no longer do, so burstiness is a weaker signal than it was in 2022, but it still appears in detector feature sets because the training data contains older model output. Repetition measures how often the same phrases recur. Model text at high temperature is less repetitive; model text at low temperature often repeats sentence openers, transition phrases, and structural clichés. The demo tracks repeated bigrams and trigrams plus duplicated paragraph-opener words. Style measures vocabulary richness (type-token ratio), the density of template connectives ("Furthermore", "In conclusion", "It is important to note"), the spread of punctuation marks, and the frequency of contractions. Formal academic register looks like model output on this axis; casual conversational writing looks human even when it is not. A real detector fuses these signals, usually via a fine-tuned classifier, and outputs a probability. The threshold that converts a probability into a binary HUMAN or AI verdict is a policy choice made by whoever deploys the tool. The same underlying score, say 0.62, reads as AI at a 0.60 cutoff and human at a 0.65 cutoff. No calibration is implied. To understand why token predictability matters, it helps to see how a model chooses its next word in the first place. The [autoregressive generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text/) traces the full path from weights to next-token logits to a sampled output. And for how the temperature parameter shapes that probability distribution before sampling (directly affecting how predictable or surprising the output looks), see the [LLM temperature explainer](https://protrailblazer.com/posts/llm-temperature-explained/). ## Why the false-positive problem is structural, not incidental Every signal the detector uses is a proxy for a property of model output, not a test of authorship. Predictability is really a fluency-and-formula proxy. Burstiness is a rhythm proxy. Repetition is a template-avoidance proxy. Style is a register proxy. None of them are causally linked to who pressed the keyboard. The writers who score highest on these proxies are people trained to write formulaically: students taught five-paragraph essay structures, non-native English speakers who learned formal written registers, technical documentation authors following a house style. In the demo, every human preset outscores the POLISHED MODEL at default settings. That is not a bug in the demo; it reflects what the signals actually measure. The false-negative side is equally structural. A single editing pass that varies sentence length, swaps template openers for concrete verbs, and introduces a contraction or two will drop most detector scores below any reasonable threshold. The demo's EDITED MIX preset scores 39 at default settings; it is the model-authored sample that was touched by a light copyedit. None of that constitutes adversarial attack; it is just editing. > **warn:** Detector output must never be the sole evidence in an academic, employment, or disciplinary decision. The false-positive rate for non-native English speakers is systematically higher than for native speakers on every major platform that has published error rates. A score above threshold is a reason to read more carefully, not a finding of wrongdoing. ## Frequently asked questions ### What does "perplexity" mean in AI detection? Perplexity is a measure of how surprised a language model would be by a sequence of tokens. Low perplexity means each word was highly predictable given the context; high perplexity means the choices were surprising. AI detectors use perplexity as a proxy because models generating text prefer high-probability (low-perplexity) continuations. The problem is that any fluent, formulaic writing, whether by a model or a human trained on templates, produces low perplexity. ### Can a detector tell the difference between a non-native English speaker and an AI? No. Non-native English writers often use restricted vocabulary, simple syntax, and formal register, exactly the features that detectors flag as machine-like. The demo's NON-NATIVE ENGLISH preset scores 70, higher than the POLISHED MODEL preset, because its signals overlap with model output on every axis. Several published audits have found false-positive rates for non-native writers significantly higher than for native writers. ### Why does editing the AI text so easily fool the detector? The detector measures statistics, not identity. Varying sentence lengths lowers the burstiness signal. Replacing template openers with concrete verbs lowers the repetition and style signals. Adding a contraction loosens the register signal. None of these changes require knowledge of how the detector works; they are just normal editing. The demo shows the score drop from 68 to 58 after COPYEDIT then ADD VARIATION, which is enough to fall below the default threshold. ### Is there a threshold that eliminates all false positives and false negatives? No, not on any dataset where the two populations overlap on the measured signals. The confusion matrix in the demo makes this visible: sliding the threshold from 30 to 90 continuously trades false positives for false negatives. The overlap is a property of the signals themselves, not of the threshold choice. A threshold is a policy decision about which error is more acceptable. ### Are the scores in the demo real detector scores? No. The demo uses transparent heuristic proxies labeled SIMULATED: word-frequency tiers stand in for token probabilities, sentence-length coefficient of variation stands in for burstiness, and so on. The purpose is to let you observe the structure of the problem, not to reproduce any specific commercial detector. Real detectors use trained classifiers and proprietary calibration. The directional behavior, such as formulaic humans scoring high and lightly edited model text scoring low, is based on what the published literature documents, not on this demo's specific numbers. ## Key takeaways - A detector never sees authorship, only statistics. Its score is a claim that the text looks like typical model output, and 'typical' is doing all the work: formulaic human writing and lightly edited model writing both land on the wrong side of it. - The decision threshold is a policy choice, not a property of the text. On the demo's 24-sample dataset no threshold zeroes out both false positives and false negatives; sliding it only trades one error for the other. - Predictability is largely a fluency-and-formula proxy, so the writers most at risk of false flags are non-native English speakers, students taught rigid templates, and documentation writers. In this demo every human preset outscores the machine-written POLISHED MODEL sample at default settings. - The signals are just as fragile in the other direction: one copyedit pass plus varied sentence openers drops the model-written sample from 68 to 58, under the default threshold. Casual edits, not adversarial tricks, are enough to slip past. - Score reliability collapses with length. Every signal is an average over sentences, so a verdict on a single short paragraph rests on almost no evidence, which the demo shows by widening its uncertainty band as the specimen shrinks. --- --- title: "HNSW vector search: how a layered graph finds neighbors fast" description: "A layered graph walks coarse to fine toward your query, checking a fraction of the vectors a full scan would. Tune efSearch and M and watch recall respond." published: 2026-06-16 updated: 2026-06-16 canonical: https://protrailblazer.com/posts/hnsw-vector-search/ tags: [machine-learning, demo, hnsw, vector-search, approximate-nearest-neighbor, vector-database, embeddings] --- # HNSW vector search: how a layered graph finds neighbors fast **Answer:** HNSW (Hierarchical Navigable Small World) is the graph index most vector databases use to find nearest neighbors without measuring the distance to every stored vector. It stacks a few sparse layers of long-range shortcuts on top of the full dataset, hops greedily from coarse to fine, then runs a narrow beam search on the bottom layer. The demo below animates every hop and counts every distance check, so you can watch it land on near neighbors after touching only a small slice of the data. The interactive below builds a real HNSW index in your browser from seeded 2D points and runs a genuine graph traversal on it. Pick a dataset size, adjust M and efSearch, and either tap the base layer to place your own query or pick one of the presets. You can race HNSW against brute force in compare mode, step through one hop at a time, or orbit the isometric stack to see how the layers relate. *Interactive demo: [open the hnsw vector search demo](https://protrailblazer.com/demos/hnsw-vector-search/)* ## What you're seeing The canvas renders an isometric stack of four graph layers, labeled L0 through L3. L0 at the base holds every vector point. L1, L2, and L3 above it are progressively sparser random samples of those same points, connected by longer-range edges. The search enters at the top layer and descends, greedy-hopping toward the query on each layer before dropping to the one below. On the base layer it switches to a beam search, keeping the efSearch best candidates in a priority queue until no closer node can be found. The HUD in the top-right tracks everything as it happens: current layer, total graph hops, distance checks so far, number of beam candidates, best distance found, recall@5 versus a brute-force scan of the same graph, and the share of all vectors visited. Distance checks and visited share are measured on the rendered graph, not simulated, so the numbers reflect the index actually built in the page. In compare mode, two searches race side by side: the HNSW path in red and a brute-force sweep shown in white. The left panel's progress bars fill at their own pace so you can watch the check counts diverge. At 10K vectors the gap is dramatic. A few things to try: - **Jump from 100 to 10K vectors and watch the visited share.** Brute force always checks every stored vector, so its bar fills at the same rate regardless of size. The HNSW visited share shrinks as the dataset grows because the upper layers span more ground per hop. - **Drag efSearch down to 2 and then back up to 64.** Recall@5 responds: at low efSearch you may see 3/5 or 4/5 rather than 5/5. This is the core tradeoff, not a bug. When recall drops below what the application needs, raise efSearch at serve time. - **Switch M from 4 to 16 and rebuild.** Higher M means more connections per node on every layer, so the beam has more routes to explore. Distance checks typically rise, but recall tends to hold at lower efSearch values. M is a build-time decision, unlike efSearch. - **Tap the BOUNDARY query preset.** Edge queries are harder because the true neighbors cluster in one direction. Watch whether the greedy descent from upper layers enters the right neighborhood before the base-layer beam begins. - **Use the step button to advance one hop at a time.** You can follow each layer transition and see exactly when the search commits to a neighborhood. The top-layer hop count is usually very small (one to three) before the algorithm descends. ## How HNSW actually works A vector database stores embeddings as high-dimensional points and answers nearest-neighbor queries over them. The naive approach measures the distance from the query to every stored point and returns the closest ones. That is exact, but it costs one distance check per stored vector on every query. At tens of millions of vectors the latency is unacceptable. HNSW avoids the full scan by building a hierarchy of navigable graphs during index construction. Every vector is assigned to a level: all vectors live on L0, but a random fraction (controlled by a level multiplier) also appear on L1, a smaller fraction on L2, and so on. On each level, each node is connected to its M nearest neighbors among the nodes that share that level. The resulting graph has small-world properties: a few long-range shortcuts at the top and dense local connections at the base. At query time, the search enters at the highest non-empty layer at a fixed entry point. It greedily hops to whichever neighbor is closest to the query until no neighbor is closer than the current node, then descends one level and repeats, starting from the landing node. On the base layer (L0) it switches from pure greedy to a beam search: it keeps an ordered list of the efSearch best candidates seen so far and continues expanding the closest unvisited one until the list stops improving. The result is the ef closest nodes found on L0. The demo's construction is a teaching simplification. Each node is connected to its M kNN neighbors per layer rather than using the insert-one-at-a-time pruning heuristic that real HNSW implementations use. The demo also caps the graph at 4 layers for visual legibility. These differences are disclosed in the NOTES panel. The search itself is a genuine greedy-beam traversal on the graph actually built in the page. ## What efSearch and M actually control M is the graph's connectivity, baked in at index build time. It controls how many neighbors each node stores on each layer. Low M means sparse connections and fast hops but higher risk of the search entering a dead-end neighborhood and missing a true near neighbor. High M means more routes, better recall, and more memory per node. You cannot change M without rebuilding the index from scratch. efSearch is the beam width on the base layer, and you can tune it per query at serve time without touching the index. A higher efSearch keeps more candidates in the priority queue at once, so the beam explores more of the local neighborhood before committing to a result. The tradeoff is latency: each additional candidate means additional distance checks. This is why operators typically build with a moderate M and adjust efSearch dynamically when downstream accuracy requirements change. The recall@5 number in the HUD compares the 5 results HNSW returned to the 5 true nearest neighbors from a full scan. A score of 4/5 means one true neighbor was missed. That is the contract of an approximate index: you trade a small miss probability for skipping most of the distance checks. When you need guaranteed exact results, brute force (or exact indexes like flat FAISS) is the right tool, but at the scale where vector search matters, approximate is almost always the right default. For a broader view of how retrieval fits into an LLM pipeline, the [retrieval-augmented generation explainer](https://protrailblazer.com/posts/retrieval-augmented-generation/) walks through how nearest-neighbor search connects a query to a knowledge base before generation. And if you want to understand what goes into the vectors being searched, [embeddings in 3D](https://protrailblazer.com/posts/embeddings-in-3d/) shows how semantic similarity maps onto geometric distance. ## Frequently asked questions ### Why is HNSW approximate instead of exact? The greedy descent from upper layers commits to a neighborhood early. If the entry point on a layer happens to be far from the query's true neighborhood, the beam may converge on a local cluster that misses one or two true neighbors. Raising efSearch reduces this risk by keeping more candidates alive on the base layer, but it never eliminates it entirely without degenerating into a full scan. The approximation is the whole point: the algorithm is fast precisely because it stops exploring before it checks everything. ### What is recall@5 and why does the demo track it? Recall@5 is the fraction of the true 5 nearest neighbors that appear in the 5 results HNSW returned. A score of 5/5 means HNSW found every true neighbor; 4/5 means one was missed. The demo computes this live by running a full brute-force scan in the background and comparing the result sets. In production systems, recall@K at a target latency is the primary metric for tuning efSearch and M. ### How does M affect index quality versus cost? Higher M gives each node more neighbors on each layer, which means more paths through the graph and higher recall at the same efSearch, but memory per node grows linearly with M. A common production default is M=16 on the base layer and M=8 on upper layers (some implementations use 2M on L0). The right M depends on your dataset's intrinsic dimensionality and how much memory you can spare. ### Why do the upper layers hold fewer points? Each vector is assigned a maximum level during construction by drawing from a negative-exponential distribution. Most vectors land on L0 only; a fraction make it to L1; fewer still to L2; and so on. This exponential thinning is what gives HNSW its small-world topology: the sparse top layers span large distances quickly, and the dense base layer provides fine resolution. Without the thinning, all layers would be equally dense and the speed advantage would vanish. ### Can HNSW handle filtered search or deletes? The demo does not cover either, and they are genuinely hard problems. Deletes typically require a tombstone plus periodic re-indexing or a specialized soft-delete mechanism; true node removal risks disconnecting the graph. Filtered search (return neighbors that also satisfy a metadata predicate) can cause recall to collapse if the filter eliminates most candidates from the beam. Production systems handle this with pre-filtering, post-filtering, or hybrid indexes, each with its own recall tradeoffs. ## Key takeaways - HNSW is approximate on purpose. A recall@5 below 5/5 is not a failure, it is the contract: you trade a small chance of missing a true neighbor for skipping almost all of the distance checks, and you buy recall back with efSearch when you need it. - efSearch and M are different kinds of knobs. efSearch is a per-query setting you can raise at serve time, while M is baked in when the graph is built, so fixing recall in production usually means turning up efSearch, not rebuilding the index. - The upper layers hold no extra information. Every vector lives on the base layer, and the layers above are just sparse random samples of the same points, acting as long-range shortcuts that get the search near the right neighborhood in a few hops. - The payoff scales with the dataset. Brute force pays one distance check per stored vector on every query, while HNSW's path barely grows, which is why the demo's visited share collapses when you jump from 100 to 10,000 vectors. --- --- title: "Continuous batching: why your API latency varies" description: "A GPU scheduler timeline runs static and continuous batching on the same requests. Watch idle lanes refill, queues drain, and latency percentiles move live." published: 2026-06-13 updated: 2026-06-13 canonical: https://protrailblazer.com/posts/continuous-batching/ tags: [machine-learning, demo, llm, inference, batching, gpu, scheduling, latency] --- # Continuous batching: why your API latency varies **Answer:** Continuous batching is the scheduling strategy LLM servers use to keep a GPU full: the moment one request in the batch finishes, its slot goes to the next request in the queue instead of sitting idle until the whole batch drains. The refill happens between token steps, which is why it fits LLM inference so well. The demo below runs static and continuous scheduling on the exact same request stream so you can watch the idle gaps, the queue, and the latency percentiles diverge. The interactive below runs two schedulers on the same seeded request stream. Switch between Static Batch, Continuous, and Compare modes, adjust the traffic preset and batch capacity, and step through one iteration at a time to see exactly when a lane goes idle and when it gets refilled. *Interactive demo: [open the continuous batching demo](https://protrailblazer.com/demos/continuous-batching/)* ## What you're seeing The scene is a Gantt-style scheduler timeline. On the left, a queue column shows requests waiting to be assigned a lane. In the center, each horizontal lane is one slot in the batch. Each cell in a lane is one token step: solid red means the lane is decoding (producing one token per step), hatched dark red marks the prefill pass (processing the prompt before the first output token appears), and striped gray marks idle static-batch dead capacity. Completed requests stack at the right with a latency bar broken into wait, prefill, and decode time. The HUD shows GPU utilization, tokens per step, queue depth, p50 and p95 latency in token steps, throughput per 100 steps, and total completed requests. In Compare mode, both panes run on one clock so the gap between schedulers accumulates in real time. The queue drains at different rates, the utilization sparklines diverge, and the latency percentiles in the HUD update side by side so you can read the exact delta between strategies. All values are labeled SIMULATED. Time is measured in token steps, not milliseconds. The simulation uses a deterministic seed (1337) so Reset and preset changes replay the same trace for the same settings. A few things to try: - **Start in Compare mode with the Mixed Lengths preset.** Watch the striped idle gaps accumulate in the static pane while the continuous pane keeps lanes filled. The p95 latency numbers in the HUD drift apart within a few dozen steps. - **Slide length variability to zero with Steady traffic.** When every request decodes the same number of steps, every lane finishes at the same time, no lane strands early, and the two schedulers behave nearly identically. The utilization gap nearly closes. - **Push the arrival rate above what the lanes can handle.** The queue column grows in both panes under overload. Continuous drains it faster but does not repeal queueing. This is the clearest signal that batching strategy and capacity are separate levers. - **Enable Prefill cost and switch to Long Context.** Long prompts charge several steps of hatched prefill before decode starts, delaying the first output token and slowing the lanes around them. You can see why long-context requests raise p95 latency more than p50. - **Use Step +1 while paused.** Each click advances one scheduler iteration. In the continuous pane, you can catch the exact step where a finished lane refills from the queue. In the static pane, that slot stays gray until the rest of the batch finishes. ## How continuous batching actually works LLM generation is iterative. Each forward pass produces one output token per active lane and returns control to the scheduler. That natural decision point is what makes continuous batching possible: after every iteration, the scheduler checks whether any lane finished and, if so, immediately pulls the next request from the queue into that slot. Static batching does not check between iterations. The scheduler assembles a batch, runs it until the longest request in the batch finishes, and only then accepts the next group. If one request decodes 8 steps and another decodes 40, the 8-step lane sits dark for the remaining 32 steps while the batch holds its size. The mechanism that makes this affordable is the KV cache. Each request stores its attention keys and values for every position generated so far. When a new request joins mid-batch, it computes its own KV entries from scratch during prefill, then runs in parallel with the requests already decoding. Memory for the KV cache grows with the number of active requests and their sequence lengths, which is why real serving systems add KV cache memory management and paging on top of the basic continuous-batching logic. Prefill and decode are distinct phases. During prefill, the server processes the entire prompt in a single (or a few) forward passes before the first output token appears. During decode, the model generates one token per step per lane. Long prompts can occupy a lane for many prefill steps, which is why the simulation separates the two visually and why real systems sometimes schedule prefill and decode on separate resources. ## Why utilization is not the whole story Continuous batching raises GPU utilization, but utilization is not the metric users experience. What users feel is time-to-first-token (TTFT) and inter-token latency. A fully utilized GPU can still have high TTFT if the queue is long and each incoming request has to wait for a lane to open. The context window length of each request determines how much KV cache memory it needs and for how long. A server running 8 lanes at a long context window may be as memory-constrained as one running 32 lanes at a shorter window. Real serving systems use admission control to refuse requests that would overflow KV cache memory, which is a limit the simulation's caveats panel calls out explicitly. The p50 latency the demo shows is the median total request latency in token steps. The p95 is the 95th percentile. In traffic with uniform decode lengths, the two track closely. Add length variability and the p95 climbs because a small fraction of long requests block lanes and delay the queue behind them. Continuous batching reduces that effect but does not eliminate it, because the long requests are still consuming a lane for their full decode length. To see how each output token is produced step by step, the [autoregressive generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text/) covers the full path from prompt through the KV cache to a sampled token. For the tradeoff between compute per step and output quality, see [inference-time compute](https://protrailblazer.com/posts/inference-time-compute-explained/). ## Frequently asked questions ### What is the difference between static batching and continuous batching? Static batching locks a batch together until the longest request finishes, leaving short-finishing lanes idle for the remainder. Continuous batching checks after every token step and refills any freed lane immediately from the queue, so idle time between requests is measured in one step rather than the full remaining decode length of the slowest batchmate. ### Why does continuous batching help more when decode lengths vary? When all requests in a batch decode the same number of steps, every lane finishes at the same time and there is no dead capacity to reclaim. When lengths vary, some lanes finish tens of steps early. Continuous batching fills those lanes immediately; static batching strands them until the batch ends. The larger the spread in decode lengths, the bigger the utilization and throughput gap. ### Does continuous batching always reduce p95 latency? It reduces the component caused by static-batch stranding, but not all sources of p95 latency. A long request still occupies a lane for its full decode length and holds up everything behind it in the queue. Under overload, the queue grows in continuous mode too, just more slowly. Real systems add request prioritization, speculative decoding, and capacity scaling on top of continuous batching to address those remaining sources. ### What role does the KV cache play in continuous batching? The KV cache stores the attention keys and values for every token generated so far, per request. When a new request joins a running batch, it needs its own KV memory for the prefill pass and then for each decode step. Memory for all active lanes accumulates in GPU VRAM. Real serving systems limit the number of active requests partly by KV cache memory, not just by the number of lanes the hardware can process in parallel. ### Why does the simulation measure latency in token steps rather than milliseconds? Token steps are the natural unit of the scheduler's logic and make the relative cost of static versus continuous batching visible without hardware-specific numbers. Real API latency adds network round-trip time, queuing before the request reaches a GPU, prefill compute time (which scales with prompt length), and provider-side load. The simulation's caveats panel lists those additional factors explicitly. ## Key takeaways - Static batching's real cost is variance, not batching. Every request in the batch holds its lane until the longest one finishes, so a lane that finishes early sits as dead capacity and a short request inherits the latency of its slowest batchmate. - Continuous batching only works because LLM generation is iterative. One token per lane per step gives the scheduler a decision point at every iteration, so a freed lane can be refilled between tokens instead of between batches. - When decode lengths are uniform, the two schedulers behave almost the same. Load the steady preset and slide length variability to zero and the gap in utilization and p95 nearly closes, because no lane ever finishes far ahead of its batch. - Continuous batching raises utilization, it does not repeal queueing. Push the arrival rate past what the lanes can decode and the queue grows in both panes; the continuous pane just drains it faster. - Prefill and decode are separate costs. A long prompt occupies a lane for several steps before its first output token appears, which is part of why long-context requests slow down the requests around them. --- --- title: "Constrained decoding: forcing an LLM to emit valid JSON" description: "Watch a schema grammar mask illegal tokens, renormalize the survivors, and force a simulated LLM to emit valid JSON while a prompt-only run fails beside it." published: 2026-06-09 updated: 2026-06-09 canonical: https://protrailblazer.com/posts/constrained-decoding-json/ tags: [machine-learning, demo, llm, constrained-decoding, json] --- # Constrained decoding: forcing an LLM to emit valid JSON **Answer:** Constrained decoding is how you make an LLM emit output that always parses. At every generation step, a grammar compiled from your schema deletes every candidate token that would break the structure, renormalizes the probabilities of the legal survivors, and only then samples. The demo below runs the same loop with the mask on and off, so you can watch the difference between asking for valid JSON and guaranteeing it. The interactive below generates a small JSON document one token at a time, with a schema grammar standing between the model and the output. Watch the red sweep delete illegal candidates in the center histogram, then flip to Prompt Only, set the pressure to Hostile Logits, and watch the same distribution fall apart without the mask. *Interactive demo: [open the constrained decoding demo](https://protrailblazer.com/demos/constrained-decoding/)* ## What does the demo show? Three panels share one generation loop. The left panel is the output stream: a prompt asks for a user record as JSON, and the document below it grows one token at a time. A comment line under the prompt says whether a decoder mask is compiled or whether the model is just being asked nicely. Keys and values print in normal JSON colors, tokens that break syntax flash red, and tokens that parse but violate the schema get a red underline. The footer keeps a running tally of syntax and schema passes across runs, then prints a per-run verdict with the reason it failed, like an unquoted key or a value outside the enum. The center histogram is the heart of the sim. Each bar is one candidate for the next token, sized by its probability, and the panel header cycles through the phases of a single step: predict, mask, renormalize, sample, commit. During the mask phase a red scanline sweeps across the bars and collapses every candidate the grammar forbids, leaving a dashed outline where its raw probability used to be. During renormalize, the surviving bars grow until they sum to 1 again, and the winning bar flashes bright before its token commits to the document. Every probability here is simulated from seeded logits, not a live model, and the panel is labeled that way. The right panel is the compiled schema grammar: a state machine with one row per slot, from the opening brace through key, colon, value, and separator for each field, down to an explicit end token. The current slot glows red, finished slots dim, skipped slots get struck through, and violated slots pick up a flag. Below the rail, chips list the legal next tokens; in constrained mode the header reads enforced, in prompt-only mode it drops to advisory only. The stat block in the top right tracks the raw candidate count, how many the mask removed, how many survived, and the leftover probability mass being rescaled to 1.00, plus live JSON VALID and SCHEMA VALID readouts. A few things to try in the demo: - **Switch to Prompt Only and pick Hostile Logits.** The model keeps reaching for a prose preamble or a markdown fence, the parse-error strip lights up on the grammar rail, and JSON VALID flips to FAIL. Runs reseed each time, so some survive and many don't; the footer tally keeps score. - **Try Messy pressure in Prompt Only mode.** Messy runs usually stay parseable but pick a wrong key or an enum value the schema never allowed, so the footer shows syntax passing while schema fails. That split is the sneakiest failure mode in production. - **Arm Inject Failure.** In Prompt Only mode it forces the next step to sample the highest-probability illegal candidate, so you can watch exactly how one bad token derails a run, or quietly poisons it. - **Flip back to Schema Constrained and leave Hostile Logits on.** The sweep deletes the junk on every single step and both verdicts read PASS no matter how bad the raw distribution gets. That asymmetry is the whole argument for the technique. - **Push the **temperature** slider to 2.00 in each mode.** Constrained, the bars flatten but the output stays valid. Prompt-only, that flat distribution is exactly how low-probability junk gets sampled. ## How does constrained decoding actually work? Each step starts with the model scoring a handful of candidate continuations. A softmax with the current temperature turns those scores into the probability bars you see in the predict phase. In this sim the scores are hand-set and shifted by the pressure preset: cooperative boosts the legal candidates, messy boosts tokens that parse but violate the schema, and hostile boosts tokens that break JSON outright. The schema is compiled ahead of time into the state machine on the right rail. For these three-field schemas that is fourteen rows: an opening brace, then key, colon, value, and separator for each field, then a closing brace and an end-of-sequence token. At any moment the grammar knows exactly which tokens are legal next, and at the value slots it also encodes the type: a quoted string for name, a bare number for age, one of three enum values for role. In constrained mode, the mask runs between prediction and sampling. Every candidate outside the legal set for the current slot gets its probability set to exactly zero, and the survivors are divided by the mass that remains so they sum to 1 again. Sampling then happens only over the survivors. That is the entire trick: an illegal token is not discouraged or penalized, it is unsampleable. The renormalization also preserves the model's preferences, because the legal candidates keep the same ratios to each other they had before the mask. In prompt-only mode the sample runs over the raw distribution, so failure is a matter of odds. The sim keeps two failure classes separate. Hard failures break JSON syntax: an unquoted key, a missing colon, a markdown fence, or a prose opener like "Sure". Once prose starts there is no legal recovery in this simple grammar, so the run derails into filler words until it hits a stop token. Soft failures are sneakier: "username" instead of "name", "superuser" when the enum allows admin, editor, or viewer, or a quoted "34" where a bare number belongs. Those runs finish, parse cleanly, and still fail schema validation. Real systems play the same trick with more machinery. A production tokenizer emits subword pieces, so the engine has to track grammar legality across partial lexemes, and libraries typically compile a JSON Schema into a token-level automaton before generation begins. Masking during sampling is not the only option either: some stacks repair or re-ask after generation instead of intervening in the loop. This is the machinery behind the JSON and structured output modes that LLM APIs expose, and it is what makes reliable tool calls possible, since a tool's arguments are just JSON validated against a schema. The guarantee stops at structure. A schema-valid record can still claim the wrong age or pick a plausible but wrong enum value, which the demo's caveats dialog spells out. Constrained decoding sits in the same sampling toolbox as [top-k and top-p sampling](https://protrailblazer.com/posts/top-k-top-p-sampling-explained/) and [temperature](https://protrailblazer.com/posts/llm-temperature-explained/), and it composes with them rather than replacing them. All of it rides on the token-by-token loop covered in [our autoregressive generation demo](https://protrailblazer.com/posts/how-llms-generate-text/). ## Frequently asked questions ### What is constrained decoding in an LLM? Constrained decoding restricts an LLM's next-token choices to a grammar during generation. At each step the decoder zeroes the probability of every token that would violate the grammar, for example a JSON Schema compiled into a state machine, renormalizes the rest, and samples from the survivors. The output is structurally valid by construction rather than by luck. ### Why do LLMs produce invalid JSON without constraints? Because the model samples from a probability distribution over its whole vocabulary, and that distribution always puts some mass on prose preambles, markdown fences, unquoted keys, and near-miss values. A prompt can shrink that mass but cannot zero it, so across enough runs some outputs will break the parser or drift from the schema. ### Does constrained decoding guarantee correct output? No. It guarantees shape, not truth. Every brace, key, and type will match the schema, but the values are still the model's guesses, so a schema-valid document can carry a wrong name or a wrong number. Semantic validation stays downstream. ### Does masking distort what the model would have said? Less than you might expect. Renormalization divides each surviving probability by the total legal mass, which preserves the ratios between the legal candidates. If the model preferred one legal continuation over another before the mask, it still does after; the mask only removes options that were never acceptable. ## Key takeaways - Constrained decoding does not persuade the model to behave. It zeroes the probability of every token the grammar forbids and renormalizes the survivors, so an illegal token has exactly zero chance of being sampled. - Prompt-only failures come in two kinds, and retries only catch one. Hard failures (prose preambles, unquoted keys, markdown fences) break the parser and get noticed; soft failures (a wrong key, a string where a number belongs) parse cleanly and slip through. - Temperature stops being dangerous once the mask is on. Raising it spreads probability across the legal candidates only, so you get more varied output without ever leaving the grammar. - The guarantee is structural, not semantic. A schema-valid document can still carry a wrong or useless value, so constrained decoding replaces JSON repair code, not validation of meaning. - Production grammar engines work on subword pieces, not whole lexemes like this sim, so they compile the schema into a token-level automaton that tracks legality across partial strings. --- --- title: "How attention works: search, feeds, and LLMs" description: "See how attention works in search, social feeds, and LLM self-attention. One interactive runs the same score, normalize, allocate loop through all three." published: 2026-06-07 updated: 2026-06-07 canonical: https://protrailblazer.com/posts/how-attention-works-across-systems/ tags: [machine-learning, demo, attention, pagerank, transformers] --- # How attention works: search, feeds, and LLMs **Answer:** Attention, in any ranking system, is the same three-step move: score every candidate, normalize the scores into weights, then hand out a scarce resource by those weights. Search does it to pages, a social feed does it to posts, and a language model does it to tokens. What changes is the objective each one scores for, and that one difference changes everything downstream. The interactive below shows how attention works in three different systems: a search engine ranking pages, a social feed ranking posts, and a language model attending over words. Switch tabs to move between them, and open the compare tab to line them up side by side. Click the fullscreen button in the top right of the demo for a closer look. *Interactive demo: [open the attention three systems demo](https://protrailblazer.com/demos/attention-three-systems/)* ## What does the demo show? The demo has five tabs. **The pattern** is the shared skeleton, one diagram you can morph between the three systems. **Search** runs real PageRank on a small web of pages. **Social** runs a live feed that sorts posts by predicted engagement. **Attention** runs real scaled dot-product attention over a sentence. **Compare** lines all three up against the five differences that matter. A few things to try: - **Add a link in the search tab.** Click one page, then another, to draw a link between them. The ranking recomputes and the page you pointed at climbs. Authority is conferred by who points at you. - **Let the social feed run.** Watch the sensational posts climb while the careful ones sink, then press reveal true quality to see the gap between what the feed surfaces and what is actually good. - **Swap big for small in the attention tab.** The sentence is "the trophy did not fit in the suitcase because it was too big." Toggle the predicate and watch what the word "it" points to flip between the trophy and the suitcase. - **Toggle the attention heads.** One head tracks meaning, another tracks position. Real models run many at once, which is multi-head attention. ## What do search, social, and LLMs have in common? Underneath the different interfaces, all three run the same three steps. 1. **Score.** Give every candidate a number for how relevant it is. Pages get a PageRank score from their inbound links. Posts get a predicted engagement score from their features. Tokens get a score from the query dotted with each key. 2. **Normalize.** Turn the raw scores into comparable weights. Search and the feed sort them into a ranking. Attention runs a softmax, which squashes the scores into weights that are all positive and sum to 1. 3. **Allocate.** Spend a scarce resource by those weights. Search spends slots on the results page. The feed spends impressions. Attention spends its output on a weighted blend of value vectors. Routing inside a mixture-of-experts model is the same move again: a gate scores the experts, a softmax normalizes, and each token is sent to the top few. If you have seen [how mixture-of-experts routing works](https://protrailblazer.com/posts/mixture-of-experts-explained/), this pattern will feel familiar. ## How does attention differ across the three systems? The steps are shared. The differences are in what gets scored, what objective the score serves, how often it is recomputed, and what the output looks like. Five axes separate them, and the compare tab lays them out in full. ### Search: authority, scored recursively Search scores pages by authority. A page is important if important pages link to it, which is circular on purpose. PageRank resolves the circularity by iterating until the numbers stop moving. It is structural and slow: authority only shifts when the link graph shifts. The output is a hard ranked list, and the classic failure mode is people gaming that graph with link farms. PageRank is the classic core of search ranking, but modern engines layer hundreds of additional learned signals on top, so treat it as the foundation, not the whole system. ### Social: engagement, scored by a proxy A feed scores posts by predicted engagement, not by quality. It runs live: the current ranking decides who gets impressions, those impressions generate real engagement, and that engagement re-sorts the feed for the next round. That feedback loop is what makes the feed different from the other two. It produces rich-get-richer dynamics, and because engagement is only a proxy for what you want, the feed can drift away from quality while its own metric keeps climbing. ### Attention: relevance, scored fresh per input Self-attention scores tokens against each other for one job: predicting the next token. Each token emits a query, a key, and a value (the query, key, value framework), the query dots with every key to make scores, and the softmax turns those scores into weights that sum to 1. The output for each token is a blend of the value vectors, not a single winner. It is recomputed from scratch for every input, and it all [serves next-token prediction](https://protrailblazer.com/posts/how-llms-generate-text/). There is no human feedback loop inside the forward pass. ## Why is engagement the dangerous objective? Here is the honest part. The same score, normalize, allocate machinery is harmless when the target is relevance or prediction, and corrosive when the target is engagement. > **warn:** Engagement is a stand-in for what you actually want. It can pull away from that, and the feedback loop amplifies the gap. Search can be gamed and attention is just arithmetic, but only the engagement objective has an incentive to show you something that is worse for you, because it predicts you will click. Change the objective and you change the whole system. That is why "the algorithm" is not one thing. A search ranker, a feed, and a transformer share a skeleton, but the goal you point that skeleton at is what decides whether it serves you or feeds on you. ## Frequently asked questions ### What do search engines, social feeds, and language models have in common? All three run the same three-step pattern: they score every candidate for relevance, normalize the scores into weights, then allocate a scarce resource by those weights. Search scores pages, a feed scores posts, and a language model scores tokens. The machinery is shared. The objective each one optimizes is what makes them behave so differently. ### Is LLM attention the same as a search ranking? No. A search ranking ends in a hard ordered list where the top results win and the rest are cut below the fold. Attention ends in a soft blend: a softmax gives every token a weight, the weights sum to 1, and the output mixes their values. Attention never crowns a single winner, and it serves next-token prediction rather than authority. ### Why do social feeds show sensational content? Because they optimize predicted engagement, and sensational, novel posts reliably win clicks, likes, and shares. Engagement is a proxy for what you want, so the feed can surface low-quality content that happens to be engaging. A live feedback loop makes it worse: the posts on top get the most impressions, which gathers the most engagement, which keeps them on top. ### Does PageRank still power Google search? PageRank is the classic core idea, that a page is important if important pages link to it. It is still a foundation, but modern search layers hundreds of additional learned signals on top of it, so no serious ranking today is PageRank alone. ### What does the softmax do in attention? The softmax turns raw relevance scores into weights that are all positive and sum to 1. That is the normalize step. It lets attention treat the scores as a distribution and produce a weighted average of the value vectors instead of a single pick. ## Key takeaways - All three systems run the identical skeleton: score every candidate, normalize the scores, allocate a scarce resource. The shape is shared even though the engineering is not. - The objective is the whole story. Link authority, predicted engagement, and next-token prediction produce completely different behavior from the same machinery. - Search and a feed both end in a hard ranked cut. Attention never picks a winner. It outputs a blend whose weights sum to 1. - The feed is the corrosive one, not because its math is worse, but because engagement is a proxy for what you want, and its feedback loop amplifies the gap between the two. - PageRank is only the classic core of search, and attention is one layer among many in a real model. Both are foundations, not the whole system. --- --- title: "Context window: what an LLM sees, and what it forgets" description: "Watch two LLM context windows, 2k and 32k tokens, slide through the same text and drop exactly the tokens that scroll off the back of the smaller one." published: 2026-06-02 updated: 2026-06-02 canonical: https://protrailblazer.com/posts/context-window-explained/ tags: [machine-learning, demo, llm, context-window, tokens] --- # Context window: what an LLM sees, and what it forgets **Answer:** A context window is the fixed span of recent tokens an LLM can actually read at once. Everything inside it can shape the next answer, and everything that scrolls off the back is gone, as if it was never said. The demo below slides two window sizes, 2,000 and 32,000 tokens, through the same document so you can watch exactly which tokens each one forgets. The interactive below slides two context windows through the same document at the same speed. The top lane is a 2,000-token window, the bottom is 32,000. Press play, try the preset documents, or paste your own text and watch which tokens scroll off the back of each window as the playhead advances. *Interactive demo: [open the context window demo](https://protrailblazer.com/demos/context-window/)* ## What does the demo show? Two stacked lanes scroll the same text from right to left. The top lane is the context window of a 2,000-token model; the bottom lane is a 32,000-token one. Each small block is a token, not a word. As the playhead advances, new tokens enter on the right and old ones slide toward the left edge, where the smaller window starts dropping them. The strip across the top is the whole document, with two highlighted bands showing how much of it each window currently covers. The smaller band makes the gap obvious: on a long input, the 2k window only ever sees a thin slice. The strip along the bottom plots tokens lost over time for both windows. The readouts track POS (the playhead's position in tokens), 2K_LOST and 32K_LOST (how many tokens have fallen off the back of each window so far), and FLUX (tokens per second at the current speed). A few things to try in the demo: - **Load the long contract preset and press play.** The 2K_LOST counter starts climbing within seconds while 32K_LOST stays at zero. That gap is the whole point. - **Paste your own text.** A long email thread or a book chapter makes the difference visceral. Watch the token count as you paste: it climbs faster than the word count does. - **Drag the scrubber back and forth.** Scrubbing pauses autoplay and lets you line up the exact moment a specific sentence drops out of the 2k window. - **Bump the speed to 4x.** On a long document the small window churns through its entire capacity again and again while the large one barely fills. ## How does a context window actually work? A context window is the maximum number of tokens a model can attend to in a single forward pass. It is a hard architectural limit, fixed when the model is trained, not a preference you can talk it past. Everything inside the window is fair game for the next prediction. Everything that has scrolled past the edge is gone, and the model has no memory of it at all. The window is measured in tokens, not words or characters. This demo uses the GPT-4 `cl100k_base` pre-tokenizer, which splits text into subword pieces and punctuation. As a rough rule, 1,000 tokens is about 750 English words, so a 2,000-token window holds maybe three or four pages. Code, non-English text, and unusual formatting tokenize less efficiently and fill the window faster. When a conversation or document runs longer than the window, the runtime slides the window forward, dropping the oldest tokens to make room for new ones. That is what the lanes show. In a chat app it is why a long session eventually 'forgets' how it started: those early tokens were evicted to fit the latest turn. A larger window helps, but it is not perfect memory either. Models tend to attend most strongly to the beginning and end of the window and least to the middle, an effect often called 'lost in the middle.' A fact sitting in the center of a nearly full 32k window can be present but effectively ignored. When the window is the bottleneck, the fix is rarely to prompt harder. It is to put the right tokens back in front of the model. [Retrieval-augmented generation](https://protrailblazer.com/posts/retrieval-augmented-generation/) does this by fetching only the most relevant chunks at query time, and a running summary can compress old turns into a few tokens. The window itself is held in GPU memory as the [KV cache](https://protrailblazer.com/posts/how-llms-generate-text/), which is why bigger windows cost more to serve. ## Frequently asked questions ### What is a context window in an LLM? A context window is the fixed maximum number of tokens a model can read at once, counting your prompt, any system instructions, the retrieved or pasted text, and the response it is generating. Anything beyond that limit cannot influence the output. It is set by the model's architecture, so a 32k model simply cannot consider a 40k-token input without dropping part of it. ### How many words fit in a context window? Roughly 750 English words per 1,000 tokens, so a 2,000-token window holds about 1,500 words and a 32,000-token window about 24,000. The exact ratio depends on the text: code, punctuation-heavy content, and non-English languages use more tokens per word, so they fill the window faster. ### What happens when text exceeds the context window? The runtime slides the window forward and drops the oldest tokens to make room, so the model loses access to whatever scrolled off the back. It does not summarize or compress on its own; those tokens are simply gone from its view. This is the usual reason a long chat session seems to forget how it began. ### Does a bigger context window always give better answers? Not necessarily. Beyond fitting more text, models often recall information at the start and end of the window better than the middle, an effect called 'lost in the middle.' A focused, well-retrieved 2k of context can beat a padded 32k window, which is why retrieval and good prompt ordering still matter even on long-context models. ## Key takeaways - A context window is a hard limit, not a soft preference. Once a token slides past the edge the model cannot reach it, so there is no fading memory, just an abrupt cutoff. - Token count is not word count. The cl100k_base tokenizer splits on subwords and punctuation, so 2,000 tokens is roughly 1,500 English words, and code or non-English text fills the window faster. - A bigger window does not mean even recall across all of it. Models tend to attend most to the start and end of the window, so a fact buried in the middle of a full 32k window can be present but effectively ignored. - Most 'the model forgot what I said' moments are context-window evictions, not reasoning failures. The fix is usually to put the dropped tokens back in front of the model with a summary or retrieval, not to re-prompt harder. --- --- title: "36 ways AI works alongside a freelance web developer" description: "A 3D tour of six freelance web project phases with six AI assists each, plus a toggle showing which 16 of 36 handoffs can drop the human approver." published: 2026-05-30 updated: 2026-07-07 canonical: https://protrailblazer.com/posts/freelancer-ai-workflow/ tags: [ai workflow, freelancer ai, web development ai, llm as judge, agentic workflow, human in the loop, ai assist, freelance web developer, machine learning, automation] --- # 36 ways AI works alongside a freelance web developer **Answer:** Of the 36 things a freelance web developer typically does on a project, AI now meaningfully helps with all 36, and an LLM acting as the judge can take over 16 of those handoffs without a human pressing approve. The rest stay human because they are relational, accountable, or out of distribution, not because the model cannot draft them. The interactive below tours one freelance web project end to end, six phases of six AI assists each: discovery, design, build, QA, launch, support. Press play to ride through the whole thing, click any node or row to inspect a single handoff, or flip the LLM as judge toggle to see which 16 of the 36 hand-offs can drop the human approver and run on their own. *Interactive demo: [open the freelancer ai workflow demo](https://protrailblazer.com/demos/freelancer-ai-workflow/)* ## What you're seeing The figure at the center is the freelancer. Six clusters of nodes orbit around them, one per phase of a typical web project. Each cluster holds six AI assists, each one a real handoff a freelancer does on most jobs (intake questionnaire, sitemap drafts, layout variants, regex authoring, release notes, SEO meta review). The thin red beam from the freelancer's head to the active node is the current ask. Faint trails are previous asks in the same tour, so the constellation builds up as you go. The right panel shows only the current phase's six assists, split into two roles per row: what the AI drafts (red hex) and what you still own (warm-white person silhouette). The top-right HUD tracks Phase, Assists lit out of 36, Coverage percent, and Fully auto, the count of handoffs that have flipped to a fully-AI loop. A few things to try in the demo: - **Press Play.** The camera pans to Discovery, ignites all six Discovery assists in sequence, then pans to Design and repeats through Support. The bottom-left card always names the current handoff and what it does. - **Toggle LLM as judge.** The 16 eligible nodes morph from red wireframe octahedrons into glowing white atoms with three orbiting electrons. The Fully auto count in the HUD lights up to track how many of those have lit up in the current tour. - **Click any row or node.** The camera jumps to that phase if needed, the panel scrolls to the row, and the bottom-left card shows the full description of what AI does and what you still do. - **Use the phase tabs.** Tabs 1 through 6 jump between Discovery, Design, Build, QA, Launch, and Support. Press Play again to resume from where you left off. - **Pull the speed slider.** 0.5x lets you read every assist as it ignites. 3x is the high-altitude flyover. ## How it actually works Each of the 36 nodes is a real freelancer handoff with three fields: what the AI does, what you do, and (for 16 of them) what an LLM judge does instead. The assists are not invented for the demo. "Bullets transcript into decisions", "spits out three to four layout options", "writes rules and messages", "audits and proposes rewrites". These are the reasoning steps a freelancer actually outsources to a model on real jobs, the same shape as a chain of thought that produces an artifact instead of an answer. When LLM as judge is off, every row has two roles: AI drafts, you validate. That is the human-in-the-loop pattern that runs almost every freelancer-with-AI workflow today. You hold the approve button. The 16 nodes marked judge-eligible have a single property in common: their human role is rubric-shaped. Compare against a transcript. Score brand fit against guidelines. Verify behavior preservation. Surface missing test cases. These tasks are not creative, they are evaluative, and they have a clear right answer or a clear scoring criterion. Flip the toggle and those rows swap to a JUDGE role. The pattern, LLM as judge, is exactly what it sounds like: a separate model call (often the same family) graded against the rubric instead of the human. In the demo this is visualized as the white atom with three electrons orbiting, deliberately distinct from the red octahedron. When the judge agrees, the handoff is fully autonomous. When it does not, the work goes back to the AI for a retry, and only escalates to you on repeated failure. For a deeper look at the bias modes a judge can hide, see our [full LLM-as-judge breakdown](https://protrailblazer.com/posts/llm-as-judge-explained/). Three details map to how this looks in real systems: 1. **Drafting and judging are the same model class.** The judge is not magically smarter, it is the same kind of LLM call with a different prompt and a clear criterion. The architecture that wires drafter, judge, and retry loop together is an agentic workflow, not a single super-model. Our [agentic workflow guide](https://protrailblazer.com/posts/agentic-workflow-smb-guide/) walks that whole pattern end to end. 2. **Each judge call is a scoped agent.** It has one job ("score this draft against this rubric"), no other tools, and a fixed output shape. That makes it a textbook agent in the strict sense, narrow and disposable, not the chat-anything kind. 3. **Some assists call out to real systems.** Bug reproduction needs a code runner. SEO meta review needs a crawl. Test fixture data needs a database connection. Those steps are tool use inside the assist, not just text generation, and the judge has to be allowed to read the same tool output to score against ground truth. On the layout-variant assist ("spit out three to four options so you pick rather than start blank"), the judge step is roughly a best-of-N pattern: generate four candidates, have the judge score each on the rubric ("matches brand, balanced grid, clear hierarchy, accessible contrast"), and surface the top one. The freelancer never sees the losers. That is a meaningful change in where attention gets spent. ## Where this breaks for a real freelancer The constellation is clean. Live workflows have four kinds of mess that don't show up in a node-and-beam diagram. First, the rubric is the product. A judge with a vague rubric ("is this on brand?") will hallucinate a confident yes on bad work, then a confident no on good work, and the variance hides inside the average. The 16 judge-eligible assists are eligible because their rubric can actually be written down. If your brand voice can't fit in five lines of criteria, the judge cannot evaluate it, and that step belongs back with you. Second, accountability does not delegate. "Tailors and sends to client" is not judge-eligible. Not because the AI cannot draft the email, but because the freelancer is the one whose name is on the relationship. A judge can score the draft against a rubric all day. It cannot take responsibility for the send. Third, the long tail is where judging fails. On the most common case ("a standard validation form for a standard signup"), the judge will be right almost every time. On the weird case ("the client wants the date picker to default to last Wednesday because of a regulatory thing"), the judge has no rubric for that and will silently approve a wrong implementation. A judge-replaceable workflow is only safe with an escalation rule when confidence is low or the input is out of distribution. Fourth, the work that survives is hard to explain. After the judge takes over the rubric-shaped 16, what is left for the freelancer is exactly the relational, the accountable, the contextual, and the genuinely novel. That is harder to scope on a quote and harder to bill hourly. The economics shift faster than the contracts do. ## Key takeaways - AI assist is no longer a yes-or-no question on a freelance web project. It shows up in every phase, from intake to support, and the meaningful axis is which steps still need a human approver, not which steps the AI can touch at all. - Of 36 typical handoffs, 16 are rubric-shaped enough for an LLM judge to take over. The remaining 20 stay human because they are relational, accountable, or context-bound, not because the model cannot draft them. - The boundary between human-in-the-loop and fully autonomous tracks the shape of the validation work, not the difficulty of the creative work. A drafting step can be cheap and a checking step can be expensive, but only the rubric-shaped checking step is safely judge-replaceable. - An LLM judge is not a smarter LLM. It is the same model class with a clear criterion and a narrow prompt. It works exactly as well as the rubric you can write. - A solo freelancer who routes the 16 judge-replaceable steps to an automated reviewer does not just go faster, they raise the floor. The bar moves from "whatever I had attention for this week" to "whatever the rubric says", which is usually higher than a tired Friday afternoon human. --- --- title: "Retrieval-augmented generation: how LLMs look things up" description: "A 3D demo of a RAG pipeline: a query vector searches a corpus, the top-K nearest chunks light up, and the LLM grounds its answer in the retrieved text." published: 2026-05-26 updated: 2026-07-07 canonical: https://protrailblazer.com/posts/retrieval-augmented-generation/ tags: [retrieval-augmented generation, rag, vector search, embeddings, semantic search, top-k retrieval, hallucination mitigation, context window, llm, machine learning] --- # Retrieval-augmented generation: how LLMs look things up **Answer:** Retrieval-augmented generation is an LLM that looks things up before it answers. A query is turned into a vector, the vector finds the most similar chunks in a document store, those chunks get stuffed into the context, and the model generates its response conditioned on them. The payoff is fresher facts and fewer hallucinations without retraining the model. The interactive below shows a RAG pipeline in 3D. A query fires into a cloud of document chunks, the closest ones light up, and those chunks are what the LLM sees when it writes its answer. Drag to orbit, scroll or pinch to zoom, and hit 'new query' to send another shot into the corpus. *Interactive demo: [open the rag demo](https://protrailblazer.com/demos/rag/)* ## What does the demo show? Each dim red point is a chunk of a document that has already been turned into a vector. A chunk is a paragraph or a section, not a whole document, because retrieval works better when each unit is small enough to be about one thing. When you fire a query, a brighter node appears at the query's location in the same vector space. Lines draw from the query to its top-K nearest neighbors, and those neighbors are what gets handed to the LLM as context. SIM_SCORE is the cosine similarity of the best retrieved chunk. TOKENS is the size of the packet being sent to the generator. A few things to try in the demo: - **Drop top-K to 1.** The model gets exactly one chunk. If that chunk is right, great. If it is almost-right, the model has nothing to cross-reference and will fall back on its parametric memory, which is where hallucinations come from. - **Raise top-K to 6.** More context, more chances of including the right chunk, and more chances of drowning it in irrelevant ones. Watch how quickly the token count climbs. - **Orbit the cloud.** Clusters are meaningful. Nearby chunks are semantically similar because they share a region of embedding space. Queries land in whichever cluster best matches their meaning. ## How does RAG work? RAG has four steps: index, embed, retrieve, generate. Index. Documents are split into chunks, typically 300 to 800 tokens each. Boundaries matter: splitting mid-sentence or mid-idea destroys retrieval quality. Structure-aware splitting uses headings, paragraphs, or sentence boundaries. Embed. Each chunk is passed through an embedding model that maps it to a high-dimensional vector. Similar text lands in similar regions. Our post on [embeddings](https://protrailblazer.com/posts/embeddings-in-3d/) goes into what those vectors actually encode. Retrieve. At query time, the query itself is embedded with the same model and the system finds the K chunks closest to the query vector. Cosine similarity is the usual distance. Production systems use approximate nearest-neighbor indexes (HNSW, IVF, ScaNN) because brute-force scoring is linear in the corpus size. Generate. The retrieved chunks are formatted into the context window alongside the question, and the LLM generates its answer conditioned on them. The prompt usually asks the model to cite which chunk it used for which claim. Because the model sees the source text at inference time instead of only relying on what it memorized during training, fresh information can be served without retraining. That is the whole value proposition of RAG. It is also why retrieval budgets matter: every retrieved chunk competes for the same finite window, a tradeoff the [context window explainer](https://protrailblazer.com/posts/context-window-explained/) demonstrates interactively. ## Where does RAG struggle? RAG is a retrieval problem wearing a generation hat. Most failures are retrieval failures in disguise: - **The answer requires synthesis across many chunks.** Top-K pulls chunks independently. If the real answer needs paragraphs from fifteen chunks scattered across the corpus, top-K equal to five will miss most of them. - **The query is vague.** A question like 'what do we know about X' will pull chunks that match the phrase, not chunks that contain the actual information. Query rewriting and hybrid lexical-plus-vector search help. - **The retrieved chunk is almost-right.** The model reads an adjacent paragraph to the one that would have answered, and extrapolates. This is the same [hallucination](https://protrailblazer.com/posts/hallucination-simulation/) failure mode, still inside the confident zone, but now with citations pointing at the wrong source. Upgrades most teams reach for first: chunk-boundary-aware splitting, re-ranking the top-K with a cross-encoder, hybrid search (vector plus BM25), and query expansion. Reach for those before you start tuning the [generator](https://protrailblazer.com/posts/how-llms-generate-text/). ## Frequently asked questions ### What is retrieval-augmented generation? Retrieval-augmented generation, or RAG, is an LLM that looks information up before it answers. The query is embedded, the most similar chunks from a document store are retrieved, and those chunks are placed in the context so the model answers from real source text. It delivers fresh, grounded facts without retraining the model. ### What does top-K control in RAG? Top-K is how many of the nearest chunks get handed to the model. Set it too low and you may miss the chunk that holds the answer; set it too high and the right chunk gets buried in irrelevant text, which can still produce a wrong answer. Most production systems land between 3 and 6 and add a re-ranking step. ### Does RAG eliminate hallucinations? No, it shrinks the hallucination zone rather than closing it. If retrieval surfaces the wrong or almost-right chunk, the model can still answer confidently and incorrectly, now with a citation pointing at the wrong source. Pairing RAG with visible citations lets readers check what the model actually read. ### Why does chunking matter so much in RAG? Retrieval works on chunks, so where you cut the documents decides what can be found. Slicing mid-sentence or mid-idea can hide the one passage that would have answered the question. Structure-aware splitting along headings, paragraphs, or sentences usually beats fixed-size slicing. ## Key takeaways - RAG is a retrieval problem wearing a generation hat. If retrieval surfaces the wrong chunks, the model's answer will be confidently wrong about the wrong documents. Tune retrieval before you tune prompts. - Top-K is the most important lever. Too low and you miss the chunk that would have answered. Too high and the model drowns in irrelevant text and hallucinates anyway. Most production systems land between 3 and 6. - Chunk boundaries matter more than chunk content. Fixed-size slicing can hide the one sentence that would have answered. Use structure-aware splitting. - RAG shrinks the hallucination zone, it does not close it. Pair RAG with citation rendering so readers can check what the model actually read. --- --- title: "Prompt engineering: how technique shapes what models say" description: "Zero-shot, few-shot, chain-of-thought, and role prompting explained with an interactive demo. See how technique shifts token probability and output quality." published: 2026-05-24 updated: 2026-05-24 canonical: https://protrailblazer.com/posts/prompt-engineering-guide/ tags: [machine-learning, demo, prompting, llm, chain-of-thought] --- # Prompt engineering: how technique shapes what models say **Answer:** Prompt engineering is the practice of changing how you phrase a request to a language model to get better output. The model's knowledge doesn't change, but technique adjusts what the model attends to and shifts the probability distribution over its next tokens toward more useful responses. The interactive below shows four prompting techniques applied to the same task. Switch techniques using the tabs in the left panel, choose a task from the dropdown, and click Run Prompt. The assembled prompt updates live so you can see exactly what changes between techniques. *Interactive demo: [open the prompt engineering demo](https://protrailblazer.com/demos/prompt-engineering/)* ## What you're seeing The left panel is the prompt builder. Switching technique tabs rewrites the assembled prompt in real time: zero-shot sends the raw task, few-shot adds example input-output pairs before it, chain-of-thought appends a step-by-step reasoning cue, and role prompt opens with an expert persona. The text that appears is exactly what would be sent to the model. The right panel shows simulated model output, streaming in character by character the way a real API response feels. The quality bar is a rough score for how well the technique fits the task. Click 'Compare all' to see a ranked breakdown of all four techniques side by side for the current task. At the bottom is the token probability histogram. It shows a simulated distribution over next-token candidates. Clearer techniques produce a taller, more peaked histogram. Drag the temperature slider toward 2.0 and watch the bars flatten. A few things to try in the demo: - **Switch from zero-shot to chain-of-thought on the same task.** Read what the prompt becomes and how the output changes. The quality difference is most visible on 'Debug code' and 'Explain a concept'. - **Drag the temperature slider to 1.8.** Watch the histogram flatten as the model becomes less certain which token comes next. Then run the same task and check whether the quality score drops. - **Click 'Compare all'.** The overlay ranks all four techniques for the current task. Haiku and sentiment have different top answers. - **Switch to 'Custom prompt'** and type your own task. The technique tabs still control how the assembled prompt is framed around your input. ## How it actually works When a language model generates a response, it picks each word one piece at a time from a ranked probability distribution. Each of those pieces is a token, typically a few characters to a short word. The full sequence of past tokens, including your prompt, conditions what comes next. That is why phrasing matters: different prompt structures activate different token-prediction patterns learned during training. The [how LLMs generate text](https://protrailblazer.com/posts/how-llms-generate-text/) post goes deeper on the token-by-token mechanics. Few-shot prompting works because the examples you include shift the conditional distribution over next tokens before the model starts generating your answer. The model doesn't learn from your examples the way fine-tuning does. It reads them as context, and that context narrows the likely range of outputs toward the format and style you demonstrated. Chain-of-thought is effective for a specific reason: the reasoning tokens that appear before the final answer condition that final answer. When the model writes 'Step 1: the bug is...' it is conditioning its own next output on what it just wrote. That self-scaffolding steers the final tokens toward more accurate responses on complex tasks, and it is why chain-of-thought dramatically improves accuracy on math, code, and multi-step reasoning while barely moving the needle on simple recall. Temperature is applied to the raw logits (the model's unnormalized scores for each candidate token) before softmax converts them to probabilities. Low temperature sharpens the distribution, making the top token dominant. High temperature flattens it, letting lower-ranked tokens compete. The histogram in the demo visualizes this directly. The [temperature explainer](https://protrailblazer.com/posts/llm-temperature-explained/) covers the full mechanics. Role prompting shifts vocabulary and reasoning style by activating associations the model built from domain-specific training text. 'You are a senior Python engineer' doesn't add new knowledge. It routes the response toward the language patterns, caveats, and structure that appeared in the training data for that type of expert. The model learned what engineers say; you're reminding it which part of that space you want to be in. ## When to use which technique Zero-shot works when the task is clear and the model has strong priors for it. Asking for a summary, a translation, or a simple classification tends to work fine without examples. It is the right starting point before you add complexity. Few-shot pays off when the output format needs to be consistent across many runs, or when the task involves a structure the model doesn't naturally produce. Add examples that show the exact format you want, not just that you want something 'like this'. Chain-of-thought earns its extra tokens when the final answer depends on getting intermediate steps right: multi-step math, code debugging, logical deduction. It is rarely worth the token cost on simple lookup or recall tasks. Role prompting helps when the response needs a specific register, vocabulary, or expertise level. It is less about changing what the model knows and more about which part of its knowledge to draw from. If the domain is specialized enough that word choice matters, a role framing is worth one or two sentences of setup. ## Key takeaways - Better prompting doesn't change the model's knowledge. It changes which part of what the model already knows shows up in the output. - Chain-of-thought improves accuracy because intermediate reasoning tokens condition the final answer tokens, not because the model becomes smarter mid-generation. - Temperature and technique are independent controls. Temperature governs how random the sampling is; technique governs what part of the distribution the model is sampling from. - There is no universal best technique. Chain-of-thought wins on complex reasoning, few-shot wins on format consistency, and role prompting wins when you need domain vocabulary. The 'Compare all' button in the demo shows this task by task. --- --- title: "Model Context Protocol: how AI apps plug into tools and data" description: "Watch JSON-RPC messages flow between an MCP host and its servers as the client discovers tools, invokes them, and streams results back to the model." published: 2026-05-21 updated: 2026-06-14 canonical: https://protrailblazer.com/posts/model-context-protocol-explained/ tags: [mcp, ai-agents, tool-use, protocol, demo] --- # Model Context Protocol: how AI apps plug into tools and data **Answer:** Model Context Protocol is a small open standard that lets an AI app ask a set of tiny programs called MCP servers what they can do, then call those tools during a conversation. It turns a model that only knows words into an app that can read your files, query your database, and post to Slack without a custom integration per service. The interactive below shows a Model Context Protocol host talking to six local servers on an isometric pixel grid. Auto-loop fires realistic prompts on a timer. Toggle any server off in the lower-right panel and watch the animation route around it. Drag empty space to rotate, scroll or pinch to zoom, and tap any cube to see the tools it exposes. *Interactive demo: [open the mcp protocol demo](https://protrailblazer.com/demos/mcp-protocol/)* ## What does the demo show? The red cube in the middle is the MCP host, the process that runs the AI model and speaks the protocol. The six cubes around it are MCP servers. Each server is a small program that advertises a handful of tools. The gray cube off to the side is you, the user, sending a prompt into the host. The tiny red cubes hopping along the grid are JSON-RPC messages. When the user (or the auto-loop) fires a prompt, the host decides which servers can answer, dispatches a `tools/call` to each, waits for a result, then replies back up to the user. The host flashes on every message it handles. Each server flashes when one of its tools runs. A few things to try in the demo: - **Toggle servers off.** Disable Slack or Weather in the lower-right panel. Prompts that needed them now return a no-servers response from the host, and the dotted edge to that server dims out. - **Fire a compound prompt.** Click "Summarize PRs to Slack" or "Free time today?" The host dispatches to two servers in parallel and only replies to the user after both have answered. - **Rotate and zoom.** Drag the grid to spin the view, scroll to zoom, or pinch on a phone. The cubes stay legible at any angle so you can watch messages from a different side. - **Crank the fire rate.** Drop it to under a second and multiple tool calls overlap in flight. That is what a busy agent session actually looks like. ## How does MCP work? Every MCP connection follows the same lifecycle. On startup the host opens a stream to each server and sends an `initialize` request. The server answers with its protocol version and a capability list. The host then calls `tools/list`, `resources/list`, and `prompts/list` to learn what that server actually exposes. At inference time the host builds a combined tool catalog from every connected server and hands it to the model. When the model decides to call a tool, the host routes a `tools/call` JSON-RPC request to the correct server, awaits the response, and feeds the result back into the next turn. If a server is toggled off, it vanishes from the catalog and the model cannot use it at all that turn. The demo reflects this: disabled servers never receive a request. Transport is either **stdio** (the server runs as a child process of the host and they exchange JSON-RPC messages over pipes) or **HTTP with server-sent events** (the server runs out of process, often remote). The wire format is JSON-RPC 2.0 in both cases, so the host needs no per-server client code. That is the whole point: add a new server, and no client changes. Servers are isolated processes on purpose. The host runs the model and holds any API keys. A server only sees the arguments it was handed and returns a result. A crashed or malicious server can fail its own call, but it cannot reach into the host's process memory or read the user's prompt history. That separation is the protocol's security story. Latency is the other design constraint. Every enabled server is a potential round trip per turn, and each tool call waits on the real external system behind it (a database, an API, a disk read). Production agent workflows usually whitelist a small default server set and enable the heavy ones only for the tasks that need them. MCP is the connective tissue behind most tool-using [agent workflows](https://protrailblazer.com/posts/agentic-workflow-smb-guide/): the model works out what it needs as it [generates its response](https://protrailblazer.com/posts/how-llms-generate-text/), and the protocol gives it a uniform way to reach the tools that can deliver. ## Frequently asked questions ### What is the Model Context Protocol? The Model Context Protocol (MCP) is a small open standard that lets an AI app discover and call external tools at runtime. The app, called the host, asks each connected MCP server what it can do, then invokes those tools during a conversation over JSON-RPC. It gives a model a uniform way to read files, query databases, or call APIs without a custom integration per service. ### What problem does MCP solve? Before MCP, every AI app needed bespoke glue code for each tool or data source it wanted to use. MCP replaces that with one protocol: write a server once, and any MCP-capable host can use it with no client changes. It standardizes the connection between models and the outside world the way a common port standardizes hardware. ### What are MCP tools, resources, and prompts? They are the three primitives a server can expose. Tools are actions the model can take, such as sending a message or running a query. Resources are content the model can read, such as files or records. Prompts are templated workflows the server offers. The host discovers all three with list calls when it connects. ### Is MCP secure? The protocol's security comes from process isolation. The host runs the model and holds any API keys, while each server is a separate process that only sees the arguments it was passed and returns a result. A crashed or malicious server can fail its own call but cannot reach into the host's memory or read the user's prompt history. You still vet which servers you enable. ## Key takeaways - MCP is a discovery protocol, not a hardcoded API. The host asks each server what it offers at runtime, which is why a new server plugs in without any client code changes. - The host runs the model, servers run the integrations. They talk over JSON-RPC on stdio or HTTP, so a failing or malicious server cannot reach into the host's process. - Tools, resources, and prompts are the three primitives. Tools are actions the model can take, resources are content it can read, prompts are templated workflows it can invoke. - Every enabled server adds a potential round trip per turn. Good agent workflows keep the active set small and enable the expensive servers only when the task actually needs them. --- --- title: "LLM hallucination: why confident answers drift from truth" description: "A 3D demo of language-model sampling: tokens drift outward as text moves from grounded to fabricated, while the model's confidence barely moves." published: 2026-05-17 updated: 2026-05-17 canonical: https://protrailblazer.com/posts/hallucination-simulation/ tags: [llm hallucination, hallucination, language model, confidence, accuracy, sampling, temperature, grounding, rag, machine learning] --- # LLM hallucination: why confident answers drift from truth **Answer:** Hallucination is what happens when a language model keeps sampling confidently past the edge of what it actually knows. Each next token is pulled from a probability distribution, and the model's confidence in that pick is not tied to whether it is right, so wrong text can come out sounding as certain as right text. The interactive below shows a language model's [sampling process](https://protrailblazer.com/posts/how-llms-generate-text/) as a 3D particle cloud. Drag to orbit through it, pinch or scroll to zoom, and watch how the field shifts when you change the temperature or the grounding slider. Every red particle is a token the model could sample next. The further it drifts from the white anchor, the further the generated text has wandered from the truth. *Interactive demo: [open the hallucination demo](https://protrailblazer.com/demos/hallucination/)* ## What you're seeing The white point at the center is the prompt. It is the one thing the model knows for sure. Everything else in the field is guessed. Each red particle is a candidate token the model could pick next. Radial distance from the anchor is how far that token has drifted from the ground truth. Brightness is the model's reported confidence in its pick. Three faint wireframe shells carve that space into zones: - **Truth.** Inside here, the token is close to what a grounded answer would say. - **Plausible.** Past this shell the token is still coherent but no longer directly supported. - **Hallucination.** Past this shell the model is inventing. Particles here are a little larger and a little brighter because the model did not lose confidence when it crossed the line. A few things to try in the demo: - **Push the temperature up.** The cloud inflates. More tokens cross the hallucination shell, and the HALLUC readout in the top-right climbs. - **Pull the grounding slider toward one.** A soft force drags the cloud back toward the anchor. This is the effect of retrieval, citations, or a tight system prompt. Accuracy goes up. Confidence barely moves. - **Watch the strip chart.** It plots running confidence (solid red) against accuracy (dashed). The shaded band between them IS the hallucination zone. When it opens, the model is confidently wrong. - **Orbit around the cloud.** The drift is spherical, not a 2D disc. Some directions get denser than others as the random walk finds local channels, the same way certain phrasings tend to pull a model into certain fabrications. ## How it actually works A language model picks each next token by turning its internal state into a set of logits, passing those through a softmax to get a probability distribution over every token in its vocabulary, and sampling one. The distribution reflects how well each token fits the current context. It does not know whether any of those tokens are factually correct. Plausible and correct are different constraints, and the model only optimizes the first one. Temperature scales the logits before softmax. Low temperature makes the distribution spiky and concentrates the sampler on the top candidate. Greedy decoding is the limit case, temperature zero, which always picks the top token. High temperature flattens the distribution and widens the range of likely picks, which is what inflates the cloud in the demo. Neither setting changes whether the top candidate is actually true, so hallucinations happen across the whole range. Other sampling constraints like nucleus sampling cap how far into the tail the sampler can reach. They reduce the chance of a wildly off-track token on any single step, but they do nothing about a confidently wrong token that is still inside the nucleus. The reason drift shows up as a gradient in the demo and not a single jump is that each sampled token is appended to the context the next step conditions on. One wrong pick shifts the distribution the model sees at the following step, which shifts the one after that. By the time a generation is fifty tokens long, a small early drift has re-anchored the rest of the answer around a fabrication, and every subsequent token is confidently consistent with the wrong premise. ## What fights hallucination in practice Production systems fight this in four places. Retrieval injects authoritative text into the context window before generation, so the grounded tokens are already in the conditional distribution. Tool use forces the model to look a fact up rather than sample one. Self-consistency samples multiple answers and votes, which smooths out single-path drifts. Post-hoc verification checks the output after the fact. Each of these is a version of the grounding slider in the demo: shrink the space the sampler is allowed to walk through. The same controls can also be tuned at the sampler itself. See our walkthroughs on [temperature](https://protrailblazer.com/posts/llm-temperature-explained/) and on [top-k and top-p sampling](https://protrailblazer.com/posts/top-k-top-p-sampling-explained/) for a closer look at the specific knobs this post introduces. ## Key takeaways - Confidence and accuracy are different signals. A model's internal probability for a token measures how well the token fits the local language pattern, not whether it is factually true. - Temperature does not cause hallucination, it spreads it out. At temperature zero the model still hallucinates, it just commits to one confident wrong answer instead of sampling many variants. - Grounding (retrieval, tool use, citations) shrinks the hallucination zone by pulling the distribution back toward verified tokens, but it does not remove the mechanism that produces drift in the first place. - Drift compounds across tokens. Each sampled token joins the context that conditions the next one, so a single fabrication early in a sequence can re-anchor the rest of the answer around it. --- --- title: "Agentic workflows: how AI agents chain business tasks" description: "A radial simulation of an agentic AI workflow, plus a plain-language look at what agents do, why the orchestrator matters, and what breaks first for SMBs." published: 2026-05-12 updated: 2026-07-07 canonical: https://protrailblazer.com/posts/agentic-workflow-smb-guide/ tags: [ai, agents, workflow, automation, smb] --- # Agentic workflows: how AI agents chain business tasks **Answer:** An agentic workflow is an AI system that splits one job into steps, assigns each step to a specialized agent, and lets a central orchestrator route the work between them. For a small business it is the difference between an AI that writes a single reply and an AI that handles the whole quote, invoice, and follow-up without a human pasting things between tabs. The interactive below shows an agentic workflow in motion. Tasks arrive on the outer ring, curve through a central orchestrator, and visit each specialist agent in sequence. Switch between three preset workflows (quote to cash, support, onboarding), push the arrival rate up, and crank the failure rate to see where the pipeline starts to bleed. Click the fullscreen button for a closer look. *Interactive demo: [open the agentic workflow demo](https://protrailblazer.com/demos/agentic-workflow/)* ## What you're seeing The bright red node in the middle is the orchestrator. Its only job is to know which agent should get the task next. The ring of panels around it are the specialist agents: INTAKE validates what came in, RESEARCH pulls context, PRICE calls a calculator or database, DRAFT writes the customer-facing content, VERIFY checks the draft, SEND pushes it out, and FOLLOW-UP nudges later if needed. A few things to try: - **Switch the workflow preset.** Quote to Cash touches every agent. Support skips PRICE and FOLLOW-UP. Onboarding moves VERIFY earlier in the pipeline. The agents don't change, the routing does. - **Push arrival rate to 4 or 5 tasks per second.** Watch the LOAD counter on DRAFT climb faster than the others. The slowest agent becomes the queue, every time. - **Set failure rate to 20 percent.** Some tasks flash white and loop back for a retry. After three attempts they get dropped with an X. The SUCCESS percentage and the dashed line in the strip tell you what your customers would actually experience. - **Hit Burst +20.** A spike of twenty tasks hits the intake ring at once. The router handles them fine. The downstream agents queue up, which is the real cost of a traffic spike. ## How it actually works An agent is not a smarter model. It is a regular [LLM call](https://protrailblazer.com/posts/how-llms-generate-text/) wrapped in three things: a scoped job ("you only draft quotes, nothing else"), a set of tools it's allowed to call (a pricing API, a CRM lookup, an email sender), and a clear output contract ("return a JSON object with these fields"). The tool wiring increasingly runs over a standard interface, the [Model Context Protocol](https://protrailblazer.com/posts/model-context-protocol-explained/), so agents can share the same connectors instead of each one hand-rolling its integrations. The orchestrator is the boring part that makes the system work. It holds the pipeline definition ("first INTAKE, then RESEARCH, then PRICE"), passes outputs from one agent to the next, handles retries when an agent returns an error or a malformed result, and decides when to drop a task that keeps failing. It is usually a few hundred lines of code, not a separate model. Two details in the simulation that map directly to real systems: 1. **Tasks always route through the center.** Agents don't talk directly to each other. This is a deliberate choice. If every agent can call every other agent, you end up with a network of hidden dependencies and no single place to change behavior. Central routing is worse at raw speed and better at literally everything else. The same pattern shows up at the model level in [mixture-of-experts architectures](https://protrailblazer.com/posts/mixture-of-experts-explained/), where a gating network routes each token to the right expert. 2. **Each agent has a processing time and a load bar.** DRAFT is the slowest because long-form generation takes the most tokens. PRICE is fast because it is really just a case of tool use. The load bar fills up as tasks queue. That bar is the single most useful operational metric in a production agentic system. ## Where this breaks for a small business The simulation is clean. Real SMB workflows have three kinds of mess that don't show up on screen. First, the inputs are never uniform. A "new quote request" can arrive as an email, a web form, a text message, or a phone call someone transcribed badly. INTAKE in the real world spends most of its effort deciding what the task even is, not validating it. Second, the tools lie. A pricing API can return stale data, a CRM can be missing the contact, the email sender can silently drop a message. The orchestrator has to treat every tool call as potentially wrong and plan for what to do when it is. That's where most of the real engineering goes. Third, humans interrupt. A customer replies mid-pipeline with new information, a staff member overrides the draft, a rule changes on the last day of the quarter. A good agentic system needs a "pause, edit, resume" path, which means every task needs persistent state, not just an in-memory handoff. If you want the same ideas at solo scale, the [freelancer AI workflow tour](https://protrailblazer.com/posts/freelancer-ai-workflow/) maps 36 real AI assists across six phases of a web project, including which handoffs can run without a human approver. ## Key takeaways - Agents are not smarter LLMs, they are LLMs plus a scoped job and a set of tools. The intelligence is in how they are wired together, not in any one call. - The orchestrator is where the business logic lives. If every agent can talk to every other agent, you do not have a workflow, you have a chat room. - Failure handling is the product. A demo that never fails hides the entire engineering problem. - Throughput is bounded by the slowest agent. Queue depth tells you where to spend money first. - Start with a preset pipeline, not a free-form agent. Most SMB workflows are four to seven fixed steps with light branching, and a DAG beats a debate every time. --- --- title: "Top-k and top-p sampling: how an LLM picks its next token" description: "A plain-English look at the two cutoff strategies that decide which tokens a language model is allowed to sample from, with a live interactive demo." published: 2026-05-09 updated: 2026-06-14 canonical: https://protrailblazer.com/posts/top-k-top-p-sampling-explained/ tags: [machine-learning, demo, inference, llm] --- # Top-k and top-p sampling: how an LLM picks its next token **Answer:** Top-k and top-p (nucleus) sampling are the two knobs that decide which tokens a language model is even allowed to pick from before it rolls the dice. Top-k keeps a fixed number of candidates. Top-p keeps whichever candidates, in descending probability order, add up to a target share of the mass. Everything below the cutoff is discarded, and the model samples from what is left after renormalization. The interactive below shows both strategies side by side. Flip between top-k, top-p, and greedy, drag the cutoff slider, and switch between three preset prompts to see how the distribution's shape changes everything. The glowing columns above the bars show which tokens are actually getting drawn, accumulating in real time. *Interactive demo: [open the sampling strategies demo](https://protrailblazer.com/demos/sampling-strategies/)* ## What does the demo show? The bars along the bottom are a language model's next-token probability distribution for a given prompt, sorted from most to least likely. Each bar is one candidate token. The tall ones on the left are the model's top guesses. The long flat tail on the right is everything it considers barely plausible. The vertical red line is the **cutoff**. Bars to the left of it are the eligible set, the tokens that are actually allowed to be sampled. Bars to the right are dimmed, meaning the model cannot pick them no matter how the dice fall. The dashed line climbing over the bars is the cumulative probability (CDF), which is what top-p is looking at when it decides where to cut. Above the histogram, every sample drawn drops a small red dot in a vertical column above the token it landed on. Over thousands of samples, the most-drawn tokens grow bright columns and the rarely-drawn ones stay dark. That column pattern is the actual output distribution of the sampler. A few things to try in the demo: - **Pick the peaked prompt ("The capital of France is") and flip to greedy.** Only one column grows. That is what deterministic decoding looks like. - **Switch to the flat prompt ("Once upon a") and drag K from 40 down to 5.** Watch how many plausible tokens you just disqualified. A rank cutoff feels brutal on a flat distribution. - **Switch to top-p mode on the same flat prompt and drag P.** The eligible set tracks the shape of the distribution instead of the position of the slider. That is the whole point. - **Hit reset and let N climb past 20,000.** The column heights stabilize into the true renormalized probabilities of the eligible set. That is what the model's output distribution looks like after sampling. ## How do top-k and top-p sampling work? A language model's final layer emits one raw score (a logit) per token in its vocabulary, which softmax turns into a probability distribution over the whole vocab. This happens at every step, because the model [writes one token at a time](https://protrailblazer.com/posts/how-llms-generate-text/). Typical vocab sizes are in the tens of thousands. If the model just sampled directly from that raw distribution, it would occasionally pick genuinely terrible tokens from the long tail, and the output would feel unhinged. Top-k and top-p exist to cut the tail off. Top-k sorts the tokens by probability, keeps the top K, zeroes out everything else, and renormalizes the kept probabilities so they sum to 1. The model then samples from that renormalized distribution. K is a fixed number, usually something like 40 or 50 in practice. Top-p (also called nucleus sampling) works in descending probability order: it adds tokens to the eligible set one at a time until the running total of their probabilities crosses a threshold P, then stops. Everything past that point is discarded, the kept set is renormalized, and the model samples. P is usually 0.9 or 0.95. The difference matters when the distribution's shape changes. On a peaked distribution where one token has most of the mass, top-p will keep almost nothing past that one token. On a flat distribution where probability is spread across many tokens, top-p will keep many of them. Top-k keeps the same number either way, which is its core weakness. Greedy decoding is the degenerate case: always pick the argmax, sample nothing. It is deterministic and occasionally useful (code generation, structured output), but on longer text it tends to fall into repetitive loops because small probability differences compound over many steps. ## What is min-p sampling? **Min-p** is a newer cutoff that anchors to the most likely token instead of to rank (top-k) or cumulative mass (top-p). You set a fraction, commonly 0.05 to 0.1, and the model keeps every token whose probability is at least that fraction of the top token's probability. If the leading token sits at 0.50 and min-p is 0.1, the bar is 0.05, so any token above 5 percent survives. If the model is confident and the top token is at 0.90, the bar rises to 0.09 and the eligible set tightens on its own. The reason people reach for it: min-p holds up well at high temperature. Because the threshold scales with the peak, you can raise temperature to loosen the writing without letting in tokens that have almost no support relative to the front-runner, which is the failure mode that makes high-temperature top-p feel incoherent. Min-p was introduced in 2024 and now ships in llama.cpp, vLLM, and Hugging Face Transformers as the `min_p` parameter, so you can try it in most local stacks today. ## How do temperature, top-k, and top-p interact? Temperature scales the logits before the softmax, which flattens or sharpens the distribution. Order matters. Apply top-p after temperature scaling and the same p value produces a very different eligible set: tight at low temperature, loose at high. Most production stacks apply temperature first, then top-k or top-p. A few apply them in the other order and get noticeably different behavior. The cutoffs can also combine. Some stacks do top-k first to cap the size of the eligible set, then top-p inside that, then sample. The exact pipeline varies by framework, which is one reason two implementations of "the same" sampler can produce different text for the same prompt and seed. For a deeper look at how that scaling reshapes the distribution, see [how temperature shapes a model's output](https://protrailblazer.com/posts/llm-temperature-explained/). ## How do you choose the right values? There is no universal "correct" setting. The right cutoff depends on what you want the output to be. - **Code and structured output:** use greedy or a very low temperature with a small k (1 to 5) or a low p (0.1 to 0.3). You want the best single guess, not creativity. - **Conversational or factual answers:** temperature around 0.7 and top-p around 0.9 is a common default. Enough variance to sound natural without letting obviously wrong tokens slip in. - **Creative writing or brainstorming:** higher temperature (1.0 to 1.3) with top-p at 0.92 to 0.95 opens the tail without descending into nonsense. A common mistake is treating these values as universal. A top-p of 0.9 that works for storytelling will not be what you want for generating SQL. Tune per task, and test at a few different settings before committing to a default. To go deeper on the upstream model mechanics, the [gradient descent demo](https://protrailblazer.com/posts/gradient-descent-demo/) shows how models get trained in the first place. For the full vocabulary of inference terms, see the [glossary](https://protrailblazer.com/terms/). ## Frequently asked questions ### Should I use top-k or top-p? For most text generation, top-p is the better default because its cutoff adapts to how confident the model is at each step, while top-k keeps a fixed number of candidates no matter the shape of the distribution. Reach for top-k or greedy when you want hard, predictable limits, such as code or structured output. Many stacks let you set both and apply them in sequence. ### What are good default values for top-p and temperature? A common starting point for conversational or factual output is a temperature around 0.7 with top-p around 0.9. For creative writing, raise temperature to roughly 1.0 to 1.3 and top-p to 0.92 to 0.95. For code or anything that must be exact, drop to greedy or a low temperature with a small top-p such as 0.1 to 0.3. Treat these as starting points and tune per task. ### Can you use top-k and top-p at the same time? Yes. A common pipeline applies top-k first to cap the size of the candidate set, then applies top-p inside that set, then samples. The order and exact behavior vary by framework, which is one reason two implementations of the same settings can produce different text for the same prompt and seed. ### Is min-p better than top-p? Not universally. Min-p tends to hold coherence better at high temperatures because its cutoff scales with the most likely token, so it is popular for creative generation in local model stacks. For everyday use at a moderate temperature, top-p around 0.9 remains a solid and widely supported default. The best choice depends on your temperature and your task, so test both. ### Does a top-p of 1.0 turn off the cutoff? Effectively yes. At a top-p of 1.0 the eligible set includes the entire probability mass, so no tokens are filtered and the model samples from the full distribution after temperature scaling. The same goes for a very large top-k: once k reaches the vocabulary size, it stops removing anything. ## Key takeaways - Top-k is shape-blind. The same k of 40 behaves almost-greedy on a peaked distribution and almost-random on a flat one, because a rank cutoff ignores what the probabilities actually look like. - Top-p is shape-aware. The eligible set grows when the model is unsure and shrinks when it is confident, because the cutoff is based on cumulative probability mass, not rank position. - Min-p is a third option that sets the cutoff relative to the top token's probability. It tends to stay coherent at high temperature, which is why it shows up in local model stacks aimed at creative writing. - The cutoff only decides which tokens are on the ballot. The actual next token is still sampled at random from the renormalized probabilities inside the eligible set, not picked by a rule. - Temperature reshapes the distribution before the cutoff is applied. Top-p at a low temperature produces a much tighter eligible set than top-p at a high temperature, even with the same p value. --- --- title: "Mixture of Experts: sparse routing for huge models at fast-model cost" description: "A plain-English explainer on MoE models with a live 3D demo showing how tokens route through a small subset of experts at each layer." published: 2026-05-07 updated: 2026-07-07 canonical: https://protrailblazer.com/posts/mixture-of-experts-explained/ tags: [machine-learning, demo, llm, architecture] --- # Mixture of Experts: sparse routing for huge models at fast-model cost **Answer:** A Mixture of Experts (MoE) model has many copies of the same sub-network in each layer, but every token only passes through a small fraction of them. A tiny gating network picks two or three experts per token per layer, and the rest sit idle. That is how a 141-billion-parameter model can run at roughly the cost of a 14-billion one. The interactive below shows a Mixture of Experts architecture routing tokens layer by layer. Switch modes to see a single token move slowly (SINGLE), a full prompt stagger through (SEQUENCE), or every token fire at once the way production inference really works (ALL AT ONCE). Try the three prompts and different expert counts to see how the lit-up path changes. *Interactive demo: [open the moe routing demo](https://protrailblazer.com/demos/moe-routing/)* ## What you're seeing Six layers of a transformer stack, rendered vertically. Each layer is a ring of cylindrical towers, one tower per expert. With the default 8 experts per layer that is 48 towers in the scene. The real model it represents would have one attention block above each layer too, but the sim focuses only on the MoE feed-forward block, which is where the routing happens. A glowing red token particle rises through the center of the stack. When it reaches a layer, the router plane pulses, beams fan out from the token to every expert at brightness proportional to the gate score, and then the top-k beams brighten sharply while the rest fade. The chosen experts flash, then settle into a dim glow over two seconds as they hand off to the next layer. Thin lines left behind are the persistent trail, giving you a running picture of every expert the model has used so far. Four stats in the top right: TOTAL_P is every parameter in every expert summed together (the number that gets put on model cards), ACTIVE_P is what the current token actually touches (a small fraction of the total), TOKEN is the current token being routed, and EXPERTS_HIT is the cumulative count of unique layer-expert pairs activated across the sequence so far. Watch ACTIVE_P stay small when TOTAL_P balloons. A few things to try in the demo: - **Switch to the code prompt ("def fibonacci(n):").** The same two towers light up at each layer, again and again. That is (simulated) expert specialization: in this sim code tokens have learned to route to the "code experts" consistently. - **Switch to the translate prompt.** English and French tokens light different expert groups as they alternate through the sequence. Punctuation routes to its own corner. - **Crank experts per layer to 64.** The stack gets much wider but the lit-up path stays just as thin. TOTAL_P jumps dramatically. ACTIVE_P does not move. That is the MoE value proposition in one visual. - **Drop top-k to 1.** Routing collapses toward deterministic (every token picks exactly one expert per layer). Raise it to 4 and coverage explodes: EXPERTS_HIT rises fast and the benefit of sparsity shrinks. - **Hit the TOP camera preset in ALL AT ONCE mode.** You get a top-down view of one layer, with every token in the prompt lighting its chosen experts simultaneously. That is what a single step of real MoE inference looks like. ## How it actually works In a dense transformer, every token at every layer runs through the same feed-forward network. In an MoE transformer, that single feed-forward block is replaced by a bank of N smaller feed-forward blocks (the experts), plus a tiny neural network called the gating network or router. The router takes the token's representation, produces a score for each expert, picks the top-k (typically 2 out of 8 or 64), and sends the token only through those. The outputs of the chosen experts are combined, weighted by their gate scores, and that becomes the layer output. "Mixture of experts" is a misleading name. There is no pre-training where expert 3 learns French and expert 7 learns Python. Every expert starts identical. Specialization emerges during training because the router learns to route similar tokens to similar experts, and similar training data refines those experts further, and the whole thing reinforces itself. The result is that any specialization you see is emergent and often illegible. A real MoE does not have a "code expert," it has experts that sometimes process code tokens in subtle patterns that defy neat interpretation. The sim shows crisp specialization on purpose, because crisp specialization is the teaching point. Real routing is noisier. Do not mistake the visualization for the model's internal structure. Why does this architecture exist? Three reasons, in order of importance: 1. **Capacity at constant compute.** A dense model with 100B parameters costs 100B-worth of compute per token. An MoE model with 100B total parameters but 10B active runs at 10B-worth. You get the expressive capacity of the big model with the throughput of a smaller one. 2. **Specialization (emergent).** Different tokens can exercise different parameters. Whether or not specialization is "clean," the model has more slots to put distinct patterns, so empirically MoE models often match dense models that are several times larger in total params. 3. **Training parallelism.** Experts can live on different GPUs ("expert parallelism"). This is a major reason MoE is practical at frontier scale. Routing becomes a network communication problem. ## Why total parameters can mislead When a model card says "141B parameters," the implied comparison is usually to dense models of similar total size. But for MoE, total parameters and inference cost are decoupled. Mixtral 8x22B has about 141B total params and around 39B active. Its inference cost is closer to a 39B dense model than a 141B one. The important metric is active parameters per token, and most MoE announcements bury that number. There is one place total parameters do matter: GPU memory. Even inactive experts need to sit on a device somewhere, because any token might need them next. That means MoE models demand more VRAM than their active-parameter cost implies. A 141B-total MoE model loaded on a single GPU needs enough memory for all 141B parameters, not just the 39B it will use. This is why MoE deployments usually span many GPUs and use expert parallelism to spread the memory cost. The other invisible cost is load balancing. If the router gets lazy and sends most tokens to the same three experts, training destabilizes: those three get overtrained while the rest stall. Every real MoE training recipe includes auxiliary losses that penalize imbalanced routing, plus a "capacity factor" cap that forces the router to spread tokens even when it would rather not. None of this is visible in the finished model. All of it matters to whether the finished model exists at all. For more on the layer architecture MoE replaces, see transformer layers. For related inference mechanics, the [top-k and top-p sampling demo](https://protrailblazer.com/posts/top-k-top-p-sampling-explained/) covers how the output side of an LLM works, and the [autoregressive generation explainer](https://protrailblazer.com/posts/how-llms-generate-text/) shows the token-by-token loop every one of those routed tokens passes through. The full [glossary](https://protrailblazer.com/terms/) has the rest of the vocabulary. ## Key takeaways - "Mixture of experts" is misleading. Experts are not specialists trained on different domains. They are identical-sized sub-networks that the gating network learns to route to, and any resulting specialization is emergent, not designed. - The total parameter count in an MoE model is mostly a marketing number. The figure that matters for inference cost is active parameters per token, which is a small fraction of the total. - Load balancing is harder than routing. Real MoE systems use auxiliary losses and capacity factors to keep expert usage roughly uniform, or training falls over. - Sparse activation buys capacity at constant compute but not at constant memory. All expert parameters still need to live somewhere. MoE models are memory-hungry even when they are fast. --- --- title: "LLM temperature: one number that reshapes the next-token distribution" description: "A plain-English look at temperature in language model inference, with a live demo that shows exactly what the parameter does to the probability distribution." published: 2026-05-03 updated: 2026-07-07 canonical: https://protrailblazer.com/posts/llm-temperature-explained/ tags: [machine-learning, demo, inference, llm] --- # LLM temperature: one number that reshapes the next-token distribution **Answer:** Temperature is a single number that reshapes an LLM's next-token probability distribution before anything else happens. T=1.0 leaves the distribution untouched. T less than 1 sharpens it toward the top token. T greater than 1 flattens it toward uniform. The logits never change. Only their softmax does. The interactive below shows temperature at work. Drag the slider between 0.1 and 2.0, click the preset buttons to see smooth transitions, and try all three prompts. The key thing to watch: the left chart never moves when you change temperature. Only the right chart morphs. *Interactive demo: [open the temperature scaling demo](https://protrailblazer.com/demos/temperature-scaling/)* ## What you're seeing Two side-by-side bar charts, 40 tokens each. The left chart is the raw logits coming out of the model's final layer. Bars rise above a zero line when the logit is positive and dip below when it is negative. These values are the model's opinion before any normalization. The right chart is the softmax of those logits, divided by the current temperature value. Bars always start from the baseline since probabilities are non-negative. They sum to 1 across the whole chart. As you drag the slider, this chart deforms in real time while the left chart stays still. The ghost trails you see on the right are the shape of the distribution a moment ago, fading as the morphing settles. Four live stats in the top-right corner: T (current temperature), ENTROPY (bits of uncertainty in the distribution), TOP_P (probability of the highest-ranked token, how confident the model is), and EFF_K (2 to the power of entropy, the number of tokens that are meaningfully in play). Watch all four move together as you drag. A few things to try in the demo: - **Pick the math prompt ("Two plus two equals") and drag T to 2.0.** The bar for "four" shrinks a little but still dominates. Temperature cannot erase a confident answer. - **Pick the color prompt ("My favorite color is") and drag T through the whole range.** The distribution redistributes dramatically. This is where temperature matters most: when the model is genuinely uncertain. - **Pick the weather prompt and click CHAOS (T=2.0).** Everything flattens into near-uniform noise. When the distribution is already flat, high temperature destroys it. - **Toggle LIN to LOG on the y-axis at T=0.1.** On linear scale the top bar dominates and the tail looks flat. Log scale reveals the actual structure in the tail, where the interesting stuff happens at low temperature. ## How it actually works A language model's final layer produces a vector of logits, one per token in the vocabulary. To turn these raw scores into probabilities the model applies softmax: exponentiate every logit, then divide each by the sum. Temperature modifies this one step by dividing every logit by T before the exponential. That is the entire math. Why does dividing by T change the shape? Because exp amplifies differences. Dividing the logits by a small T (say 0.2) before exp magnifies the gap between the largest logit and the rest, so one token ends up with most of the mass. Dividing by a large T (say 2.0) compresses the gap, so all the exponentials come out closer to each other and the mass spreads. T=1 is the neutral case. At T=0 the math breaks (division by zero). In practice every framework clamps the minimum to a tiny positive number, which is equivalent to greedy decoding: always pick the argmax. At the other extreme, T approaching infinity makes every probability equal, which is sampling uniformly at random from the full vocabulary. Temperature is applied before top-k and top-p in the pipeline, not after. This order matters. Top-p at T=0.5 sees a much sharper distribution than top-p at T=2.0, so the same P value produces very different eligible sets depending on what temperature did first. One subtle point the demo makes visible: temperature never changes the ranking of tokens. The most probable token at T=0.1 is the same as at T=2.0. Only the gaps between them change. A tied logit stays tied at every temperature, and a nearly-tied pair stays nearly-tied. This is why temperature is a blunt instrument: it cannot invent confidence the model does not have. ## Picking a temperature The right value depends on the task, not on the model. - **Deterministic or structured output:** T=0 (greedy) or very low T (0.1 to 0.3). Code generation, JSON output, structured extraction. Anything where you want the best single guess and no creativity. - **Factual Q&A:** T around 0.3 to 0.7. Enough variance for natural phrasing without drifting into made-up answers. - **Conversational defaults:** T around 0.7 to 1.0. The typical default you see in chat interfaces. - **Creative writing or brainstorming:** T around 1.0 to 1.3. Pair with top-p around 0.92 to 0.95 so the tail does not pull in genuinely broken tokens. A common antipattern is cranking temperature to 1.5 or 2.0 to make the output feel more interesting, then being surprised when the model starts producing nonsense. The bottom of the distribution is the bottom for a reason. High temperature pulls those tokens into play. For the full picture on sampling after temperature is applied, see the [top-k and top-p demo](https://protrailblazer.com/posts/top-k-top-p-sampling-explained/). For the step before any of this, the [autoregressive generation explainer](https://protrailblazer.com/posts/how-llms-generate-text/) shows where the next-token distribution comes from in the first place, and the [prompt engineering guide](https://protrailblazer.com/posts/prompt-engineering-guide/) covers the other big lever you control: the prompt itself. For the vocabulary around inference parameters, see the [glossary](https://protrailblazer.com/terms/). ## Key takeaways - Temperature happens before top-k and top-p, not after. If you chain them, the order matters, because top-p at T=2.0 sees a much broader distribution than top-p at T=0.5. - T=0 is a mathematical singularity (division by zero). Production stacks clamp to a tiny positive number which behaves the same as greedy decoding: always pick the argmax. - Temperature cannot create confidence where there is none. If two logits are nearly equal, no amount of low T makes one clearly win, because the ratio between them is preserved. - Effective k (2 to the power of entropy) is a better intuition for "how spread out is this distribution" than the top token probability alone. --- --- title: "LLM as judge: one model grading another" description: "One LLM grades another: a live demo of pairwise, rubric, and best-of-N judging, plus the position, length, and self-preference biases you have to probe for." published: 2026-04-30 updated: 2026-04-30 canonical: https://protrailblazer.com/posts/llm-as-judge-explained/ tags: [llm as judge, llm evaluation, model evaluation, pairwise comparison, rubric scoring, best-of-n, judge bias, position bias, length bias, machine learning] --- # LLM as judge: one model grading another **Answer:** LLM as judge is using one language model to grade the outputs of another, instead of asking a human. It scales evaluation cheaply, but the judge inherits its own biases, so if you do not probe for them they can quietly warp every result downstream. The interactive below puts one LLM in the judge slot and feeds it candidate answers from another. Switch between judging modes, run a bias probe, and watch the win rates drift as the judge's blind spots show up. *Interactive demo: [open the llm as judge demo](https://protrailblazer.com/demos/llm-as-judge/)* ## What you're seeing Two candidate answers enter on the left. A single judge node in the middle reads them. A verdict exits on the right. The stat readout in the top-right tracks how often each candidate is picked across repeated trials, and BIAS reports how far that ratio has drifted from a fair 50-50 split. The controls on top switch the judge's job: - **Pairwise.** Given two answers A and B, pick the better one. - **Rubric.** Give each answer an independent score on a short list of criteria, then sum. - **Best-of-N.** Generate N candidates and have the judge pick the single best. The bias probes at the bottom are diagnostic tools, not judging modes. They feed the judge comparisons where you already know the right answer, so any deviation from fair is measurable: - **Normal.** Different answers, no rigging. - **Identical.** A and B are the same text. Any deviation from 50-50 is pure position bias. - **Length bias.** A and B are equivalent in meaning, but one is padded. Measures how much the judge rewards verbosity. - **Swap order.** Runs the same comparison twice with A and B flipped. A fair judge returns the same verdict both ways. ## How it actually works Under the hood a pairwise judge is just a prompt template. Something like: 'Here is question Q. Here is answer A. Here is answer B. Which one is better? Respond with just A or B.' The judge generates tokens, the first one is parsed, and that is the verdict. Chain-of-thought prompting asks the judge to reason first and state the letter afterward, which usually improves accuracy at the cost of latency and token count. Rubric scoring uses a longer template: 'Rate this answer from one to five on correctness, from one to five on clarity, from one to five on completeness. Return JSON.' The judge emits a score per criterion, then the scores are summed or averaged. Because each candidate is scored in isolation, rubric judges are less sensitive to order effects than pairwise and more sensitive to the absolute wording of the criteria. Best-of-N runs the generator N times and then runs the judge (in either mode) across the N outputs, keeping the top-ranked one. It is how you buy quality cheaply without retraining: throw compute at the problem and let the judge filter. See our walkthrough on [inference-time compute](https://protrailblazer.com/posts/inference-time-compute-explained/) for the scaling curves. Temperature shows up in two places. The generator's temperature controls how different the N candidates are from each other (too low and you get near-duplicates, defeating the point of best-of-N). The judge's temperature is usually zero to keep verdicts deterministic across repeated runs. ## The biases you have to probe for Every LLM judge has systematic blind spots. Three common ones the demo lets you measure directly: - **Position bias.** Show the same answer twice and a biased judge will still pick one slot more often. This is the most reliably documented LLM judge bias. Mitigation: run every comparison in both orders and average the result. - **Length bias.** Judges tend to reward longer answers even when length adds no information. Mitigation: control for length explicitly in the prompt, or include a short-answer preference example in the instructions. - **Self-preference bias.** A model rates its own outputs higher than equally good outputs from a different model. Mitigation: use a stronger (or at least different) model in the judge slot. Two techniques that hedge against single-judge variance: self-consistency samples multiple verdicts from the same judge and keeps the consensus, and majority voting across a small panel of different-model judges is a stronger version of that when you can afford it. Whatever you do, hold out a small set of human-judged examples as a calibration set. Run the LLM judge on it. The gap between the LLM's verdicts and the humans' verdicts is your bias budget, and it tells you which results to trust and which to double-check. The same [trust-but-verify logic](https://protrailblazer.com/posts/hallucination-simulation/) applies to any model output you plan to act on. ## Key takeaways - Judges are not neutral. A strong LLM grading weaker outputs still shows position bias, length bias, and self-preference bias. Measure each one before you trust a score. - Pairwise comparison is usually more reliable than rubric scoring. But it only holds up if you randomize order and run both directions. - Best-of-N evaluation turns a judge into a cheap quality filter. Quality scales sub-linearly in N and degrades when the judge is weaker than the generator. - When the generator and the judge are the same model, you are grading the model's handwriting with its own ruler. Prefer a different (or stronger) model in the judge slot, and hold out a human-judged sample to bound the bias budget. --- --- title: "Inference-time compute: more samples, smarter answers" description: "A plain-English look at inference-time compute, with a live grid of reasoning chains that vote on the same question and surface the real accuracy-vs-N curve." published: 2026-04-25 updated: 2026-04-25 canonical: https://protrailblazer.com/posts/inference-time-compute-explained/ tags: [machine-learning, demo, inference, llm, reasoning] --- # Inference-time compute: more samples, smarter answers **Answer:** Inference-time compute is the idea that you can make a language model smarter without changing the model, by sampling many answers in parallel and picking the best one. Same weights, more compute at the moment you ask, higher accuracy. It is the core trick behind self-consistency, best-of-N, and the o1, o3, and R1 style reasoning paradigm. The interactive below shows inference-time compute at work. Each cell is one independent reasoning chain solving the same problem. Drag the samples slider from 1 to 64, switch between three strategies, pick a harder problem, and toggle the accuracy curve. The thing to watch: the weights never change. Only the amount of compute spent at question time does, and the accuracy climbs anyway. *Interactive demo: [open the prompt scaling demo](https://protrailblazer.com/demos/prompt-scaling/)* ## What you're seeing A grid of small red cells, each one an independent reasoning chain attacking the same question. Darker cells are idle or thinking. The brief flash and bloom is the moment a chain commits to a final answer. Cells that agree with the majority stay bright red. Cells that voted for a minority answer dim to a duller red, marking them as outliers. Each cell shows its current reasoning step as a tiny monospace string. For the linear equation problem you will see things like `3x=15` tick through to `=5`. For the prime-check problem you will see divisors being tried one at a time. These steps are the cell's internal thinking made legible. Every chain is following its own sampled trajectory, not the same path. The tally panel above the grid collects the votes. Each distinct answer gets a bar proportional to its share of committed chains. The leading answer lights up bright red and the whole panel flashes when a new commit flips the leader. The stats row in the top-right keeps the running count: N is the current sample budget, DONE is the chains that have committed, VOTE is the current majority answer, P_CORRECT is the probability that this majority is correct at this N. A few things to try in the demo: - **Set samples to 1 and run the train catch-up problem ten times.** Most runs will land on a wrong answer like 4:26 PM or 8:00 PM. A single sample is a coin flip weighted by whatever the most likely wrong paths happen to be. - **Drag samples to 32 on the same problem.** The tally stabilizes on 7:00 PM within a few commits. The correct trajectory is not the plurality on its own, but it is the most agreed-upon answer across the distinct wrong paths. - **Switch to GREEDY and drag the slider.** Nothing changes. Every cell runs the same deterministic trajectory, so P_CORRECT is flat regardless of N. This is the baseline that inference-time compute improves on. - **Turn on the accuracy curve overlay.** It plots P_CORRECT across 200 silent trials at N of 1, 2, 4, 8, 16, 32, 64. The shape is the real sub-logarithmic scaling curve from the self-consistency literature, just derived from a hand-authored distribution instead of a live model. ## How it actually works Every language model is a probability distribution over next-tokens. Sample from that distribution and you get one answer. Sample again and you get a different answer, because the middle of any nontrivial reasoning path has real uncertainty in it. A chain-of-thought that spells out the intermediate steps before the final answer exposes that uncertainty instead of hiding it. Inference-time compute leans on exactly that property: instead of asking the model once, ask it many times and treat the population of answers as the signal. Three common strategies for turning many samples into one answer: - **Greedy decoding.** At every step, take the single most likely token. One deterministic trajectory, one deterministic answer. See greedy decoding for the full picture. Sampling more than one chain buys you nothing here because every chain is identical. - Self-consistency. Sample N independent chains at a nonzero temperature, extract each chain's final answer, apply majority voting to return the most common one. This is the standard setup from the self-consistency paper, and it is what the demo's default strategy does. - Best-of-N. Sample N independent chains, score each one (by a reward model, a verifier, or the model's own log-probability on the answer), then keep the highest-scoring chain's answer. This replaces voting with picking, which works well when your scoring function is trustworthy and badly when it is not. Why does voting work? Because the model's error is not uniform. On a non-trivial question, there is usually one correct answer and many distinct wrong ones. A single chain might slip in any of a dozen ways, but slipping the exact same way twice requires two chains to make the exact same mistake. As N grows, correct answers accumulate on one bucket while wrong answers smear across many. The majority bucket wins even if it has less than half the single-sample mass, as long as it beats each wrong bucket individually. N is a hyperparameter you set per request. It trades wall-clock and dollars for accuracy. Scaling from N=1 to N=8 is usually a big jump. From N=8 to N=32 is a smaller one. From N=32 to N=64 is often negligible. The accuracy curve flattens out because you hit a floor set by how often the correct trajectory is even in the sample at all. Once you are reliably including the right answer somewhere, more samples do not help. The o1, o3, and R1 family of reasoning models take this idea one step further. Instead of just drawing N parallel samples, they are trained to generate very long chains of reasoning within a single response, and to use learned value heads to decide which internal branches to expand. The scaling axis is the same though: spend more compute at inference time, get better answers, without changing the weights. ## When this stops working Three failure modes to keep in mind. The first is when the model is systematically wrong. If the most likely wrong answer has more mass than the correct one and every other wrong answer combined, majority vote converges to the wrong answer, faster as N grows. More compute makes a confidently wrong model more confidently wrong. The second is open-ended generation. Majority voting needs a discrete answer to tally. For tasks like essay writing or code generation, there is no single correct string to vote on, so self-consistency does not apply directly. Best-of-N with a good scoring function works instead, but now everything rides on whether the scoring function is actually calibrated. The third is cost. At N=32 you are paying 32x the tokens you would for a single response. For low-stakes chat this is absurd. For high-stakes reasoning (math, code, medical, legal, trading) it can still be the cheapest way to buy accuracy, because the alternative is training a bigger model and that bill is much higher. For adjacent sampling concepts see the [temperature post](https://protrailblazer.com/posts/llm-temperature-explained/) and the [top-k and top-p post](https://protrailblazer.com/posts/top-k-top-p-sampling-explained/). For the broader vocabulary around inference, see the [glossary](https://protrailblazer.com/terms/). ## Key takeaways - Self-consistency works because wrong answers disagree with each other while correct answers tend to agree. The signal concentrates on a single number while the noise spreads across many. - The accuracy-vs-N curve is sub-logarithmic. Most of the gain is between N=1 and N=8. Doubling to N=64 usually buys you a few more percentage points at 8x the cost. - Best-of-N only beats majority vote when your scoring function is trustworthy. If the model is confidently wrong, best-of-N amplifies the wrong answer instead of averaging it out. - Greedy decoding caps accuracy at whatever a single deterministic trajectory gives you. Adding compute to a greedy chain does nothing, because there is no sampling variance to average over. - This is why the new reasoning models feel smarter without being larger. They spend more compute at inference time, not during training. The weights are the same. The wall-clock is not. --- --- title: "Autoregressive generation: how an LLM writes one token at a time" description: "A plain-English look at how LLMs actually produce text, with a 3D demo of the KV cache that makes inference feasible and what would happen without it." published: 2026-04-23 updated: 2026-07-07 canonical: https://protrailblazer.com/posts/how-llms-generate-text/ tags: [machine-learning, demo, llm, inference] --- # Autoregressive generation: how an LLM writes one token at a time **Answer:** Autoregressive generation is how every LLM actually produces text: one token at a time, each new token conditioned on every token that came before it. The KV cache is the optimization that makes this feasible at scale by storing the attention layer's intermediate state for every prior position so those positions do not have to be recomputed on every step. Without it, inference cost scales quadratically with sequence length instead of linearly. The interactive below walks the generation loop in 3D. A write head rises above the sequence, fans attention beams back to every prior token, and a new token drops in at the end. Try the three prompts, then switch from KV CACHE to NO CACHE and watch the COMPUTE stat go from linear to quadratic. *Interactive demo: [open the autoregressive generation demo](https://protrailblazer.com/demos/autoregressive-generation/)* ## What does the demo show? Tokens lay out along a horizontal axis as the sequence grows. Each red block is one token, with its text floating above. Prompt tokens (the seven or eight the user typed) are dimmer. Generated tokens (everything after) are brighter. Below every token sits a short vertical stack of six smaller blocks: that is the KV cache for that position, one block per transformer layer. The glowing sphere at the end of the sequence is the write head, the position being generated right now. At each step the head fans beams back through every prior token. Line brightness is the attention weight: stronger lines mean the model is paying more attention to that token for this step. When the step ends, the write head commits a new token into the slot, the KV cache for that position fills in, and the head shifts one slot right. A few things to try in the demo: - **Let 20 tokens generate in KV CACHE mode, then flip to NO CACHE.** Every prior cache block starts flashing red on each step: the model is recomputing the entire sequence. The COMPUTE stat jumps from about 120 to over 2400. That is what quadratic inference feels like. - **Use the TOP camera preset.** Looking straight down makes the attention fan clearly visible as a 2D graph. Notice how each step the write head reads a different pattern of prior tokens. - **Switch to the code prompt.** Attention patterns get tighter: local + a few long-range spikes back to earlier variable names and function definitions. Narrative prompts have broader long-range attention to story anchors like "kingdom". - **Crank the speed slider.** At 4 tok/s the sequence fills fast. Watch the KV_CACHE stat climb linearly with SEQ_LEN (half an MB per token in this toy), while COMPUTE per step climbs only in KV CACHE mode, rapidly in NO CACHE. ## How does autoregressive generation work? An LLM does not generate text. It generates one token at a time. What looks like a finished sentence is a loop that ran N times, with each iteration depending on every iteration before it. That loop is autoregressive generation. Each step goes like this. The model takes the full sequence so far (prompt plus whatever has been generated already), runs it through every transformer layer, and produces a probability distribution over the vocabulary for the next position. A sampler picks one token from that distribution. That token gets appended to the sequence. The process repeats until a stop condition (an end-of-sequence token, a max length, or the user hits cancel). On a million-token output, the loop runs a million times. Inside each step, the expensive operation is attention. For every position in the sequence, attention computes three vectors: a query, a key, and a value. The model then takes the dot product of the query at position T with every key at positions up to T (that is causal masking), softmaxes those into attention weights, and uses those weights to pool the values. The result becomes the attention layer's output for position T. Here is the key observation: when the model generates token T+1, the keys and values for positions 0 to T have not changed. They were computed once, back when those tokens entered the sequence. If you store them, you do not have to recompute them. That is what the KV cache does. It holds the K and V tensors for every layer, for every prior token, in GPU memory. On each new generation step, the model only computes K and V for the new token, appends them to the cache, and attends over the whole cache. Without this cache, generating token T+1 would require recomputing K and V for tokens 0 through T, then attention over all of them. That makes each step O(T). Summed over N generation steps, total cost is O(N squared). At a 100-token output that is not a disaster. At a 100,000-token output, that quadratic factor is the difference between a real-time response and a machine that is never going to finish. ## Why is the KV cache also the bottleneck? Compute is not the main constraint on long contexts in production. Memory is. The KV cache has to live on the same GPU (or GPUs) running the model, and it grows linearly with sequence length. Do the math for a frontier-scale model. Take a 32-layer model with a 4096-dimension hidden state, running at FP16 (2 bytes per number). For every token, the cache needs K and V at every layer. That is 32 layers times 4096 dims times 2 (K and V) times 2 bytes = 512 KB per token. A 100,000-token context window needs 50 GB of KV cache, on top of the model weights themselves. Most single GPUs do not have that much memory. This is why every production serving stack is heavily optimized around KV-cache management. Techniques like paged attention (serve many requests from a shared pool of cache pages), grouped-query attention (fewer K/V heads than Q heads, shrinking the cache per layer), sliding-window attention (only keep the last W tokens), and quantized caches (store K and V at 4 or 8 bits instead of 16) all exist for the same reason: the KV cache is the main memory cost of long-context inference, and everyone is trying to make it smaller. For related concepts, the [top-k and top-p demo](https://protrailblazer.com/posts/top-k-top-p-sampling-explained/) covers the sampling step (what happens to the distribution after attention), the [inference-time compute explainer](https://protrailblazer.com/posts/inference-time-compute-explained/) covers how models spend extra compute at generation time, the [context window explainer](https://protrailblazer.com/posts/context-window-explained/) shows what happens to a conversation once this cache-hungry window fills up, and the [glossary](https://protrailblazer.com/terms/) has the full vocabulary for inference-side terminology. ## Frequently asked questions ### What is autoregressive generation? Autoregressive generation is how an LLM produces text: one token at a time, with each new token conditioned on every token before it. What looks like a finished sentence is really a loop that ran once per token, and the model never sees its output as a whole, only as the growing sequence it keeps extending. ### What is the KV cache? The KV cache stores the key and value tensors that the attention layers compute for every token, so they do not have to be recomputed on the next step. It is not a memoized function result; it is the actual intermediate attention state for every prior position, held in GPU memory. Caching it turns per-step cost from quadratic to linear in sequence length. ### Why does LLM inference get slower as the context grows? Each generation step has to attend over every prior token, so the work per step grows with sequence length. Even with the KV cache making each step linear, the total cost summed across a long output is quadratic. In production the harder limit is usually memory rather than compute, because the KV cache keeps growing with every token. ### How much GPU memory does the KV cache use? It scales with the number of layers, hidden size, precision, and sequence length. For a 32-layer model with a 4096-dimension hidden state at FP16, the cache is about 512 KB per token, so a 100,000-token context needs roughly 50 GB on top of the model weights. That is why serving stacks lean on paged attention, grouped-query attention, sliding windows, and quantized caches to shrink it. ### Can an LLM run without a KV cache? Yes, but it is wasteful. Without the cache, each step recomputes the keys and values for every earlier token, which makes generation scale as O(N squared) overall. It is fine for very short outputs, and it is what the demo's NO CACHE mode shows, but no production system generates long sequences that way. ## Key takeaways - An LLM does not generate text. It generates one token at a time. What looks like a finished sentence is a loop that ran N times, with each iteration depending on every iteration before it. - The KV cache is not a cache in the usual "memoize a function" sense. It is the actual intermediate state of the attention mechanism (keys and values) for every prior token position, kept in GPU memory so they are not recomputed every step. - Per-step attention cost is linear in sequence length. Total generation cost is the sum from 1 to N, which is quadratic. Long outputs get expensive because cost accumulates with sequence length, not because each step is expensive. - Memory is the usual bottleneck for long contexts, not compute. A 100K-token context can consume tens of GB of KV cache on top of the model weights, which is why production stacks spend so much effort on paged attention, grouped-query attention, and quantized caches. --- --- title: "Gradient descent: how a model rolls downhill" description: "Watch a simulated model find a low spot on a 3D loss surface. A plain-English look at the optimizer that drives modern machine learning." published: 2026-04-19 updated: 2026-06-14 canonical: https://protrailblazer.com/posts/gradient-descent-demo/ tags: [machine-learning, demo, optimization] --- # Gradient descent: how a model rolls downhill **Answer:** Gradient descent is the optimizer that turns 'I'm wrong' into 'I'm slightly less wrong' across millions of training examples. It treats the model's prediction error as a 3D landscape and rolls the current weights downhill toward a low-error spot. The interactive below shows the algorithm in action. Drag to spin the camera, adjust the learning rate, switch between SGD and momentum, and watch the red ball find a minimum. Click the fullscreen button in the top-right of the demo for a closer look. *Interactive demo: [open the gradient descent demo](https://protrailblazer.com/demos/gradient-descent/)* ## What does the demo show? The surface is a **loss landscape**. Each point on it represents a possible setting for the model's weights. The height at that point is how wrong the model would be with those weights. The red ball is the current state. Gradient descent reads which direction is downhill (the negative gradient) and steps the ball that way. A few things to try in the demo: - **Move the learning rate slider.** Too small and the ball barely moves. Too large and it overshoots, bouncing around instead of settling. - **Switch from SGD to momentum.** SGD takes one step at a time. Momentum keeps a fraction of the previous step's velocity, so the ball can roll through small bumps instead of getting stuck on them. - **Reset and restart.** The ball starts in a randomized position. Sometimes it finds the global minimum (the white marker). Sometimes it settles into a local one. That is the entire game. ## How does gradient descent work? Real models have millions of weights, not two. The loss landscape is millions of dimensions, not three. But the math is identical: at each step, the optimizer computes the gradient of the loss function with respect to every weight, then nudges every weight a small amount in the direction that lowers loss. That nudge is the **learning rate** times the **negative gradient**. Two things make this hard in practice: 1. **Local minima.** The landscape is bumpy. The ball might find a low spot that isn't the lowest. In high-dimensional spaces this matters less than it sounds (most "minima" turn out to be saddle points), but it remains a real concern. 2. **Vanishing or exploding gradients.** In deep networks the gradient signal can shrink toward zero or blow up by the time it reaches the early layers, depending on the architecture and activation choices. Backpropagation is the algorithm that calculates the gradients efficiently across every weight in the network. The training loop repeats this forward-pass-then-backward-pass cycle billions of times. Gradient descent is how a model learns every parameter it has, including the [embedding vectors](https://protrailblazer.com/posts/embeddings-in-3d/) that turn tokens into coordinates. Once training converges on a low spot, that same model is put to work [generating text one token at a time](https://protrailblazer.com/posts/how-llms-generate-text/). ## Frequently asked questions ### What is gradient descent in machine learning? Gradient descent is the optimization algorithm that trains most machine learning models. It treats the model's error as a surface, computes which direction lowers that error (the negative gradient), and nudges every weight a small step in that direction. Repeat that across millions of examples and the model gradually gets less wrong. ### What does the learning rate do? The learning rate sets how big each downhill step is. Too small and training crawls, taking far longer than it needs to. Too large and the steps overshoot the bottom, bouncing around or diverging instead of settling. Finding a workable learning rate is one of the first things you tune when training a model. ### What is the difference between SGD and momentum? Plain stochastic gradient descent takes one independent step per update, using only the current gradient. Momentum adds a fraction of the previous step's velocity to the current one, so the optimizer builds speed in consistent directions and can coast through small bumps and flat spots. In practice momentum and its variants, like Adam, converge faster and more reliably than plain SGD. ### Do local minima trap real models? Less than you might expect. In the millions of dimensions a real model lives in, most points where the gradient is zero turn out to be saddle points rather than true minima, and the optimizer can usually escape them. Local minima are a real concern but rarely the thing that breaks training; vanishing or exploding gradients tend to cause more trouble. ## Key takeaways - Gradient descent steps weights downhill on a loss landscape. - The learning rate controls step size. Too small wastes time. Too large misses the bottom. - Momentum carries velocity from the previous step, helping the ball escape small local minima. - The same idea scales from this 3D toy to the millions-of-dimensions reality of a production model. --- --- title: "Embeddings in 3D: how models turn words into coordinates" description: "Type a sentence, watch each token land as a point in 3D space. Click any token to inspect its vector and see which other tokens it's nearest to." published: 2026-04-16 updated: 2026-06-14 canonical: https://protrailblazer.com/posts/embeddings-in-3d/ tags: [machine-learning, demo, embeddings] --- # Embeddings in 3D: how models turn words into coordinates **Answer:** An embedding is a list of numbers that represents a token's meaning. Models learn these numbers during training so words with related meanings end up close together in space. The explorer below tokenizes any sentence you type, projects the resulting vectors into 3D, and lets you click each token to inspect its dimensions and nearest neighbors. The interactive below tokenizes whatever sentence you give it, computes a synthetic 8-dimensional embedding for each token, and projects those vectors into 3D so you can walk around them. Click any token sphere (or its name in the left rail) to see the vector behind it and the four nearest neighbors by cosine similarity. *Interactive demo: [open the embedding space demo](https://protrailblazer.com/demos/embedding-space/)* ## What does the demo show? Each red sphere is a token from your sentence. Its position in 3D is a compressed projection of an 8-dimensional embedding vector. The closer two spheres are, the more similar their vectors by cosine similarity, where 1.0 means identical direction, 0 means unrelated, and -1 means opposite. A few things to try: - **Edit the sentence.** Change a word, swap word order, add punctuation. Watch how the cluster shifts. Punctuation tokens drift apart from word tokens. - **Compare similar words.** Try "cat dog wolf table". The animals tend to cluster, "table" sits further out. - **Click any token.** The right panel shows its 8D vector with each dimension drawn as a horizontal bar. Bigger absolute values mean stronger signal in that dimension. The bottom of the panel lists the four nearest neighbors. - **Watch the heatmap.** Bottom-right is a token-to-token cosine similarity matrix. Red squares mean similar tokens, dark squares mean unrelated. ## How do embeddings actually work? This demo uses a deterministic toy embedding (token character codes hashed into 8 dimensions, with a few hand-coded rules for common stopwords and punctuation). Real models do something fundamentally similar but with two big differences: 1. **Dimension count.** Production embeddings are typically 512 to 12,288 dimensions, not 8. The geometry is the same, just much higher-dimensional. We project down to 3D for visualization, the way you would project to 3D for plotting in TensorBoard or UMAP. 2. **Learned, not hand-coded.** Real embeddings are learned during pretraining. The model adjusts the numbers token by token across billions of examples, shaped by the same [gradient descent](https://protrailblazer.com/posts/gradient-descent-demo/) that trains the rest of the network, until tokens that appear in similar contexts end up near each other in space. No human ever decides "this dimension means animal-ness." Why this matters: once words are coordinates, **meaning becomes geometry**. You can add and subtract vectors (the famous "king minus man plus woman lands near queen" property in well-trained spaces). You can find synonyms by nearest-neighbor search. You can compare entire sentences by averaging their token vectors. The whole field of vector search and [retrieval-augmented generation](https://protrailblazer.com/posts/retrieval-augmented-generation/) rests on this idea. ## Frequently asked questions ### What is an embedding? An embedding is a list of numbers (a vector) that represents the meaning of a token, word, or piece of text. Models learn these numbers during training so that items with related meanings end up close together in the same space. Once meaning is stored as coordinates, you can measure similarity, search, and cluster with ordinary geometry. ### What is cosine similarity? Cosine similarity measures how closely two vectors point in the same direction, ignoring their length. It runs from 1.0 (identical direction, very similar) through 0 (unrelated) to -1 (opposite). It is the standard way to compare embeddings because direction, not magnitude, is what carries meaning. ### How many dimensions do real embeddings have? Production embeddings are typically 512 to 12,288 dimensions, far more than the 8 this demo uses and the 3 we project down to for viewing. The geometry works the same way at any dimension; high-dimensional space just has far more room to separate meanings. Tools like UMAP or t-SNE are used to project them down when you want to look at them. ### How are embeddings used in search and RAG? Search and retrieval-augmented generation both embed your query and your documents into the same space, then return the documents whose vectors are nearest the query. Because nearness means semantic similarity, this finds relevant text even when the exact words do not match. It is the retrieval half of every RAG pipeline. ## Key takeaways - Tokens are represented as points in a high-dimensional space. - Similar meaning means spatially close. Cosine similarity is the standard distance measure. - Real embeddings have hundreds to thousands of dimensions, not 8. We project to 3D only to look at them. - Embeddings are learned during pretraining, not hand-coded. The model finds the structure on its own.