What is prompt caching? Why prompt order changes the bill

What you're seeing

The ribbon at the top is one request. Each small cell is 20 tokens, grouped into blocks: SYSTEM (400 tokens), TOOLS (600), DOCUMENT (2,000), HISTORY (grows every turn), and QUESTION (40). Dim cells were read from the cache. Bright cells were computed fresh and written into the cache for next time. The white cut line is the first token that differs from the cached request. Everything before it is a read. Everything after it is a miss, no matter how much of it is identical.

The diagram in the middle is a stock-and-flow view of the same request. The request enters the prefix match and splits two ways: READ (tokens served from the cache) and COMPUTED (tokens processed fresh). Fresh tokens also flow into CACHE, because every miss on a cached request is also a write. Both branches feed COST, the running total in example units. The strip at the bottom plots cost per request, with a dashed line for what the same conversation would cost with no cache at all.

The sim loads with two turns already sent, so the first frame shows the normal case: a long dim prefix and a short bright tail. Turn 1 was a full miss, because the cache was empty. Turn 2 read 3,040 of its 3,230 tokens and computed only the 190 that were new: the previous reply and the new question. That is a 94% hit rate. A few things to try:

• Put QUESTION first. Drag it to the front, or tap it and press Earlier until it is first. The question changes every turn, so the cut lands at token 0 and the hit rate is 0% from then on. Nothing after the first difference can be reused, including 3,000 tokens that did not change.
• Turn on the timestamp. A six-token block appears at the very top and changes on every request. Six tokens, and the hit rate drops to 0% on every turn. The bill ends up higher than with no cache at all, because every token now pays the write price.
• Press Revise document. A passage near the top of the document changes. On the next request the cut moves back into the document, the hit rate drops to under 35%, and the turn after that it recovers to over 90%, because the revised document is now what the cache holds.
• Press Wait a few times. Each press adds five idle minutes. Once the idle clock passes the TTL (5:00 by default), the entry expires and the next request is a full miss, with the full write cost again.
• Turn the cache off. Every token is computed and the cost bar lands exactly on the dashed no-cache line.

What is prompt caching?

Prompt caching is a feature of a model provider's API that stores the intermediate work the model did on a prompt prefix, so a later request that starts with the identical tokens can skip that work. The provider charges a reduced price for tokens served from the cache, and usually a premium for tokens written into it. The feature exists because most of a long prompt is the same on every call: the system prompt, the tool definitions, a reference document, a few examples. Only the last part, the user's new message, changes.

The defining property is that it is a prefix match. The cache does not look for the document somewhere in the prompt and reuse it wherever it appears. It compares from the first token forward and stops at the first difference. That single fact explains every result in the sim, and it is why the order of your prompt is a cost decision, not just a style one.

How does prompt caching work?

When a transformer reads a prompt, each attention layer computes a key and a value for every token position and stores them in the KV cache. Under causal masking, the work at position 10 depends on tokens 1 through 10 and nothing after. So if two prompts share their first 3,000 tokens, the keys and values for those 3,000 positions are identical and can be stored once and loaded on the second request instead of recomputed. Prompt caching is exactly that: the provider keeps the KV cache for a prefix and reattaches it when a matching request arrives.

The same dependency is why only a prefix can be reused. Change token 10 and the saved work for position 11, 12, and every later position was computed in the presence of a different token 10. It is wrong, so it has to be thrown away. The cache can keep everything before the change and nothing after it. In the sim, each request is an array of token IDs, the sim computes the longest common prefix against the cached array, and the cut line is that length. No outcome is scripted. A console.assert in the page checks on every request that the read count equals the longest common prefix, and that the COST total equals the sum of the per-request line items.

Two consequences fall out of this. First, the first request in any conversation is always a miss, because there is nothing to match yet. Second, the model's reply joins the history on the next turn, so it arrives as new input and gets computed and written once. In the sim that is why every turn after the first computes exactly 190 tokens: the 150-token reply plus the 40-token question. The prefix grows by 190 tokens a turn, and the hit rate creeps up from 94% toward 96% as the stable part gets longer relative to the new part.

Prompt caching vs KV cache

They are the same data structure at two different time scales. Inside one response, the KV cache is what lets the model generate the next token without re-reading the whole sequence; the autoregressive generation demo shows that loop, including a NO CACHE mode where every step rereads everything. Prompt caching takes the same keys and values and keeps them between requests, across seconds or minutes, keyed on the exact prefix. The first is an inference optimization you get for free. The second is a billing feature with a price list, a minimum size, and an expiry clock.

Does prompt caching save money?

Usually, and sometimes it costs more. The sim uses example prices in relative units: a fresh input token costs 1.0, a cache read 0.1, and a cache write 1.25. Every fresh token on a cached request is also a write, so a miss is more expensive with caching on than off. With that pricing, the default six-turn conversation works out like this:

• Default order, six turns. Turn 1 costs 3,800 units instead of 3,040, because all 3,040 tokens are written. Turn 2 costs 542 instead of 3,230. The six-turn total is 6,698 units against 21,090 with no cache, about 32%.
• Timestamp at the top, six turns. Every request is a full miss and every token pays the write price. The total is 26,408 units against 21,126 with no cache, about 125%. The cache made the conversation a quarter more expensive.
• Question first. Same outcome as the timestamp: 0% hit rate, write price on everything.

Real prices differ by provider and change often, so here is one set of numbers with a date on it. As of November 4, 2026, Anthropic's prompt caching documentation and pricing page state the following:

• Opt-in. You mark a breakpoint with cache_control, either on a content block or once at the top level of the request. Up to four breakpoints per request.
• Prefix order. The cache covers tools, then the system prompt, then messages, in that order, up to the breakpoint. The docs put it plainly: "Changes at each level invalidate that level and all subsequent levels," and a hit requires "100 percent identical prompt segments."
• Minimum size. 512 tokens on the current models (Claude Opus 5.5, Sonnet 5.5, Haiku 5.5), 1,024 on several earlier ones, and up to 4,096 on some older models. A shorter prefix is processed without caching and no error is returned.
• Prices. A cache write costs 1.25 times the base input price for the default 5-minute lifetime, or 2 times for a 1-hour lifetime. A cache read costs 0.1 times base on most models, and less on the newest: 0.05 times on Opus 5.5 and Sonnet 5.5, and 0.025 times on Fable 5.1.5. In dollars, Opus 5.5 lists $4 per million input tokens, $5 for a 5-minute write, $8 for a 1-hour write, and $0.20 for a read.
• Lifetime. Five minutes by default, and "the cache is refreshed for no additional cost each time the cached content is used." The sim's Wait button models exactly this: the clock resets on every hit and expires once the gap between requests is longer than the TTL.

The sim's example multipliers (0.1 read, 1.25 write) match that standard schedule, so the percentages above are a fair guide for it. Other providers document their own versions, with different discounts, minimum lengths, lifetimes, and opt-in rules, and I did not verify their current pages for this post. Check the provider's own docs before you budget, and date the numbers when you write them down.

Why is my prompt cache not hitting?

Almost always because something near the top of the prompt is changing, even by one token. Work through these in order:

1. A date, time, request ID, or user name in the system prompt. This is the timestamp toggle in the sim. Six tokens at the top cost you everything beneath them. Move it to the end of the prompt, or drop it.
2. Content that is stable but not identical. Tool definitions serialized in a different order, a document re-fetched with trailing whitespace, JSON with unsorted keys. The cache compares tokens, not meaning.
3. The breakpoint sits before the content you expected to reuse, or there is no breakpoint at all on a provider where caching is opt-in.
4. The prefix is under the minimum size. Short prompts are processed normally and silently never cache.
5. The entry expired. A conversation that pauses longer than the TTL comes back to an empty cache. Low-traffic apps hit this constantly with a 5-minute lifetime.
6. You edited the middle. Revising a document, trimming old history, or changing the model's effort setting mid-conversation all move the cut backward for one request. That is expected. If it happens every request, something upstream is regenerating content.

The fastest diagnosis is to read the usage block the API returns. If cache reads are zero on a request you expected to hit, diff that request against the previous one token by token and find the first difference. It is the cut line, and it is always somewhere specific.

Where to put the system prompt for caching

First, with the other things that never change, and with anything that changes per request moved out of it. The sim's default order is the general rule: tools and system prompt, then large reference material, then the conversation history, then the new question. Sort by how often each block changes, most stable first. Memories and user preferences that stay fixed for a whole session can live in the system prompt, which is how the memory demo injects them. Anything that changes per request, such as the current time or a per-call instruction, belongs after the last breakpoint, at the end.

History deserves a note. It grows at the end on every turn, which is the ideal shape for a prefix cache: the growth lands after the cut, and last turn's growth is inside the prefix next time. But history also eats the context window, and the moment you trim or summarize it, you edit the middle and take a one-request miss. The context window demo shows what falls off the back. Plan the trim as a deliberate, occasional event rather than something that happens a little on every turn.

What the demo simplifies

• One cache entry, one model, and a match at any token. Real providers add a minimum cacheable length, may match only at the breakpoints you set, and keep separate entries per model and per organization.
• Token IDs are seeded random integers, and an edit regenerates the IDs from the edit point on. The matching logic is the same longest-common-prefix rule a real cache applies to real token IDs.
• Each turn adds 90 seconds of idle time and the entry refreshes on every use. Providers differ on whether reads refresh the clock and on what lifetimes they offer.
• Costs count input tokens only, in relative units. Output tokens are priced separately and the cache does nothing for them.
• Latency is not modeled. Cache hits also cut time to first token, but the size of that saving depends on the provider and the prompt, so this post does not put a number on it.

Frequently asked questions

Key takeaways

• Prompt caching is a prefix match. The provider reuses the model's work from the first token up to the first difference, and nothing after it. That makes prompt order a cost decision.
• One changing token at the top makes everything beneath it uncacheable. A six-token timestamp in the system prompt took the sim's hit rate from 94% to 0%.
• Writes cost more than fresh tokens, so a cache that never hits costs more than no cache. In the sim the timestamp conversation cost 125% of the no-cache price; the default order cost 32%.
• An edit in the middle is a one-request miss, not a permanent one. The cut moves back to the edit point, and the revised content is in the prefix on the next turn.
• The first request is always a miss, and the reply you got back is new input next turn. Savings start on the second request and grow as the stable prefix gets longer relative to the new part.