Prefix Caching for Shared System Prompts

Every production LLM application shares one structural quirk: the system prompt is identical on every single call, and only the user's message changes underneath it. Without caching, the model reprocesses that entire shared prefix from scratch, every time, for every user. A 5,000-token system prompt served across 10,000 conversations means the model computes those same 5,000 tokens 10,000 times before it even looks at what anyone actually asked. That waste appears twice: once in the bill, and once in how long the user waits for the first token to appear.

What the KV cache is and why prefix reuse is mechanically possible

At every attention layer, a transformer produces a key tensor and a value tensor for each token it processes. During normal autoregressive decoding, the model stores these so it doesn't have to redo the math for tokens it already saw. This is the KV cache, and it's what lets generation move token by token instead of recomputing the whole sequence each step.

Prefix caching just stretches that same idea across requests instead of confining it to one. If two different users send the exact same system prompt, the key and value tensors for that shared prefix come out identical no matter whose request triggered them. So instead of two computations, an engine can do the work once and hand the second user the first user's cached result.

The lookup itself is simple in concept. Before prefilling a new sequence, the engine checks the cache for the longest prefix it already has stored. If it finds a match, it pulls the cached KV state and starts computing only from the point where the new request diverges from what's cached. Once the full sequence finishes decoding, its states get added back into the cache for the next request to use. Matching relies on hashing the prefix tokens, so lookup stays fast, roughly constant time, regardless of how large the cache has grown.

Memory doesn't get managed one token at a time. KV tensors are typically grouped into fixed-size blocks, and when the cache fills up, something has to go. Least-recently-used eviction is the standard policy: whatever hasn't been touched in a while gets thrown out to make room.

Where shared system prompts appear in practice

Three kinds of applications get outsized benefit from this. Chatbots and assistants hide a system prompt behind every single turn, and multi-turn conversations make the redundancy worse over time, because each new message has to re-encode the entire prior exchange along with it.

Document QA and retrieval-augmented generation setups are arguably the cleanest case. The system prompt and any few-shot examples stay fixed across queries; only the retrieved chunks and the actual question shift. Placing a cache boundary right at the seam between the static instructions and the dynamic retrieved content produces the payoff. Research on high-traffic RAG endpoints points to token reductions in the 60 to 80 percent range when that boundary is placed correctly.

Agentic and coding workflows are the third case, and maybe the least obvious. Each inference step reuses the system prompt, the tool definitions, and a conversation history that keeps growing. Coding agents that run many rounds of environment interaction (reading a file, running a command, reading the output, deciding the next command) are especially repetitive by nature, since most of the context from one step survives untouched into the next.

What ties these three together: the cached part is long, and the uncached part is short and unique to the request. That ratio decides whether caching is worth the trouble. Workloads with a short stable prefix, or ones where nearly every request looks different from the last, will see a low hit rate and correspondingly thin savings, caching isn't magic, it just moves cost from where the prefix repeats.

Research finds that 31 percent of LLM queries show semantic similarity to previous requests. Roughly a third of production traffic, in other words, is already structurally reusable before anyone does anything deliberate about it.

How major hosted APIs implement prefix caching and what they charge

Anthropic runs an opt-in model built around explicit cache_control breakpoints placed in the prompt. Writing to the cache costs a modest premium over the normal input rate; reading from it costs just 10 percent of that rate, a 90 percent discount. In dollar terms, cache reads run $0.30 per million tokens against $3.00 per million for fresh processing. The default cache lifetime is 5 minutes, extendable to 1 hour by explicitly requesting the longer TTL, which carries a steeper write premium than the shorter option. Break-even is 2 reads: hit the cache twice and the savings outweigh the write cost. Anthropic shipped this in public beta on August 14, 2024, and moved to general availability by December 17, 2024. Minimum cache sizes vary by model, 512 tokens for some, 1,024 for others like Sonnet 4.x and Opus 4.x, and 4,096 for the newest Opus releases.

OpenAI took the opposite design path: automatic caching, no code changes needed. The system detects the longest prefix match above 1,024 tokens and recognizes further hits in 128-token increments after that. Cached tokens bill at 50 percent of the normal input rate, and there's no write premium at all, unlike Anthropic's model. The cache expires after 5 to 10 minutes of inactivity. This rolled out on October 1, 2024, and newer models have pushed the discount as high as 90 percent on cached input. The tradeoff against Anthropic's approach is straightforward: no write penalty, no integration work required, but also no control over exactly where the cache boundary sits.

Google's Gemini splits the difference with a two-tier system. Explicit context caching was introduced at Google I/O in May 2024, giving developers configurable TTLs and the ability to share a cache across requests deliberately. Implicit, zero-setup caching arrived later, added for Gemini 2.5 models on May 8, 2025, with a 90 percent discount on cached input requiring no developer action.

DeepSeek prices cache reads at roughly 10 percent of the standard input rate, putting it in the same range as Anthropic's read pricing.

Prompt engineering rules that keep the cache hit rate high

The cache only fires on an exact match. Byte for byte identical, not "logically the same." Whether any of this works depends on prompt structure.

Static content needs to come first. System instructions, reference documents, few-shot examples: all of it belongs at the front of the prompt. User messages, retrieved chunks, anything that varies request to request, goes at the end. Putting dynamic content anywhere near the front breaks the cache for every single token that follows it, not just the changed part.

This is where per-request variables sneak in and quietly wreck things. A timestamp embedded early in the prompt, a session ID, a personalization field, any of these sitting inside what's supposed to be the stable prefix will force a full recompute on every request. Even something as small as a formatted date that changes once a day will force a full recompute for every request, once a day, for as long as the pattern goes unnoticed.

Serialization has to be deterministic too. JSON key ordering needs to stay fixed, because a serializer that reorders keys produces a different byte sequence even when the underlying content hasn't changed. Whitespace, trailing newlines, encoding, all of it needs to match consistently across whatever services are assembling the prompt on the way out.

vLLM's block hashing versus SGLang's radix tree

Both vLLM and SGLang are open-source inference runtimes that implement prefix caching, and both get real results. Where they differ is in the data structure, and that difference has consequences.

vLLM organizes its KV cache as a flat hash table of fixed-size token blocks, each one independently addressable through a chain hash. Eviction runs on LRU, tuned to approximate what a radix-tree structure would do for typical, mostly-linear workloads. The weak spot occurs in branching scenarios: Monte Carlo tree search, agent rollbacks, prompts with variables scattered mid-sequence. Flat hashing wasn't built to represent a tree that forks in multiple directions, so blocks in these cases end up missing the cache or getting evicted before they're reused, simply because the structure has no way to represent the branch.

SGLang takes a different approach with RadixAttention, which organizes the KV cache as a compressed prefix tree instead of a flat table. Each node in the tree holds KV tensors for a token sequence, and a new request walks the tree from the root, consuming cached nodes for as long as its tokens keep matching, then computing fresh nodes only for wherever it diverges. That gives token-level granularity: SGLang can detect a partial overlap at any boundary, wherever it happens to occur. If a request shares 800 tokens out of a 2,000-token system prompt, those 800 tokens still get reused, block alignment doesn't get in the way. The advantage compounds at high concurrency, where many requests overlap heavily on the same prefix.

FuriosaAI's implementation also uses a radix tree with token-level matching and LRU eviction at the leaves, and extends the idea to hybrid attention models that combine global and sliding-window attention, where a cache hit can end up shorter than the raw token overlap would suggest, because sliding-window validity runs out before the token match does.

Prefix caching in hybrid LLMs (attention + recurrent layers)

Hybrid architectures, ones that mix ordinary attention layers with State Space Model layers, have picked up adoption for long-context serving, largely because SSM layers offer subquadratic compute cost as context grows. That efficiency comes with a catch for caching.

SSM states get updated in place. Unlike KV tensors, which represent a clean, addressable history of every token seen so far, an SSM state can't be rolled back to represent just a prefix of a sequence. A cache entry has to match the exact sequence it was built from, or it isn't usable.

That forces a workaround: cache fine-grained state checkpoints at frequent intervals throughout the sequence, just to have enough entry points for partial reuse. But each SSM checkpoint is large, and most end up sparsely hit, so the cache fills with low-value entries and starts thrashing rather than helping.

Marconi is presented as the first system built specifically to handle prefix caching for hybrid LLMs. It manages SSM states and KV tensors together, making sure every preceding state needed for a given cache entry actually exists before that entry gets served. Its admission and eviction policies weigh reuse likelihood and the compute savings a hit would deliver against the memory the entry costs to keep, rather than falling back on plain recency. The reported results: a 71.1 percent token hit rate, far higher than prior prefix caching systems, and a 617 millisecond reduction in time-to-first-token compared to those same systems.

Why cluster-scale deployments break single-node caching assumptions

On a single machine, prefix caching is a data-structure problem: pick a good hash table or tree, tune the eviction policy, done. At cluster scale, it turns into a distributed-systems problem, and the two aren't the same difficulty wearing a different costume.

Each replica in a cluster keeps its own GPU-resident KV cache, and by default these caches don't talk to each other. A prefix cached on worker A isn't visible to worker B just because both are serving the same model. Round-robin load balancing makes this worse: it scatters requests with identical prefixes across different replicas essentially at random, so each one ends up recomputing the same prefix from zero, and tail latency spikes as a result.

Three architectural answers exist, and each comes with a real cost attached. Centralized routing keeps a global view of what's cached on which worker and sends requests accordingly, which works, but introduces a coordination bottleneck, added latency on every routing decision, and a new failure mode if that router goes down. Shared cache pools, Mooncake and LMCache are examples, let workers pull KV states from a shared remote memory tier, which is powerful inside a datacenter but carries real transfer cost: even a modest prefix takes up meaningful memory, and long-context KV states from larger models can get very large, very fast. Decentralized peer-to-peer routing sidesteps a central bottleneck entirely: each node keeps a local radix tree of its own cache plus periodically refreshed, approximate estimates of what its peers hold, updated through anti-entropy exchange rather than a constantly-consulted central authority.

None of these three is a free lunch. Each trades a different kind of latency, coordination overhead, or memory cost for a cache hit rate that a single node could never reach alone, and which architecture wins depends entirely on the traffic pattern sitting on top of it.