vLLM Architecture for High-Throughput LLM Serving
vLLM uses paged memory and intelligent scheduling to cut GPU idle time from 60-80% to under 4%.
Every large language model server running in production faces the same arithmetic problem: the KV cache, the running record of attention states that lets a model remember what it has already generated, grows linearly with sequence length and with batch size. Get that memory management wrong and most of the GPU sits idle before a single useful token gets produced. vLLM, the open-source inference engine that came out of UC Berkeley, is built almost entirely around solving that one problem, and its fixes form a chain: each closes a gap the last one exposed.
What vLLM set out to do
Before vLLM, most serving systems allocated a fixed, contiguous block of memory for each request's KV cache, sized to the maximum sequence length the system supported. That design makes sense if the workload is predictable. It fits language models poorly, since one request might stop after 50 tokens and another might run for several thousand, with no way to know which in advance. Static batching made the waste worse: group 16 requests together, and if one needs far more tokens than the rest, every shorter request sits there holding its GPU memory slot open for tokens it will never produce. Before PagedAttention, fragmentation and over-allocation wasted somewhere between 60 and 80 percent of allocated KV cache memory. That is most of a GPU doing nothing.
vLLM came out of UC Berkeley's Sky Computing Lab in 2023, built around the paper "Efficient Memory Management for Large Language Model Serving with PagedAttention." The "v" in the name originally stood for "virtual," a direct nod to virtual memory, the decades-old operating system trick that gives every running process the illusion of a clean, contiguous address space even though the underlying physical memory is scattered and shared among competing processes. That borrowed idea turned out to be the right lens for a problem that looked, on the surface, like it belonged purely to machine learning. The project is licensed under Apache 2.0 and lives at github.com/vllm-project/vllm. It has become a widely adopted engine for teams serving open-weight models at scale, and treating it as one option among many misses how thoroughly it reset expectations for what serving efficiency should look like.
PagedAttention: how borrowing an OS idea eliminated KV cache fragmentation
The mechanism is straightforward once the analogy lands. Instead of reserving one contiguous block of memory per sequence, PagedAttention splits the KV cache into fixed-size blocks, pages, that can live anywhere in physical memory. A block table maps each sequence's logical positions to wherever its pages actually sit, the same indirection an operating system uses to give a process a clean virtual address space over messy physical memory underneath.
The payoff appears in three places. Pages get handed out on demand as a sequence generates tokens, so there is no big upfront reservation sized to a worst-case length nobody expects to hit. Variable-length sequences no longer require reallocation or copying as they grow, since new pages just get added to the table. And because pages are addressed independently, two requests with identical token sequences can point to the same physical pages instead of each storing its own copy, a property that becomes the foundation for prefix caching later on. The measured effect: PagedAttention brought KV cache waste down from that 60 to 80 percent range to under 4 percent. Anyone still running fixed-block allocation in production today is leaving most of a GPU's memory on the table for no reason beyond inertia.
Continuous batching: keeping the GPU busy as requests arrive and finish at different rates
Memory efficiency solves half the problem. The other half is scheduling, and static batching fails there for a related reason: a batch gets treated as one unit, launched together and finished together, so a request that could have returned in half a second sits waiting on whichever request in the batch takes longest.
Continuous batching removes that boundary. The moment one sequence finishes, a new request slots into the batch, with no waiting for the rest of the group to catch up. Under the hood, the engine runs a loop, commonly structured as a step() call, that repeats three stages every iteration: scheduling, where it decides which requests run this round (decode steps, chunked prefill chunks, or both); model execution, the forward pass over whatever got scheduled; and postprocessing, where sampled tokens get appended, detokenized, and checked against stop conditions. The forward pass processes the whole batch together, handling requests at different stages simultaneously, so this scheme works even inside vLLM's simpler, synchronous engine path. It never needed some exotic async mode to pay off.
Chunked prefill: preventing long prompts from stalling in-flight decode requests
Continuous batching has a blind spot. A genuinely long prompt, 32,768 tokens, say, can occupy the GPU for a substantial stretch of uninterrupted computation during prefill, and every decode step for every other in-flight request queues up behind it. That is head-of-line blocking: one long request monopolizes the GPU and drags up latency for users who have nothing to do with it.
Chunked prefill fixes this by breaking a long prefill job into smaller pieces, sized by a token budget, and interleaving those pieces with decode steps from other requests instead of running the whole prefill in one uninterrupted block. It is turned on with --enable-chunked-prefill, and as of vLLM's V1 engine, it is on by default, which is the right default: the alternative reintroduces exactly the stalling this mechanism exists to prevent. The chunk size is set through --max-num-batched-tokens, and the choice is a real tradeoff. A smaller budget, 2,048 tokens, forces more frequent interleaving and better latency fairness across requests. A larger budget, 16,384 tokens, favors raw throughput at the cost of that fairness. For latency-sensitive, multi-tenant serving, the smaller budget is the sane starting point; throughput can be bought back other ways, but a stalled decode step is a stalled decode step.
Automatic prefix caching: skipping recomputation when prompts share a common opening
Chatbots, retrieval-augmented generation pipelines, coding assistants: a huge share of real production traffic sends requests that all open with the same system prompt, sometimes running to hundreds of tokens, before the user's actual input even starts. Recomputing the KV cache for that shared prefix on every single request is pure waste, and treating it as an acceptable cost of doing business is the mistake plenty of teams still make.
Because PagedAttention already stores the KV cache in discrete, independently addressable pages, vLLM can detect when two requests share an identical token sequence and point both at the same physical pages instead of computing them twice. If two requests share the first 512 tokens of a system prompt, the prefill cost for that shared portion approaches zero on a cache hit, since the KV blocks are already sitting in memory from the first request. With prompts structured well, cache hit rates above 87 percent are achievable, which turns a meaningful chunk of every request's prefill work into a lookup instead of a computation. Given that payoff, the failure to structure prompts with a stable, shared prefix up front is not a minor inefficiency; it is giving away most of the benefit for free.
Speculative decoding: using a cheap draft model to generate tokens the target model then verifies in parallel
Autoregressive generation is sequential by design: producing the next token requires knowing the one before it, so a full forward pass through the model is the price of every single token. Speculative decoding gets around that constraint without changing the model's output distribution. A small, cheap draft model proposes several tokens ahead, and the large target model checks all of them in a single forward pass, accepting the ones that match what it would have generated anyway and rejecting the rest.
vLLM supports several variants of this, and they differ in where the draft tokens come from. Medusa attaches extra lightweight decoding heads directly onto the target model, so there is no separate draft model. The heads predict several future positions at once and get evaluated through tree-structured attention, with reported speedups varying across configurations and workloads. EAGLE, and its successors EAGLE-2 and EAGLE-3, take a different approach, using learned draft mechanisms that go beyond simple token-level prediction to improve acceptance rates. EAGLE-2 added dynamic draft trees on top of that, and EAGLE-3 has reported substantial speedups on large models. A fourth option, n-gram proposals, skips learned models entirely and pattern-matches against the prompt itself to guess likely continuations, a lightweight approach that composes cleanly with chunked prefill under V1. On the ShareGPT dataset, speculative decoding in vLLM has delivered around a 21 percent throughput improvement alongside roughly a 20 percent cut in latency, according to published benchmarks.
One compatibility detail matters for anyone actually deploying this. Draft-model-based speculative decoding (Medusa, EAGLE) has carried compatibility constraints with --enable-chunked-prefill in certain engine configurations. N-gram-based speculative decoding on GPU has generally offered broader compatibility under V1, which makes it the more dependable default for anyone unwilling to give up chunked prefill's latency guarantees just to chase a bigger speedup number.
The V1 engine rewrite: fixing scheduling overhead that accumulated as concurrency scaled
Every optimization above ran inside vLLM's original engine, and that engine carried its own hidden tax. As concurrency scaled up, the scheduler itself started eating into throughput. The prior architecture incurred growing overhead as concurrency scaled, quietly eroding the very gains PagedAttention and continuous batching were supposed to deliver.
V1 was announced in alpha in January 2025 and became the default engine in a subsequent vLLM release that same year. Its core fix is architectural: pin host memory and use direct DMA transfers, zero-copy, to eliminate the redundant back-and-forth that happened during token sampling and output processing. vLLM's own announcement of the alpha cited meaningful speedup gains, alongside a simpler and cleaner scheduler, near-zero-overhead prefix caching (the same mechanism covered above, now with far less bookkeeping cost), cleaner tensor parallelism, a multiprocessing-based API server, and a set of optimizations that used to require manually set flags and now ship on by default. The lesson buried in that changelog is one to take seriously on its own: a memory-management win at the kernel level does not survive contact with production if the scheduler wrapped around it is still paying a hardware transfer tax on every step.
Prefill-decode disaggregation: separating two fundamentally different compute phases onto different hardware
Prefill and decode are not the same workload wearing different clothes, and treating them as such on a single GPU is the last major inefficiency this chain of fixes has to confront. Prefill, processing the full input prompt before generation starts, is compute-bound: it benefits from large token budgets and from the chunking strategy described earlier. Decode, generating one token at a time afterward, is memory-bandwidth-bound and latency-sensitive instead, and it wants small batch sizes with tight, predictable scheduling. Asking one GPU to do both well, at the same time, for different requests, means asking it to optimize for two contradictory things at once.
Even with chunked prefill smoothing things out, a heavy prefill job sharing a GPU with active decode requests still degrades their latency, because the two workloads compete for the same memory bandwidth and the same scheduler's attention. Disaggregation is the structural answer: run prefill and decode on separate pools of GPUs. Prefill nodes process incoming prompts and ship the resulting KV cache across the network to decode nodes, which then run generation uninterrupted by any prefill traffic.
The payoff goes beyond removing interference. Prefill and decode pools scale independently to match however the workload's shape shifts, which matters because the ratio of prompt length to generation length varies enormously across use cases, from a short chat turn to a long document summarization job. Hardware heterogeneity becomes something to exploit rather than a problem to route around, since different GPU types suit each phase's demands differently: decode's bandwidth hunger and prefill's compute hunger do not need to be served by the same silicon. Each phase gets tuned on its own terms, its own tensor-parallel or pipeline-parallel strategy, its own token budget, without either one paying a penalty for compromises made to suit the other. For any team running at real scale, colocating the two phases on the same GPU is no longer a reasonable default; it is a legacy pattern that disaggregation has already made obsolete.
