Continuous Batching vs. Static Batching in LLM Serving

GPU memory bandwidth, not compute, is the bottleneck that shapes almost every serving decision in production LLM systems, and batching strategy is the primary lever operators have to fight it. Loading a model's weights onto the GPU costs real time and real memory, and that cost only earns its keep when it's spread across as many concurrent requests as possible. Static batching, dynamic batching, and continuous batching answer the same question: how do you keep the GPU fed with useful work instead of idling while it waits on one slow sequence? Only continuous batching actually solves it. Teams still defaulting to the other two are leaving throughput on the table for no defensible reason, and the rest of this piece is an argument for why.

Autoregressive generation creates the underlying problem. Each output token depends on every token that came before it, so a single sequence can't be parallelized the way a normal forward pass can. One request, served alone, leaves most of the GPU's capacity unused. That constraint determines how much wasted capacity a batching method can claw back, and the method a server picks determines how much of it actually gets clawed back.

Static batching and why variable output length breaks it

Static batching is the simplest approach on offer. The server waits until a fixed number of requests accumulate, runs one forward pass across the whole group, and returns results only when the last sequence in the batch finishes. It amortizes one weight load across many activations, a genuine efficiency win for the right workload. For uniform, latency-tolerant work, bulk offline inference over a fixed dataset, say, this is close to the ideal way to use a GPU.

The flaw becomes visible in output the moment output lengths vary, and in real traffic they always do. Every request in a static batch waits for the slowest one before the batch releases and new work can enter. Picture a batch where the longest sequence generates many times more tokens than the shortest. The moment the short sequence finishes, its slot goes idle. It does no useful work, and it can't be reassigned either, because the batch stays locked until the last sequence completes. That slot sits empty for the entire remaining span of the longest generation, burning compute capacity nobody gets back. This is the design flaw that makes static batching wrong for anything but bulk, uniform workloads, and yet it still occurs in production systems that serve genuinely variable traffic.

What dynamic batching fixes and its limits

Dynamic batching fixes exactly one half of the problem: the queueing delay, not the padding waste. Instead of waiting indefinitely for a fixed batch size to fill, the server sets a maximum batch size and a timeout window, and whichever limit hits first triggers execution. Partial batches are allowed. That caps the worst-case queue time before processing even starts, and for latency-sensitive traffic, that cap matters quite a bit.

What it does not touch is the throughput cost. Once a dynamic batch starts running, every sequence inside it is still held hostage until the slowest one finishes, so idle GPU slots persist exactly as they do under static batching, just with a shorter wait bolted onto the front. Vendor documentation tends to blur "dynamic batching," "continuous batching," and "iteration-level scheduling" into loosely interchangeable terms. "Dynamic batching" in particular gets conflated with a scheme where batch size is chosen dynamically but the batch still completes as one locked unit. Those are different mechanisms with different failure modes, and treating them as synonyms is how production systems end up misconfigured, sized for latency they never actually get.

The design logic of continuous batching: scheduling at the token step, not the request

Continuous batching throws out the request as the unit the scheduler cares about. It schedules at the level of a single decode iteration instead: one forward pass, one token generated per active sequence. That shift in granularity is the whole idea, and everything else about the method follows from it.

At each decode step, the server runs a forward pass across every currently active sequence. Any sequence that produces an end-of-sequence token gets pulled out immediately. A new request waiting in the queue slides into that freed slot on the very next iteration. No request waits for an entire batch to clear. It waits, at most, one iteration. That's the mechanism that keeps GPU slots from sitting idle the way static and dynamic batching both allow: short sequences finish, their slots get reused instantly, and the GPU stays full.

None of this is a new idea wearing new branding. The Orca paper is the first published system to implement iteration-level scheduling at this scale, and the Orca team coined the term "iteration batching," now more commonly known as continuous batching. The method carries patent protection in both the US and Korea, which is unusual for a scheduling technique and says something about how foundational it turned out to be.

Why continuous batching needs PagedAttention to deliver its promise

Continuous batching solves compute scheduling, but it opens up a memory-management problem that turns out to be harder to close. Because requests now arrive and finish at unpredictable rates, VRAM fills up with partially used allocations. Aggregate free memory can look perfectly healthy on a dashboard while no single contiguous block is large enough to seat a new request.

The root cause is how KV cache gets reserved in naive implementations: a large context window requires reserving its full KV cache upfront, whether the request ends up generating five tokens or five thousand. That upfront reservation is wasteful by design, and it's exactly the kind of waste continuous batching's iteration-level scheduling cannot fix on its own.

PagedAttention solves it by borrowing a page, almost literally, from operating systems. Virtual memory paging split a process's memory into fixed-size pages decades ago so the OS wouldn't need contiguous physical memory for every process, and PagedAttention does the same thing for KV cache. The cache splits into fixed-size blocks, allocated on demand as generation proceeds rather than reserved in one lump sum upfront. When a request finishes, its blocks return to the pool immediately. Each sequence addresses its own cache through a logical block table that maps to physical blocks that don't need to sit next to each other in memory. That single design choice, non-contiguous physical storage behind a logical address space, removes the contiguous-reservation requirement that was strangling memory efficiency.

The measured effect isn't subtle. PagedAttention improved throughput by 2 to 4x at equivalent latency compared with FasterTransformer and Orca, on the hardware and workloads it was tested against. Continuous batching without it is a half-measure, and treating it as a complete solution causes an unexplained memory ceiling to appear later. Continuous batching with it is what actually ships in production today. Any team running the former without the latter is paying for hardware it isn't using.

Benchmark numbers that show the gap between static and continuous batching in practice

The number that put iteration-level scheduling on the map came from the Orca paper itself: a 36.9x throughput improvement over FasterTransformer at equivalent latency targets, a gap that changes what architecture teams default to before a single line of new research gets published. That's not a marginal optimization. A gap of that size reshapes infrastructure defaults in ways that compound across the industry.

Independent deployment benchmarks backed it up later, under production conditions rather than paper conditions. Anyscale's benchmarks found vLLM delivering a 23x improvement in inference throughput over naive HuggingFace Transformers serving, and an 8x improvement in a separate comparison. Those numbers come from running fleets, not from a controlled lab bench, which matters more to anyone actually paying the GPU bill.

One caveat belongs next to every one of these figures: the gains scale with concurrency. At low concurrency, the bottleneck is the model's own compute cost, not the scheduler, so smarter batching buys you little. The advantage becomes visible in throughput measurements once concurrency climbs. At 64 or more concurrent requests, an optimized configuration sustains roughly 25% higher throughput. For a sense of absolute scale, vLLM benchmarked against TensorRT-LLM and SGLang reached approximately 2,400 tokens per second at 100 concurrent requests under default settings. None of this makes continuous batching a universal win regardless of traffic shape. The win concentrates exactly where production traffic actually lives: moderate to high concurrency, not single-request demos.

Prefill stalling and the chunked-prefill fix in continuous batching

Continuous batching is not a solved problem, and the first place it shows real strain is prefill. When a new request with a long prompt joins the batch, its prefill computation, processing the entire prompt to build the initial KV cache, runs in one single iteration. Every other sequence in the batch, all of them mid-decode, has to wait for that iteration to finish before generating its next token.

This is called prefill stalling, and it causes the p95 latency spikes that occur even in deployments that are otherwise tuned well. The scale of the problem is concrete: a 32,000-token prompt means 32,000 tokens of computation crammed into a single step. On an H100 serving a 70B-class model, that step takes something like 200 to 400 milliseconds. Every other active request sits frozen for that entire window, no matter how short or urgent its own remaining work happens to be.

Chunked prefill is the fix, and any deployment serving interactive traffic without it is making a mistake it just hasn't paid for yet. Instead of executing a long prompt's prefill in one shot, the computation splits across multiple decode steps, so other sequences aren't blocked for the full duration. Prefill throughput itself drops slightly, since it's now interleaved instead of run flat-out, but tail latency across the batch improves substantially, and that's the better trade for anything interactive. Sarathi-Serve is associated with the technique, and chunked prefill support is available in vLLM.

The second failure mode: how memory pressure under continuous batching can collapse throughput entirely

Prefill stalling is a latency problem. This second failure mode is a throughput collapse, and it's more dangerous because the system can look busy while producing almost nothing.

As aggregate KV cache demand climbs toward the hardware's memory ceiling, the scheduler starts preempting active requests, evicting their KV blocks and shoving them back into the waiting queue. That alone would just be an inconvenience. But a preempted request has to recompute its KV cache from scratch when it re-enters, including every token it had already generated before eviction. If new requests keep arriving while this happens, the GPU ends up spending its cycles reprocessing prefills for sequences that were already partway done, producing zero new output tokens in the meantime.

The signature of this failure mode is distinctive, and it pays to recognize it before it appears on a dashboard: GPU utilization is 100%, throughput goes flat or drops outright, preemption counters climb, and P99 latency jumps from normal ranges into several seconds. The system looks maximally busy while it delivers less and less. Hardware budgets make the mechanism concrete. On a 40GB A100 serving a 13B-class model, the model's weights alone consume roughly 26GB, leaving around 14GB for KV cache across every concurrent request. That headroom disappears fast under real traffic, and once it's gone, the preemption spiral is what fills the space where throughput used to be.

Continuous batching implementation in production frameworks and what differentiates them

Continuous batching itself is table stakes now. Virtually every serious inference framework has it, so it's not where the real competition sits. What separates vLLM, SGLang, and TensorRT-LLM is how each one handles the two failure modes above, along with the memory management and scheduling policy decisions that produce them.

vLLM introduced PagedAttention alongside continuous batching, and both run on by default rather than as opt-in features. Reported throughput figures place vLLM at around 2,400 tokens per second at 100 concurrent requests under default settings, and chunked prefill support is available as a configurable option.

SGLang builds on the same continuous batching foundation but adds RadixAttention for prefix caching, along with compressed finite-state machines for efficient constrained decoding, which matters for structured output like JSON generation. Benchmarks from 2024 showed SGLang reaching up to 3.1x higher throughput than vLLM on Llama-70B, though the gap depends heavily on workload shape and how much prefix sharing is actually available to exploit.

TensorRT-LLM takes a different route, targeting deep hardware-level optimization for NVIDIA GPUs. Where vLLM and SGLang compete on scheduling policy and memory strategy, TensorRT-LLM orients its design around extracting throughput from the specific hardware it runs on.

None of the three is a strict superset of the others, and picking one by benchmark headline alone is the wrong method. Each trades off raw throughput, latency consistency, and workload flexibility differently, and the right pick depends on which of the two failure modes above a given deployment is actually most exposed to: a service drowning in long, variable prompts needs the chunked-prefill discipline vLLM offers, one serving highly repetitive structured queries gets more out of SGLang's prefix caching, and one locked into NVIDIA hardware with a fixed serving shape has real reason to reach for TensorRT-LLM. That decision belongs to the failure mode, not to whichever framework has the biggest number on a slide.