Continuous Batching vs. Static Batching in LLM Serving
Continuous batching keeps GPU slots filled while static methods leave them idle.
Naomi Ibrahim
Section
1 story in Inference Throughput.
Continuous batching keeps GPU slots filled while static methods leave them idle.