Inference Throughput · KV Cache Architecture · Serving Frameworks

A blog and newsletter on KV cache architecture, management, and optimization and GPU inference performance and throughput engineering.

Cover illustration for “Continuous Batching vs. Static Batching in LLM Serving”

Inference Throughput

Continuous Batching vs. Static Batching in LLM Serving

Continuous batching keeps GPU slots filled while static methods leave them idle.

September 23, 202610 min read

Cover illustration for “Prefix Caching for Shared System Prompts”

KV Cache Architecture

Prefix Caching for Shared System Prompts

September 23, 20269 min read

Cover illustration for “vLLM Architecture for High-Throughput LLM Serving”

Serving Frameworks

vLLM Architecture for High-Throughput LLM Serving

September 23, 20269 min read

Latest