04 section
Inference Optimization
Making generation fast and cheap: the prefill and decode split, KV cache pressure, speculative decoding, continuous batching, PagedAttention, fleet serving, and a cost playbook
What is in here
The fundamentals page sets up compute-bound versus memory-bound, and every page after it is a specific answer to one of those two constraints — read it first even if you came here for PagedAttention. Serving infrastructure moves up a level to the fleet. The cost playbook at the end is the page to hand a manager.
01
3 min
Inference Fundamentals
Prefill is compute-bound, decode is memory-bound, and almost every optimization follows from that split — starting with TTFT, TPOT and FP8 hardware paths
inferencelatencyfundamentals
02
3 min
KV Cache and Context Caching
The KV cache is the memory wall at long context; grouped-query attention, prefix reuse and API prompt caching are the three ways around it
cachinginferencecontext
03
3 min
Speculative Decoding
Guess several tokens with a cheap draft model, verify them in one parallel pass — draft-verify, Medusa heads, lookahead decoding, and their acceptance-rate economics
inferencelatencyserving
04
3 min
Batching Strategies
Static batching wastes GPUs on variable-length generations; continuous and in-flight batching plus chunked prefill are how modern engines keep utilization high
servinginferencecost
05
3 min
PagedAttention
vLLM's borrowing of OS virtual memory: block tables, copy-on-write KV sharing, and the fragmentation waste it removes from every long-running request
servingcachinginference
06
3 min
Serving Infrastructure
Above the engine sits the fleet: inference gateways, tensor versus pipeline parallelism, cache-affinity routing, and holding streaming connections open under load
servingproductionlatency
07
3 min
Cost Optimization Playbook
Ten-times cheaper inference without a quality drop, via model cascading, small language models, spot capacity, and cutting the tokens you never needed to send
costservingmodel-selection