04 Inference Optimization 3 min read 548 words
Batching Strategies
Batching is the primary lever for increasing LLM throughput and reducing cost. In 2025, serving frameworks have moved beyond simple request-level batching to sub-token, iteration-level orchestration.
Static vs. Dynamic Batching
In traditional ML (Classification), we use Static Batching where all requests must be the same size and start/end together. This is inefficient for LLMs due to variable response lengths.
Continuous Batching (Iteration-level)
Continuous batching (pioneered by Orca and vLLM) allows new requests to join the batch and finished requests to leave at the end of every individual token generation step.
| Aspect | Static Batching | Continuous Batching |
|---|---|---|
| Join/Leave | Only at start/end | Any iteration |
| GPU Utilization | Low (waiting for longest) | High (always saturated) |
| Throughput | 1x | 4x - 10x |
| Latency | Highest for shortest | Balanced |
In-Flight Batching (Prefill-Decode Fusion)
Previously, serving engines processed a batch of "Prefill" (heavy compute) OR a batch of "Decode" (heavy memory). In-Flight Batching (TensorRT-LLM) allows mixing them:
- 1 request is in the Prefill phase.
- 15 requests are in the Decode phase.
- Benefit: The Prefill request utilizes the GPU's idle compute cores while the Decode requests utilize the memory bandwidth.
Chunked Prefill & RAD-O (Dec 2025)
Massive context prompts (1M+ tokens) can hang a batch for seconds during the Prefill phase, causing "stalls."
The 2025 Fix: Chunked Prefill Instead of prefilling 128k tokens at once, the engine breaks the prefill into smaller chunks (e.g., 4k tokens each) and interleaves them with the ongoing Decode steps of other users. This maintains a steady TPOT even when heavy requests arrive.
Interview Questions
Q: Why is Continuous Batching superior to Static Batching for LLMs?
Strong answer: Static batching forces all requests in a batch to wait for the longest generation to complete (the "longest tail" problem). If one user asks for 500 tokens and another for 5 tokens, the GPU remains idle for the 5-token user for 495 cycles. Continuous batching allows the 5-token user's request to exit the GPU immediately after its last token, freeing up VRAM and compute slots for a new request from the queue. This maximizes "Tokens per Second" across the entire hardware cluster.
Q: What is a "stall" in LLM serving, and how does Chunked Prefill mitigate it?
Strong answer: A "stall" occurs when a massive new request arrives and its Prefill phase (which is compute-hungry) takes 2-3 seconds to complete. During this time, the GPU is so busy with the prefill that it cannot generate tokens for existing users in the "Decode" phase, causing their TPOT to spike. Chunked Prefill breaks that 3-second prefill into small 200ms "chunks," processing one chunk and then doing one round of decoding for everyone else, before returning to the next prefill chunk. This ensures a consistent, smooth experience for all users.
References
- Yu et al. "Orca: A Distributed Serving System for [Transformer] Models" (2022)
- NVIDIA. "TensorRT-LLM: In-Flight Batching" (2023)
- vLLM Project. "Iteration-Level Scheduling" (2023)
Next: PagedAttention
Key takeaways
01
Static batching wastes the short requests
Every request waits for the longest generation, so a five-token reply idles for 495 cycles behind a 500-token one: the page's longest-tail problem.
02
Continuous batching schedules per token
Requests join and leave the batch at any iteration rather than at batch boundaries, which the comparison table puts at four to ten times static throughput.
03
Mixing prefill and decode uses both resources
In-flight batching runs one compute-heavy prefill alongside fifteen memory-bound decodes, so arithmetic units and memory bandwidth stay busy at the same time.
04
Chunked prefill is what kills stalls
A 128K prefill split into 4K chunks interleaved with other users' decode steps turns a two-to-three-second freeze into 200 ms slices and holds TPOT steady.