LLM Serving Internals
Before a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
Key Concepts
- 1Decode is bandwidth bound: every generated token requires re-reading the model's weights from card memory, so one request alone pays the entire read for one token. Aggregate decode throughput climbs nearly free as the batch grows, because the same read serves every seat in it. The ceiling follows from that read: a 70B model in bf16 keeps about 140 GB of weights on a card whose memory moves about 3.35 TB per second, so a single request streams no faster than about 24 tokens/s no matter how many FLOPs the card offers, and interviewers expect that derivation on the whiteboard
- 2KV cache bytes per token = 2 x bytes per number x layers x kv_heads x head_dim. For an 8B-class model (32 layers, 8 kv heads, 128 head dim, bf16) that is 131,072 bytes, 128 KiB, per token of every running sequence; a 70B-class model occupies 320 KiB per token, over twice as much
- 3Weights come first: a model of P parameters in bf16 takes 2 x P bytes split across the tensor-parallel cards, and the KV budget is whatever memory is left after the weights. A 70B model on two 80 GB cards leaves roughly 19 GB for all scratch space, and few long-context seats to show for it
- 4Prefill is compute bound and costs roughly 2 x parameters x prompt tokens in FLOPs; a shared cached prefix turns that work into a cheap KV read, which is why requests that reuse each other start so much faster and why one fresh value stamped into a shared preamble destroys the discount
- 5The node count, not the request count, prices dedicated serving: cards are rented by the hour and sized for the peak-minute arrival rate, so the monthly bill is roughly peak arrival rate x request duration divided by granted seats, rounded up to whole nodes whether the pool runs hot or idle
- 6Static batching finishes a whole wave before admitting the next, so one long straggler idles every seat behind it; continuous batching retires and admits requests every step, keeping seats full while work remains queued. The vLLM launch post measured the gap on one 13B model: about 81 decode tokens/s under static batching, about 1,900 under continuous batching
- 7Production allocators name the memory problem, not the scheduler: PagedAttention splits each request's KV cache into blocks and pays essentially no overallocation waste, against the 60 to 80 percent fragmentation waste static reservation left; interviewers expect the term and the mechanism, one sentence each
- 8Speculative decoding is the batch-of-one trick that does not survive a crowded pool: a small draft model proposes the next few tokens, the big model verifies the whole proposal in one pass and keeps the matching prefix, so a lone request can stream two to three times faster, and the same trick pays nothing once the batch has saturated the weight read
The trade-offs to name out loud
A larger batch spreads the weight read over more seats and lifts aggregate throughput per dollar, but every request's wait absorbs the slowest work running beside it, so inter-token latency and queue tails grow. The right setting sits before the memory wall and before the latency knee, and only measurement finds it. The tail diagnosis is a standard follow-up: under static batching the p99 wait is set by whichever straggler holds the wave open, so a sudden p99 regression points at the convoy effect and queueing before it points at the cards.
Every concurrent seat costs memory proportional to its context length, so long contexts shrink the seat budget and a scheduler that ignores the wall keeps asking for seats memory cannot grant, sending the excess to the queue with no error. Trimming context buys back concurrency directly, and so does shrinking the cache itself: halving the bytes each stored number occupies, which is what swapping the KV cache to FP8 does, doubles the seat budget.
A 70B model doubles the cards in a node, eats the memory headroom for its own weights, and more than doubles the scratch space per token, so the seat budget collapses. The quality premium is priced in concurrency and hardware, not just in rate, and the arithmetic decides whether the upgrade survives the budget.
Run the serving pool yourself
The serving internals walkthrough
Behind every streamed answer sits a machine: rented GPUs running the model, admitting requests, and deciding every millisecond how much work to run at once. This walkthrough is the inside of that pool - the view between the highway lab's “pool slot acquired” and the first streamed token. The pool here is dedicated hardware: one tensor-parallel node (the model's weights split across one or two H100 cards, each with 80 GB of memory) rented at a fixed monthly rate, sized for the peak-minute arrival rate rather than the average.
Three arithmetic facts drive everything you are about to touch. Decode (the steady token-by-token writing of the answer) is bandwidth-bound: every step re-reads the entire model from card memory, so serving one request at a time pays that full read for one token. Batching shares the read across seats, buying aggregate throughput almost free - until the KV cache (the per-request scratch memory the model keeps so it can continue each prompt) runs out of room and the batch dial quietly stops mattering. And prefill (reading the prompt to produce that scratch) bills in FLOPs, which is why prompts that reuse each other are so much cheaper to start. Every number below comes from this arithmetic with sourced hardware and pricing constants.
Pick the workload
The SLO: first token in under 1200 ms at p95. Sizing happens at the peak-minute rate. The ceiling: $6,500 a month.
The knobs
Scheduler
Batch size (seats): 16
Model tier
Model tier: small
Request shape
Prompt length (tokens): 1200
Prefix cache hit rate: 30
live seats
16
vs defaults: +0 seats
TTFT p95
27 ms
vs defaults: +0 ms
aggregate decode rate
1698 tok/s
vs defaults: +0 tok/s
monthly bill
$2,913 / mo
vs defaults: $0
The memory ledger
Memory grants all 16 requested seats: 22k KV tokens in use against 464k in the budget. The next constraint is queueing, not memory.
Same config, the other workload
On document analysis this exact config reads latency: pass, capacity: pass, cost: pass. One config never covers both workloads: the arrival rate, prompt shape, and answer length all move the same knobs.
The scheduler runs up to 16 seats at once, and the memory grants all of them.
The 8B class spans 1 card using 16 GB each, leaving 60.8 GB of KV room - room for 463k tokens of cache.
Every request carries a 1,200-token prompt, taking 1.4k KV tokens per seat and prefilling 840 cache-miss tokens in tensor time on arrival.
30% of arrivals skip prefill entirely on a stored prefix, so their first token waits only for the KV read and the queue.
Mattered vs noise on this workload (which knobs actually moved the pool)
Load a famous failure, then fix it yourself
The memory underneath the schedule
The KV cache calculator
The batch walkthrough beside this one schedules requests; this one answers the question underneath every schedule: what does memory grant? Every running request holds a KV cache, the scratch space the model writes so it can continue an answer, and that space is the real resource being scheduled. Dial the four knobs below and watch who pays: one cached token, one running seat, the room left after the weights, and the first-token cost of editing a cached preamble.
The knobs
Model shape
Model tier: small
Weights take their share of every card first, so the bigger model starts from a smaller KV room.
Cache storage: bf16
fp8 stores each cached number in one byte instead of two: only the cache changes, the weights and answers do not.
The pool
Sequence length (tokens): 4000
Every seat pays cache bytes for its whole sequence, prompt plus answer so far, at this length.
Concurrent requests: 16
The dial is the ask. Memory grants what fits, and the excess waits with no error anywhere.
one cached token
131,072 bytes
one seat's cache at this length
0.5 GB
seats granted, room for 115
16 of 16
cache share of the weights
52%
largest sequence the room holds
464k
one sequence outweighs the model
122k
The card pool's memory, weights first
The memory ledger
Memory grants all 16 requested seats: 8.4 GB of cache against a 60.8 GB budget. The binding constraint right now is hardware in front of this lab, not this room.
The 8B class spans 1 card with 16 GB of weights, leaving 60.8 GB of KV room at the 5% margin. A single sequence overtakes the whole model at about 122k tokens of cache.
Storing cache in bf16 costs 2 bytes per number, so one token of cache is 131,072 bytes, the full bf16 bill.
A 4,000-token sequence holds 0.52 GB of cache; the room grants 115 such seats and holds 464k tokens in one sequence at most.
All 16 requested seats run at once, holding 8.4 GB of cache between them, 52% of the weights' memory.
Mattered vs noise (which knobs actually moved the room)
Load a famous failure, then fix it yourself
The prefix lab
Every request opens with the same 1,200-token preamble: role, rules, the shared document. Cached, that opening costs each arrival only a memory read. Edited, it voids the whole prefix and every arrival re-reads it in tensor time. This lab uses the current model and cache knobs from above.
Press both states below and compare the first-token bills.
See the whole lifecycle as traffic: The Request Highway lab turns every AI request into a vehicle that stops at the edge cache, the guardrails, and the rate limiter before the model pool - and drives it home live, with a surge dial and an outage switch.