Context window

The context window is the model-side promise: this many tokens fit in one request, prompt and generated reply together. It is an architecture property, set at training time, not a serving configuration. What it costs the serving layer is direct, because the KV cache that supports the window scales with every token in it: more window means more cache bytes per user at every layer, for every concurrent request the card holds.

The bytes are the pricing. Per user, the cache is cache bytes per token multiplied by the tokens actually resident - so a 4k window and a 32k window differ by eight times the per-user memory, on the same model at the same precision. Since free memory after weights is fixed, the concurrency a card can grant falls in the same proportion, which is why the window a product needs should be sized from the documents it actually serves rather than from the headline the model market quotes.

The honest answer names the trade in seats. A short window with paged, quantized, or architecturally slim cache keeps per-user memory low and concurrency high; a long window buys the ability to answer on long documents and pays per-user memory for every request, busy or not, resident or not. Both are valid products; only one of them has done the per-user arithmetic it is charging.

The same 70B-class model at 160 KiB of cache per resident token on a card with 18 GiB free after its weights, serving users whose full request fits the window.

1. 4k window: 4,096 tokens x 160 KiB = 640 MiB of cache per user.
2. 32k window: 32,768 tokens x 160 KiB = 5,120 MiB, about 5 GiB of cache per user.
3. Seats: 18 GiB holds 18 x 1,024 / 640 = about 28 users at 4k, and 18 x 1,024 / 5,120 = about 3 at 32k - the same card, 8x fewer seats.

A 32k window costs 5 GiB of cache per user against 640 MiB at 4k, so the same 18 GiB of card drops from about 28 seats to about 3.

Where this shows up

• Prompting & Context WindowsYou have typed the same question into a chat window twice, gotten two different answers, and watched a third rewording produce a third answer. Nothing about the model changed between those calls. What changed was the wording - and the wording is close to the whole input the model ever sees. That input is the prompt: the entire text sent with each call, your question plus every instruction and worked example around it. It matters because a model reads a fixed amount of text per call, called the context window, counted in tokens, the word-sized pieces models are measured in at roughly 4 characters each; everything you send counts against the window, and so does everything the model writes, and when the count runs out the oldest text drops silently, exactly like a capped log buffer, with no error anywhere. Two skills inside prompting carry the interview weight. One is format control: a program can only route on replies whose shape it can read, and there are two ways to get the shape, describing the rule in prose, which models follow loosely, and few-shot examples, worked question and answer pairs shown instead of described, which anchor the format at the price of tokens carried on every call. The other is the skill interviewers probe hardest: when you change the wording, prove the change made it better before anyone ships it, the same fixed question set graded under the old wording and the new - show me you would not ship a prompt change on vibes. The three challenges below build exactly that: the swap audit that catches what a single pass rate hides, the packing rule that keeps the most recent turns inside the window, and the label contract that shows what format control has to survive.
• LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.

Related terms