Model architecture
Mixture of experts
A model built from many expert blocks, only a few of which run per token. Memory rent stays large, compute per token stays small, and a router picks experts.
A dense model sends every token through every parameter. A mixture of experts replaces one large feed-forward block per layer with several smaller ones, and a small gate picks the two or three that look most relevant. Total parameters - and therefore what has to sit in memory - grow with the number of experts, but compute per token only grows with the few that actually activate.
That split is the term to know cold. FLOPs follow active parameters, memory follows resident parameters. A mixture with four times more parameters than it activates computes like a small model while occupying memory like a big one, which is a deliberate trade: capacity bought with expert count, latency paid in routing hardware rather than raw size.
The routing creates second-order effects that interviews like. Expert load balancing controls the real speedup, because one overloaded expert serializes the whole layer. Memory is the real bill: resident parameters of a large MoE need many GPUs even though each token only touches a few. And training is harder - the router can collapse onto a few experts unless load-balancing losses keep the rest alive.
For serving economics, the pitch is capacity per FLOP: you hold a very knowledgeable model in memory and pay decode only for its active slice. Whether that beats a smaller dense model depends on card prices, memory per card and whether your workload actually benefits from the extra capacity the experts encode.
A worked example
A synthetic mixture of experts: 10B shared parameters, 32 layers, 8 experts of 3B parameters each per layer, top-2 routing, weights held in 16-bit.
- Resident parameters: 10B + 32 x 8 x 3B = 778B.
- Active per token: 10B + 32 x 2 x 3B = 202B, which is about 26 percent of resident.
- Weight memory in 16-bit: 778B x 2 bytes = 1,556 GB, about 20 80 GB cards just to hold the weights.
- Compute per token: 202B parameters x 2 FLOPs each = 404 GFLOPs per token, the work of a dense 202B model.
Memory follows total parameters, FLOPs follow active ones: this mixture computes like a 202B dense model while needing the memory of a 778B one.
Where this shows up
- Quantization & the Startup StackThe second serving job is not the fleet; it is the one card, when the budget is what a startup can put on the balance sheet instead of a Vercel bill. Open weights changed the question: the model can be downloaded, and the job becomes making it fit and making it fast on hardware a 24 GB card describes. Three compression decisions run in a fixed order. Weight precision comes first: a model of P parameters takes 2 x P bytes at bf16, 1 x P at INT8, and 0.5 x P at 4-bit, and because decode is bandwidth bound - the card re-reads every weight byte for each token - cutting the bytes in half doubles tokens/sec per stream exactly. An 8B brain goes from 16 GB to 4 GB, and a card re-reading 4 GB through a 1008 GB/s bus decodes 252 tokens a second per stream. KV precision comes second and is the cheaper knob: the per-seat rent of 128 KB a token at bf16 for an 8B-class model collapses quarter-price at 4-bit, which is what llama.cpp KV quantization and vLLM FP8 KV do, while costing about a quality point. Model size, the third knob, is the expensive one - but distillation is how startups move it: train a smaller student on the outputs of a bigger teacher and keep most of the quality at a fraction of the arithmetic. The follow-up interviewers run at startups is the trap answer: quantization is never one decision. Weights, KV, and context length are three separate dial groups, and the 4-bit-weights-on-4-bit-KV stack is the combination that lives just under the quality floor, which is why the quality regression has to be measured, not assumed - bad quantization scales have produced long-context regressions that dropped retrieval recall from 91 to 13 percent before a recalibration put it back. The three challenges below work the stack: the bytes of weights a precision choice buys, the seats a card grants after the per-seat KV rent, and the tokens/sec the memory bus lets through.
- LLM Serving InternalsBefore a streamed answer reaches the screen, a scheduler admits the request into a batch of others running on rented GPUs, decides when its prompt gets read, and keeps it fed token by token until the answer finishes. Serving is that layer, and it runs on arithmetic that is unforgiving. Decode, the steady writing of the answer one token at a time, is bandwidth bound: every step re-reads the model's weights from card memory whether the batch holds one request or sixty, so a pool serving one request at a time pays its most expensive constant for a single token. Batching is the fix, sharing that read across every request running at once, which is why aggregate throughput climbs almost free as the batch grows. The resource that bounds the batch is not compute but memory: the KV cache, the per-request scratch space the model keeps so it can continue each prompt, grows with every layer and with sequence length, and when it fills, the batch size dial quietly stops mattering while the rest of the traffic waits. Prefill, the reading of the prompt that produces that scratch space, is compute bound and costs about 2 FLOPs per parameter per prompt token, which is why two requests sharing a cached opening start dramatically faster than two strangers. Interviewers test this material because the numbers decide the architecture: whether to rent dedicated cards sized for a peak, paid by the hour whether busy or not, or pay per token on a shared pool; how many cards a model swallows for its weights alone before a single seat exists; and why the tail, the p95 wait experienced by the slowest request, is what the service level agreement actually measures. The coding version of this topic is real: an xAI onsite round asks candidates to implement a simplified inference engine, meaning request batching, per-batch inference, and token-level result collection, and other labs ask the design version, batching requests up to a fixed size or a wait cap, whichever comes first. The three challenges below work the core arithmetic: the bytes one cached token occupies, the seats a card's free memory can grant, and the seconds prefill spends before the first token exists.
- Semantic Search & Document QAWhen you type a frustrated question into a help-center search box and the right article still comes back, something matched your words to documentation words. Retrieval is that step, and nearly every AI assistant runs it before answering: gather a handful of candidate documents, score each one against the question, rank the scores, and keep the winners under a reading limit. The scoring comes in two flavors. Keyword matching compares the literal words, using a family of scoring formulas called TF-IDF and its refinement BM25. Meaning matching converts text into vectors, long lists of numbers, then compares those lists with a similarity score called cosine similarity. The reading limit is counted in tokens, the word-sized pieces of text that models are billed and measured in. This matters because search quality bounds answer quality: an answer to a question whose source document was never fetched cannot exist. Interviewers like this topic because it is small, checkable math rather than a vocabulary test, and the classic request is to build all three steps in plain Python with no libraries. The three challenges below do exactly that: the cosine similarity number that orders meaning-based results, the TF-IDF formula that orders keyword results, and the packing step that keeps the selection under a token budget.