AI inference glossary

The model's running notes on text it has already read, kept so each new token reuses them.

A scheduling style that admits a new request the moment an old one finishes.

KV cache memory managed in small blocks instead of one slab per request, the way an operating system pages RAM.

A saved set of KV states for text that repeats across requests, usually a shared system prompt.

Partial output is sent to the client token by token as the model writes it, so the user starts reading seconds before the full reply exists.

The stretch a newly started server spends loading weights and allocating cache before it can answer.

Adding or removing identical inference servers based on a live signal such as queue depth.

The latency targets a service commits to, split per phase: time to first token, speed while reading, and the whole reply.

Standby servers with the model already loaded, ready to admit traffic the moment a burst starts.

How long a caller waits before the reply's first word: queue wait plus the prefill pass over the prompt.

Requests line up when they arrive faster than the server admits them.

Signals from a saturated server that ask callers to slow down - bounded queues, admission limits, refusals - so overload surfaces early, not as wait times.

The wait the slowest requests get, read at p95 or p99.

Spreading requests across replicas so no card queues while another idles - and choosing what to balance for, because perfect balance can destroy cache locality.

Routing each request family to a card by hashing its key, so most families keep their card and their resident prefix even when the fleet grows or shrinks.

Designing requests so a retry cannot bill twice or answer twice, because a provider retry re-bills the whole request at full prompt and output price.

One user request that spawns many model calls at once, multiplying token volume and provider pressure.

A counter at the gateway, often a token bucket, that refuses arrivals above a chosen share before they reach a card - where a refusal costs nothing at all.

A trip switch between your code and a misbehaving provider: past a failure threshold, calls fail fast and take a fallback rung instead of a dead wait each.

The stack of caches above a model - reply, prompt prefix, and gateway - where a hit at a higher layer costs less of the bill than any hit at the GPU layer.

A cheap draft model proposes several tokens and the main model verifies them in one pass, so a step can emit more than one token while the wording stays exact.

How many tokens one GPU emits per second, for one user or a batch.

Storing model weights as 8-bit numbers instead of 16-bit halves their size.

A rule for guessing a kernel's speed: the smaller of what its math could reach and what memory bandwidth allows.

Small adapter matrices that add fine-tuned behavior on top of shared base weights.

Two serving scores: tokens per second across the whole fleet versus how fast one user sees a response.

The share of a card's billed time that actually does serving work, and the share of its bandwidth serving reaches.

Storing the per-token cache in 8 bits instead of 16 halves its memory, roughly doubling the concurrent users one card holds, at one measurable risk.

What a full prefix cache forgets when a new family needs the space.

The accuracy a model gives up when weights or cache move to fewer bits.

What a million tokens cost, listed for input and output separately.

Calling a provider's pooled GPUs per request instead of running your own.

Spare GPU capacity rented at a deep discount, with the provider allowed to reclaim it on short notice.

The arrival-rate arithmetic that decides how many cards to rent before the first bill arrives: peak arrivals, seats granted per card, and one spare.

A model built from many expert blocks, only a few of which run per token.

The model's weight matrices are split across several GPUs that compute every layer together.

The model is cut into stages that run on different GPUs, one after another.

An attention variant that shares each key-value head across several query heads, shrinking the KV cache per token and raising how many users one card holds.

How much text one request can hold, prompt plus reply.