All question banks

Inference GPU Selection Questions

GPU questions come up the moment someone has to pay for serving: how much memory a model and its cache need, what the bandwidth ceiling means for one user, and when a cheaper card or an API is the better buy. Every answer states its assumptions and shows the arithmetic, so you can quote and defend the numbers in an interview. Terms link to the glossary where they are defined.

Warm-up

1. How do you size a GPU card count for a given model?

Count the two residents: the weights (parameters times bytes per element) and the KV cache (per-token cache cost times the context and the concurrency you promise), plus a margin for activations and fragmentation. A card that cannot hold the weights for one user is a non-starter; after that, concurrency is what the leftover memory buys.

Worked answer

  1. A 70B model in 8-bit weights: 70B x 1 byte = 70 GB, so it needs an 80 GB card or two smaller ones.
  2. Two 80 GB cards hold the model twice, wasteful; tensor-parallel across both leaves 160 - 140 = 20 GB for the KV cache.
  3. A 4,096-token user of this model costs 640 MiB of cache, so 20 GB / 0.625 GB = about 32 concurrent users.

Weights decide whether the card is possible; the leftover memory decides how many users it holds - and the two numbers are computed separately, then the cards are chosen once.

Core

2. Why is memory bandwidth the spec to read first, and FLOPs second?

Because decoding is memory-bound: each step reads every weight byte to produce one token, so the tokens-per-second ceiling is bandwidth divided by model size. Peak FLOPs barely enter it until batching or prefill push the arithmetic intensity up. The band check: compute the ceiling from the bandwidth, compare the quoted throughput against it, and distrust specs quoted without context length and precision.

Worked answer

  1. Ridge point of an H100-class card: 989 TFLOPs/s / 3.35 TB/s = about 295 FLOPs per byte.
  2. Decode intensity for a 16-bit 70B model: 140 GFLOPs / 140 GB = 1 FLOP per byte, deep in the memory-bound region.
  3. Decode ceiling: 3.35 TB/s / 140 GB = about 24 tokens per second, set by bandwidth even though compute headroom is vast.

At 1 FLOP per byte against a 295 crossover, bandwidth is the wall - which is why two cards with equal FLOPs but different bandwidth are not the same card.

Core

3. When does a smaller, cheaper card win?

When the demand is concurrency rather than single-stream speed. Compare candidates on tokens per second per dollar at a stated precision: a small card with a small model on it converts its cheap hour into tokens efficiently, while a big card shines when the model itself is too wide for it. The trap is comparing hourly prices without the throughput each price buys.

Worked answer

  1. Card A: 70 GB of 8-bit weights decode at 3.35 / 70 x 1,000 = 47.9 tokens per second, rented at $2.50 per hour.
  2. Tokens per second per dollar for A: 47.9 / 2.50 = about 19.2.
  3. Card B: a 0.86 TB/s card decodes the same weights at 0.86 / 70 x 1,000 = 12.3 tokens per second, rented at $0.90 per hour.
  4. Tokens per second per dollar for B: 12.3 / 0.90 = about 13.6, so the cheap card still loses on this model.

Card A emits 19.2 tokens per second per dollar against B at 13.6 - the cheaper hour buys less bandwidth, and bandwidth is what decode sells.

Hard

4. The model does not fit on one card. What are your options, in what order?

Shrink the bytes before buying cards: quantize the weights (16-bit to 8-bit halves them, 4-bit halves them again), then consider splitting what remains - tensor parallelism when the interconnect is fast, pipeline parallelism when links are slow and aggregate throughput is the goal. A smaller model of the same family is also a legitimate answer when the accuracy budget allows.

Worked answer

  1. A 70B model in 16-bit is 140 GB: no 80 GB card holds it.
  2. In 8-bit it is 70B x 1 = 70 GB: one 80 GB card holds it with about 10 GB left for the cache.
  3. In 4-bit it is 35 GB, leaving 80 - 35 = 45 GB for the KV cache and activations on the same card.
  4. Failing precision wins, two 80 GB cards in tensor-parallel hold the 16-bit model: 160 - 140 = 20 GB of cache headroom, less than the 4-bit single card.

Quantization first: 4-bit on one card leaves more cache headroom than 16-bit across two cards, and it saves the interconnect tax.

Core

5. Does more VRAM always mean more capacity?

No, for two reasons. The cache need scales with context, so a big card at a long context is not as roomy as it looks. And bandwidth differs across card classes, so the same free memory can decode at different speeds - capacity without bandwidth buys concurrency that queues.

Worked answer

  1. A 4k user costs 640 MiB of cache: 9 GiB free holds 9 / 0.625 = 14.4, about 14 whole users.
  2. Doubling the headroom to 18 GiB holds 18 / 0.625 = 28.8, about 29 whole users at 4k.
  3. But a 32k user costs 160 KiB x 32,768 = 5.0 GiB: 18 GiB holds 18 / 5.0 = 3.6, so 3 whole users. The doubled VRAM bought no additional whole 32k user.

VRAM is the concurrency budget, and the context price per user sets how far it stretches - 29 whole users at 4k is 3 whole users at 32k on the same headroom.

Warm-up

6. How many users can one card hold?

Divide the memory left after the weights by the cache each user costs, and say the context explicitly. Cache per user is per-token cost times context length; memory left is card size minus weights and overhead. Two cards of the same size differ once precision differs: an 8-bit cache doubles the same arithmetic.

Worked answer

  1. Per-token cache cost: 2 (a key and a value) x 40 layers x 8 KV heads x 128 dims x 2 bytes = 160 KiB.
  2. One user at 4,096 tokens: 160 KiB x 4,096 = 640 MiB.
  3. About 9 GiB free for the cache: 9 / 0.625 = 14.4, about 14 whole users.
  4. With the cache in 8-bit: 320 MiB per user, 9 / 0.3125 = 28.8, about 29 whole users.

About 14 whole users in a 16-bit cache, about 29 in 8-bit - same card, same weights, same context; the cache bytes decided it.

Hard

7. You are quoted two card classes at different prices. What single number settles it?

Tokens per second per dollar, computed the same way for both: bandwidth divided by model bytes, divided by the hourly price. It collapses bandwidth, card count and rate into one comparable figure. Then check the conditions the number hides: context length, precision, and whether the throughput was sustained or a burst on an empty machine.

Worked answer

  1. Card A: 3.35 TB/s / 70 GB (8-bit) = 47.9 tokens per second at $2.50 per hour.
  2. Tokens per second per dollar: 47.9 / 2.50 = about 19.2.
  3. Card B: 2.0 TB/s / 70 GB = 28.7 tokens per second at $1.80 per hour.
  4. Tokens per second per dollar: 28.7 / 1.80 = about 15.9, so the higher hourly rate wins on this model.

A at 19.2 against B at 15.9 tokens per second per dollar - the bandwidth-to-price ratio, not the sticker, settles it.

Hard

8. A vendor quotes a lower hourly rate. Why can that still cost more per token?

Because the hourly rate is a denominator that hides the numerator: what the card emits per hour depends on its bandwidth against this model. Divide the hourly price by the tokens the card actually delivers in an hour and compare that, not the rate card. The cheaper card can easily cost several times more per million tokens.

Worked answer

  1. The small card (0.86 TB/s) reads 140 GB of 16-bit weights each step: 0.86 / 140 x 1,000 = 6.1 tokens per second.
  2. In an hour that is 6.1 x 3,600 = 22,114 tokens, so the $0.90 hour prices them at about $41 per million tokens.
  3. An H100 in 8-bit runs the same model at about 47.9 tokens per second, so 172,285 tokens an hour at $2.50: about $14.5 per million tokens.
  4. The cheaper hourly rate costs nearly 3x more per token - the rate said one thing, the per-token price said the truth.

About $41 per million tokens on the cheap card against about $14.5 on the H100 in 8-bit: price the tokens, never the hour.

Core

9. Your team serves 50,000 requests a day of 500 tokens each. When is renting GPUs worth the switch away from an API?

Utilization decides. A rented card bills its full hour whether or not it computes, while an API bills only the tokens, so the switch pays when the fleet is busy enough. Compute the token volume, price it through the API rate, price the same volume through a rented fleet at an honest sustained throughput, and let the numbers land where they land.

Worked answer

  1. The workload: 50,000 requests a day x 500 output tokens = 25 million output tokens a day.
  2. API at an assumed $4.80 per million output tokens: 25 x $4.80 = $120 per day.
  3. Rent 2 H100s: 2 x $2.50 x 24 = $120 per day whether used or idle.
  4. At half the workload, the API bill halves to $60 while the rental stays $120.

At the full workload they tie at $120 a day; below it, the API wins by exactly the idle share, which is why utilization decides.

Hard

10. Long-context traffic is the growth area. How does that change GPU choice?

The cache becomes the budget, so memory first and bandwidth second. Long contexts cost more cache per user, which eats the very memory larger cards provide, and each decode step reads that longer cache, so bandwidth matters for keeping their stream steady. Plan to route long contexts to the roomiest machines and keep their traffic priced for its own cost.

Worked answer

  1. A 32k-token user costs 160 KiB x 32,768 = 5.0 GiB of cache - eight times a 4k user.
  2. The 4-bit single card leaves 45 GB for the cache: 45 / 5.0 = about 9 concurrent 32k users.
  3. A 24 GB card with about 5 GB to spare for the cache holds 5 / 5.0 = about 1 such user, at full.
  4. The same card count is worth several times more long-context capacity when the memory is bigger - but their decode also reads 8x the cache per step.

Long contexts buy their capacity in memory, not FLOPs: about 9 concurrent 32k users on one roomy card versus 1 on the small one, with bandwidth the next constraint.

Need the vocabulary first? The glossary defines each of these terms with a worked example, and the other banks cover the other half of the interview.