Fleet Efficiency & Eviction
When a serving pool grows past a handful of cards, someone has to judge the cards, and that job is a real interview subject: a senior inference-gateway role lists it as fleet-efficiency work, autoscaling and scoring a rented fleet, evicting underperformers, all aimed at peak tokens per dollar. The model in this topic is one month of one rented pool. Every card in the pool is the same class, but not the same card: each draws a quality lottery ticket at boot, a true serving rate for the rated number, and rides silent dips, rate blips that cost part of the window while they last. The autoscaler never sees the truth. It sees a measured score read through a measurement window, and the window is a lever: over one hour a card can look 25% better or worse than it is, over a day 5%, over a week 2%. The month's bill has three parts: the cards the run set keeps warm bill their rent in full, the cards the score evicts bill 15% of a node-month on the way out (a churned card holds reserved capacity while it drains), and the cards outside the run set never bill at all. Three checks judge the month, and the interview version of this topic is knowing which failure each check can see. The capacity check reads the delivered tokens against demand, because no eviction policy repairs a rate shortage; the convergence check reads the real bill against the ideal operator's bill, the perfect selection with no churn fees sized to the demand shape, because that is what an autoscaler is actually judged on; and the eviction check counts wrong evictions, the cards whose true rate sat at or above the fleet median when the score threw them out, because a wrong eviction is the one failure that refunds nothing. The three presets are the month's three verdicts, and all three pass at least one check, which is why each is dangerous as an incident report: the golden idler bills $69,940 keeping 26 flagship cards warm for a 60,000 tok/s average and burns $44,348 of it on capacity that moved no tokens, yet shows zero unmet demand and zero wrong evictions; the noise evictor scores through the hour window and evicts 12 of 32 cards on a 25% error, 4 of them healthy, 33% wrong; the bargain shortfall bills a tempting $28,840 and covers only 61,015 of 90,000 tok/s, a 32% shortage the bill never could buy back. The fixes carry verified numbers too: the idler on steady mode bills $25,740 for the same demand and converges at 90.9%; the evictor moving to the day window at a 10% eviction share converges at 97.9% with no wrong evictions, and the shortfall swap to flagships covers 90,000 tok/s for $35,360 at 95.6% convergence. The three challenges below work the arithmetic the autoscaler acts on: the tokens-per-dollar score of one card, the whole-card greedy that sizes a run set, and the eviction policy that decides who leaves.
Key Concepts
- 1Rate per dollar, not price per card, is the pool currency: one card scores true_tok_s x 2,628,000 s per month over rent, in millions of tokens per dollar, so the flagship rates 10.108 at $2,600 a month, the bargain card 8.447 at $1,400, and a flagship having a bad month at 2,400 tok/s rates 2.426, a 4x spread on the same invoice line
- 2The score is a window, not a truth: noise lets a card misread by up to 25% of its true rate through an hour, 5% through a day, and 2% through a week, and the eviction policy acts on the misread - which is why a short measurement window evicts healthy cards
- 3A wrong eviction is the only loss that refunds nothing: the card leaves for good and a churn fee of 15% of a node-month still bills (a $2,600 card costs $210 to say goodbye to), which is why the eviction check is not about cost, it is about the 20% wrong-eviction cap
- 4The run set is a policy, not physics: jumpy keeps every survivor warm against a burst that peaks near 1.8x average, steady covers average demand plus headroom, lazy trims to the average - the golden idler (26 cards on jumpy) burns $44,348 of idle bill while the same pool on steady burns $3,595 for identical unmet demand of zero
- 5The bill is the wrong first question: the bargain shortfall shows $28,840 and reads cheap for one second, then the delivered line shows 61,015 tok/s against 90,000 demanded. Capacity fails a 5% unmet cap on delivered tokens; the bill cannot say what the tokens never existed to prove
- 6Convergence is what the autoscaler is graded on: the ideal operator picks by true net efficiency, evicts nobody, and sizes to the demand shape, so it bills $29,400 where the noise evictor bills $30,520 (96.3% converged) and $39,000 where the golden idler bills $69,940 (55.8%)
- 7The three checks stack into one signature month after month: capacity (unmet under 5%), convergence (at least 80% of the ideal bill), evictions (at most 20% wrong) - a month that reads 101, 110, or 011 has exactly one incident, and the interview answer names which knob repairs it
The trade-offs to name out loud
Keeping every survivor warm covers the peak that lands, and bills the peak that never does: the golden idler pays $44,348 idle for capacity that moved no tokens, while the same demand on a trimmed run set pays the whole bill for cards carrying the buffer. The design answer is sizing the burst on the count knob (12 jumpy cards bill $26,780 and converge) instead of parking the whole fleet on the policy knob.
An hourly score reacts to a card degrading the same hour it degrades, and the same hour evicts a healthy card on a blip: 33% of the hour-window evictions in the noise evictor are cards at or above the fleet median. A week window is near-free of error and misses a degrading card for days. The eviction check (at most 20% wrong) is the contract that decides which window is acceptable.
The bargain class rates 4,500 tok/s at $1,400 a month and scores 8.447 tokens per dollar, near the flagship's 10.108, which is the argument used to buy it. The lottery is priced in: a bargain card that draws a bad ticket still carries 45% of its rated rate, and a pool built on cheap cards needs a deeper margin to stay covered when the tickets run cold - the shortfall preset is what the uncovered version looks like.
Balance the fleet's idle time
The fleet-efficiency walkthrough
The meter walkthrough beside this one prices the tokens; this one prices the pool that serves them. The pool is rented by the card, every card carries a quality lottery ticket and silent dips, and the autoscaler only ever sees a measured score through the window you set. Set the pool (cards, card class), the demand (average, headroom, shape), and the score (window, eviction share). Watch three numbers instead of one: the month's bill with its idle and churn lines, the tokens actually delivered against demand, and the tokens per dollar the ideal operator would bill at.
The knobs
The pool
Cards in the pool: 16
The pool is rented by the card, so fleet size is the bill's first multiplier: every card kept warm while idle is a card the bill pays whether or not it moved a token.
Card class: Flagship (H100-class): 10,000 tok/s at $2,600/mo
Rate per dollar is the sticker; the lottery is the fine print. B-stock rates high and is cheap, but a bad ticket carries a quarter of the rated rate and blips twice as deep as the flagship's.
The demand
Average demand: 60,000
Demand in tokens per second is what the pool has to cover at the average minute: everything the pool runs beyond it is either headroom for the spike or dollars for nobody.
Steady headroom: 10
The buffer the run set covers above average demand in steady mode. Headroom is insurance: it keeps cards on whose only job all month is to exist.
Demand shape: Steady: average + headroom
The run-set rule is the fleet's policy, not physics: jumpy keeps every survivor warm for the burst that may never land, and that policy is where a clean capacity story hides an expensive one.
The score
Score window: Day
The shorter the window, the louder the noise in the measured score: an hour lets a card look a quarter better or worse than it truly is, and the autoscaler evicts on it.
Eviction share: 20
The share of the pool churned each month by the score, worst first. Every eviction still bills 15% of a node-month on the way out, so wrong evictions buy nothing and refund nothing.
the month's pool bill (run rent plus churn fees)
$27,170
cards warm in the run set, each billing rent in full
10 of 16
tok/s delivered against 60,000 demanded
60,000
unmet demand, cap 5%
0.00%
billed idle: warm cards beyond the demand
$3,876
churn fees: 3 cards evicted at 15% of a node-month
$1,170
million tokens delivered per dollar (ideal 6.065)
5.803
Where this month's pool bill came from
The month's verdict
The month reads clean: 10 of 16 steady demand cards cover 60,000 of 60,000 tok/s for $27,170, scoring 5.803 million tokens per dollar where the ideal bills 6.065. 0.0% unmet, 4.3% of the bill in churn fees, 95.7% converged. The score, the window, and the run set agree.
16 cards of one class rent as a pool: 10 cards run this month (bill $26,000), 3 get churned (bills $1,170 on the way out), and cards the run set skips never rent at all.
10,000 tok/s rated, $2,600 per month: the fleet's median true rate is 6,723 tok/s, and every slot in the pool draws its own dip and its own measured error on top.
The product demands 60,000 tok/s on average and the pool delivers 60,000: covered, with the bill to prove it.
Steady mode covers average demand plus 10%: 66,000 tok/s of run-set target, so 10 cards carry the buffer all month.
Steady demand: the run set is the measured prefix covering average demand plus 10% headroom.
The score reads through the day window, so in a single window a card can look 5% better or worse than it is (a middling reading): evictions ride this error, not the truth.
20% of the pool is evicted each month by the measured score: 3 cards bill $1,170 in churn fees, 0 of them judged wrong (true rate at or above the median).
Mattered vs noise (which knobs actually moved the bill)
Load a famous fleet month, then fix the plan yourself
The card classes (rated rate, rent, lottery shape)
Python Implementation
Complexity Analysis
Lock in $4.99/mo forever
Joining the founding waitlist locks the $4.99/mo founding price for you, at the lowest price we will ever run. First 100 members. No payment now.