Serving AI reliably: watch, fall back, roll back
Nobody told me this at the time, so I learned it from a support ticket: a feature can be broken for two days with every dashboard green, because the dashboards were watching the software and nobody was watching the model. Serving an AI feature reliably means three habits stacked on top of the usual backend ones, and they line up as watch, fall back, and roll back. Watching means measuring quality, not just uptime: keep a fixed set of realistic inputs, run them against the live system on a schedule, and grade the answers, because a model can be up and wrong at the same time. Latency is watched as a percentile, not an average. p95, the wait that one request in twenty exceeds - you line up every request's wait and read the one that 95 percent of requests came in under - because averages hide the exact users who leave, and p99 pushes that out to the slowest one in a hundred. Falling back means a ladder standing behind the primary call, in cost order: first retry - trying again after a pause that doubles each time, the pattern called backoff - because many failures self-heal in seconds; then a cheaper second model, because rate limits and outages are per-model properties and the primary being down does not mean everything is; then a canned answer, a fixed honest response written in advance, the rung that guarantees the user never meets a bare error. Rolling back means the graded set doubles as the tripwire: when a new model version or prompt change ships and the grades move the wrong way, the change goes back out, and the graded set is both what noticed and what proved it. All three habits exist for the same reason: the model is a dependency that changes with no code change, and every other kind of dependency ships with a changelog.
Most people have this backwards
A 100 percent uptime number says nothing about whether your AI feature is working.
The catch
It holds once answer quality is something a user can notice: the feature can return confident wrong answers, or serve the canned fallback to every fourth request, and both count as fully up in the infrastructure sense. It weakens where the AI call is genuinely plain infrastructure - a classification step whose correctness the surrounding code enforces anyway - because there, uptime is nearly the whole story. The interview version of the question: what exactly did your dashboards see on the day the answers went wrong?
Baseline first - your honest read.
Every call returns success and the feature has been giving confidently wrong answers for a day. Which alert catches it first?
Make each call, then see the verdict
Predict first
An upstream model version lands with no code change on your side. The feature keeps answering, and calls keep succeeding. Over the following week, what happens to measured answer quality before anything alerts?
Four incidents, one ladder. Pick the rung that serves each one.
The feature works all morning and starts failing in bursts at peak. The errors say rate limited, and the same request succeeds a few seconds later.
The primary model provider has a full outage midday. Traffic keeps arriving, and the product cannot drop to zero without becoming unusable.
Yesterday a new upstream model version landed with no code change on your side. This morning the scheduled replay of the graded set reads down: answers parse, but one whole request category now comes back wrong.
Both the primary and the fallback model are failing, the retries have burned their attempts, and traffic keeps arriving. The feature is a refund-status answer inside a checkout flow.
How it works
The three habits attach to three different moments. Watching attaches to steady state: uptime and error rates from the ordinary dashboards, a scheduled replay of the graded set for quality - two or three dozen realistic inputs run against the live system and graded, by mechanical checks where the answer is checkable - and latency read at the tail. The tail is where averages lie: ten fast requests and one nine-second one produce a 1.8-second average, and the user who waited nine seconds is the one who wrote the ticket. p95 and p99 exist to stop that arithmetic from lying. Falling back attaches to the failure moment, and its order is a cost ladder, not a preference. A retry with backoff absorbs the blips that heal themselves, and the growing pause keeps the second attempt from landing back in the same crowded second. A cheaper second model absorbs failures specific to one model, since rate limits and outages are per-model. The canned answer absorbs everything else, and its job is honesty: a fixed, clearly-worded response beats an invented one, and it converts an outage from a correctness event into an availability event. Rollback attaches to the change: the graded set that replays on a schedule is the same set that gates changes, so a new model version or prompt edit that moves the grades the wrong way is reverted with the numbers attached. Each rung, said out loud in an interview, names its trigger and its cost - which is the difference between a ladder and a pile of ideas.
The missed delivery
I picture a restaurant counter on the night the main supplier misses the drop, because the ladder people build there is this page's ladder, in the right order. First, call again after a wait: plenty of missed drops are traffic, and the driver arrives twenty minutes late - retry with backoff. Second, cook from the backup freezer: a different supplier's food, a little different from the usual, still a real dish - stepping down to the cheaper model. Third, print the short menu: the card that lives in the drawer for exactly this night, a smaller honest list of what the kitchen can actually serve - the canned answer. The kitchen learned the hard way that improvising dishes from missing ingredients is how a supplier problem becomes a customer problem nobody ordered. And the recipe file on the wall plays the graded set: when a new supplier's flour quietly changed every loaf, the receipts from the good weeks showed which batch broke, and the old supplier came back the same afternoon. That last move is the rollback, and it worked because the receipts existed before the bad delivery.
Where it breaks
The analogy breaks at how quietly the flour changes. A kitchen catches a bad delivery by tasting it the same afternoon, while a model version can land upstream with no code change, no delivery note, and answers that stay fluent - invisible to the status dashboards, caught only by the scheduled graded replay, hours after users did. The kitchen also tastes the ingredient directly; a service watching an AI feature only ever tastes answers, which is why the graded set needs inputs whose right answers somebody wrote down first.
Why interviewers lean on it
Reliability is where AI engineering sounds most like ordinary backend work and then stops, which is the sound interviews test for. The follow-ups run: define p95 in a sentence you own; name what the dashboards saw on the day quality dropped while calls returned success; explain why the cheaper model stands above the canned answer and what each rung costs when it fires. A candidate who has watched these systems answers with the percentile split, the replay cadence, and the rung triggers; one who has not says monitoring and alerting and hopes. The canned rung carries its own product question - what does your user see when everything fails - and the answer is drafted, approved, and versioned, never improvised. The rollback half closes the trust question every interviewer eventually asks: how do you know a change helped, and how fast can you take it back. Those two sentences together - the graded set that gates and the version that reverts - are the difference between operating an AI feature and being operated by one.
The follow-up chain
Interviewers do not stop at the surface answer. Answer the way you would out loud, or skip - skipping reveals the strong answer and ends the chain.
The interviewer's next question
Your AI feature went wrong for two days and no alert fired. The calls all succeeded. What do you add, and why does ordinary monitoring miss this?
Now answer one you have not seen
You own an AI feature with one provider call, no fallback ladder, and no quality instrument. Leadership wants a plan this week. What do you ship first, in what order, and what do you defer?
Expires fast - as of 2026-10-06
As of 2026-10, percentile latency and error-rate alerting are table stakes in backend practice, while scheduled quality replays of a fixed graded set remain the less common half - the gap this concept exists to close.
The cost-ordered fallback ladder (retry with backoff, then a cheaper model, then a fixed honest response) appears consistently in provider reliability guidance and practitioner writeups, with no canonical retry count or pause curve; those stay product decisions.
Version-pinning the model plus eval-gated rollback is the emerging norm in 2026 writeups, mirroring how the inference-economics concept treats upstream version drift as an invoice event: the provider can ship the change, only your gate can return it.