Latency decides whether people keep the feature
Nobody puts "fast" in the AI pitch deck, and then users abandon the feature because it takes eleven seconds to answer. Latency is the silent killer of AI products: invisible in demos, decisive in production.
Set a budget before the architecture is fixed
Set a latency budget before you build, the same way you'd set any performance SLO. What's acceptable depends on the interaction: under 2 seconds for inline assistance, under 5 for a generated draft, longer only for explicitly asynchronous work ("we'll email you the report"). Then design the architecture to hit the budget, because retrofitting speed into a slow pipeline is miserable.
Stream, trim, route, cache, and parallelize
The toolkit is well-known but rarely applied systematically. Stream tokens instead of waiting for the full response — perceived latency drops enormously when users see words appearing immediately. Shrink the context: every thousand tokens of retrieved context is latency you're paying on every request, so rerank-then-trim beats stuff-then-pray. Route easy requests to smaller, faster models. Cache aggressively: a surprising share of production queries are repeats. And parallelize independent calls instead of chaining them — agent pipelines that run three tool calls sequentially when they could run concurrently are leaving seconds on the table.
The streaming point deserves emphasis because it's mostly a UX decision with architectural consequences. To stream well you need the UI to render partial output gracefully, the pipeline to yield incrementally, and error handling that works mid-stream. Teams that bolt streaming on at the end get flickering interfaces and broken states.
Watch percentiles on the same dashboard as quality
Measure p95 and p99, not averages. Your average response might be 3 seconds while one user in twenty waits 25 — and that user is the one writing the review. Latency percentiles belong on the same dashboard as your quality evals.
Speed is a feature users feel in every interaction. We treat it as one in every AI system we ship — budgeted upfront, engineered in, monitored continuously.
Field note
Streaming cut perceived latency on a client's drafting feature from 9 seconds to under 1 — same total generation time, radically different feel. But it required real work: the UI renders partial markdown gracefully, the pipeline yields deltas instead of buffering, and errors mid-stream produce a clean "interrupted, retry" state instead of a half-rendered corpse. On measurement: their average was 3.1s, comfortably inside budget — but p99 was 24s, driven by a retrieval path that occasionally scanned the full corpus. The average said "fine"; the percentile said "one user in a hundred is furious." Budget the percentiles, not the mean.
Related service AI products
