← All articles
Production AI3 min read

Stop building dashboards for your AI. Build alerts instead

Dashboards look like observability — until your agent fails at 2 a.m. and nobody finds out until Monday. Alert on output quality, behaviour change, and cost per task instead.

Stop building dashboards for your AI. Build alerts instead

Every team that ships an AI feature builds a dashboard first. Token usage over time. Requests per minute. A latency histogram, a cost chart, a pie chart of model versions. It looks like observability. Then the agent starts failing at 2 a.m. on a Saturday, and nobody finds out until a customer screenshots the damage on Monday morning. The dashboard was green the whole time. It just wasn't watching the right thing.

Dashboards answer questions nobody asks at 2 a.m. They are for the retrospective, not the incident. What you need when an agent goes wrong is not a chart — it is a tap on the shoulder within minutes, with enough context to act on. That is the difference between monitoring and alerting, and most AI products only have the first.

What to alert on

Start with the three failures that cost real money. First, output quality collapse: the share of responses flagged by your evals, your guardrails, or your users as wrong, unsafe, or unhelpful, crossing a threshold you set in advance. A slow rise here is the earliest warning that a model change, a data change, or a prompt edit has quietly broken something.

Second, behaviour change, not just errors. An agent that suddenly starts calling tools twice as often, or stops calling them at all, is telling you something even when every request returns HTTP 200. Alert on the shape of behaviour — tool-call rates, refusal rates, escalation rates — because AI failures rarely arrive as error codes.

Third, cost and latency spikes per task, not per request. A single user request can fan out into dozens of model calls. Aggregate cost per completed task is the number that matters, and a spike there usually means a loop, a retry storm, or a context that grew out of control.

Make the alert actionable

An alert that says "something is wrong" is barely better than no alert. Each one should carry the failing slice — which model, which prompt version, which customer segment, which time window — and a link to the exact traces. The person paged should be able to tell in sixty seconds whether to roll back a prompt, disable a tool, or escalate to a human. If they can't, the alert is noise, and noise gets muted, and muted alerts are how a bad Saturday becomes a bad quarter.

Set thresholds from your baselines, not from vibes. Run the agent for two weeks, measure the normal range of each signal, and alert on sustained deviation — not on a single bad response. A single bad response is a Tuesday. A sustained shift is an incident.

We build AI systems that tell you when they're failing — with quality alerts, behaviour monitoring, and rollback paths — because an agent nobody watches is an agent that fails quietly, and quiet failures are the expensive kind.

Field note: A client once showed us a beautiful observability dashboard with fourteen panels. The same week, their agent had spent three days confidently giving customers a discontinued product's pricing. No panel had flagged it — the requests succeeded, the latency was fine, the cost was normal. We added one alert: the share of responses mentioning the discontinued product name. It fired within an hour of deployment. The dashboard is still there. Nobody looks at it anymore. The alerts do the watching.

Related service AI products

Building something worth shipping?

We take on a small number of AI product engagements. Tell us what you are building — we reply quickly.