The most dangerous moment in an agent's day is not when it fails. It is when it succeeds at the wrong thing, confidently, without asking anyone. An agent that refunds the wrong order or emails the wrong client has not malfunctioned. It has done exactly what it was built to do — act on its best guess — in a situation where a guess was never good enough.
Most teams discover this backwards. They ship an agent, watch it handle the easy cases beautifully, then widen its permissions until one day it takes an irreversible action on ambiguous input. The post-mortem always finds the same thing: the agent had no line between "I know what to do" and "I should check first." Nobody drew it.
Draw the line with reversibility, not confidence. Confidence scores lie — models are most certain exactly when they are wrong in a plausible way. Reversibility doesn't lie. If the action can be undone cheaply — drafting a reply, looking up a record — let the agent do it. If it cannot be undone cheaply — sending money, deleting data, telling a customer something binding — the agent drafts and a human decides.
Then name the askable moments explicitly. "Unclear customer intent" is too vague to implement. "Two orders match this customer," "the refund exceeds $50," "the request mentions a contract term the agent cannot see" — those are checkable conditions. Write them as a list, in the prompt and in the evals. Every time the agent guesses through one of those moments in testing, add it to the list. The list is the product of every near-miss you caught, and it compounds.
The handoff itself needs designing too. "Escalate to a human" is not a design. Who gets it, with what context, and how fast? An escalation that arrives as a bare "agent unsure, please review" with no transcript and no suggested action will be ignored or rubber-stamped — both worse than the agent guessing. The escalation should carry the full picture: what the customer asked, what the agent found, what it was about to do, and why it stopped. A human who can decide in thirty seconds will decide.
There is a cost to asking, and teams feel it: every escalation is a case the agent didn't close. But the right comparison is not escalations versus autonomous resolutions. It is escalations versus the cost of the one wrong irreversible action that erases a month of good resolutions. Measure the escalation rate, tune it down as the list sharpens, and treat a falling rate as the sign the agent is learning — not as a reason to remove the line.
We build AI agents that know when to stop and ask — reversible by default, explicit escalation rules, and handoffs a human can actually decide on — because the agent that asks once at the right moment is worth more than the one that acts a hundred times and gets one of them catastrophically wrong.
Field note: A client asked us why their support agent kept escalating "simple" refund requests. The logs showed it stopping whenever two orders shared a surname. The team called it a bug and wanted the check removed. We kept it — and found that one in twenty of those cases really was the wrong order, usually a gift recipient with the same last name. The fix was not fewer questions. It was a better question: the agent started asking the customer which order, with both listed, instead of escalating to staff.
Related service Workflow automation



