Every team building an LLM product asks the same question sooner or later: where do we get evals? The usual answer is a workshop, a spreadsheet, and a heroic afternoon writing fifty synthetic test cases that resemble what users might ask. They resemble it the way a showroom resembles a road. The real questions are already sitting in your support queue, written by the people the evals are supposed to protect.
Your support tickets are the eval set you already have. Every confused user, every wrong answer someone bothered to report, every "it said this, but I expected that" is a labelled example of your system failing in the wild — with the exact wording and the exact expectation attached. Synthetic evals guess at failure modes. Tickets document them.
How to turn tickets into evals
Start with the last ninety days. Pull every ticket where the user quoted the agent's answer and disagreed with it, or where the agent's answer was correct but the user still couldn't complete their task. Tag each one with what went wrong: wrong fact, missing step, hallucinated policy, misunderstood intent, or correct answer, unusable format. You will find that four or five failure categories cover most of the pain, and that the categories are not the ones your synthetic tests covered.
Then convert the best examples into eval cases. Keep the user's original wording — do not clean it up, because the messiness is the test. Record the expected outcome in plain language: what a good answer had to contain, and what it must never claim. A ticket saying "the agent told me the refund takes 3 days, it's actually 5" becomes an eval that catches any future version claiming 3 days.
The loop that makes it compound
The point is not a one-off harvest. Wire the loop in: every week, the support lead picks the three tickets that hurt most, and those become evals before the next release. Over a quarter, your eval set stops guessing what users ask and becomes a record of what they actually asked. Regressions show up as eval failures before they show up as tickets, because the tickets already taught the evals what failure looks like.
There is a second benefit nobody plans for. When the eval set comes from real tickets, the whole team starts reading support differently — not as a cost centre, but as the cheapest user research the company owns. The tickets stop being complaints and start being requirements, written in the user's own words, with the severity already attached.
We build AI systems that learn from their own failures — evals harvested from real usage, not imagined in a workshop — because the fastest way to make an agent better is to listen to the people it already disappointed, and then make sure it never disappoints them the same way twice.
Field note: A client once spent a month building a 200-case synthetic eval suite for a support agent, and it passed with flying colours — while real users kept hitting the same wall. We pulled their last hundred tickets and found the pattern in an afternoon: users pasted screenshots instead of describing problems, and the agent guessed at the screenshot instead of asking for the text. None of the 200 synthetic cases included a screenshot. We added fifteen ticket-derived cases, the eval caught the failure on the next release, and the fix was a one-line prompt change: ask for the text when the input is an image. The synthetic suite is still there. The tickets do the teaching now.
Related service AI products



