We lose deals to competitors regularly, and sometimes the client comes back six months later. The stories are consistent enough that we've learned to spot the agencies that generate them. Here's what to watch for — from inside the industry.
Flag one: they lead with the model, not the problem. If the first conversation is about which LLM they'll use rather than what outcome you need, they're selling technology, not results. The model is an implementation detail; any agency that treats it as the headline hasn't thought about your problem yet.
Flag two: no talk of evals. Ask "how will we know it's working?" If the answer is vibes — a demo, stakeholder feelings — walk away. Serious AI work defines measurable success criteria before building, because without them the project can never be finished, only abandoned.
Flag three: hourly billing with open-ended discovery. AI projects have genuine uncertainty, and some agencies monetize that uncertainty instead of managing it. Look for fixed-scope milestones where the risky assumptions get attacked first. An agency confident in its process will price the outcome, not the clock.
Flag four: they can't show you something running. Not slides, not mockups — a live system with real users. AI demos are cheap to fake and expensive to confuse with products. Ask what broke in production and how they fixed it; the answer tells you everything about whether they've actually shipped.
Flag five: no opinion on data. If they don't interrogate your data quality early, they haven't learned the field's hardest lesson: models are rarely the bottleneck, data is. An agency that nods along to "our data is fine" is planning to discover otherwise on your budget.
The best hiring test we know: ask them what they'd refuse to build. A good agency has a list. Ours starts with "anything we can't eval" — and we'd rather lose a deal than ship something we can't stand behind.
Field note: Contrast two first calls we've actually seen. Agency A spent forty minutes on model comparisons and named three LLMs. Agency B spent forty minutes on the client's invoice workflow, asked to see the actual documents, and asked what "good" would look like in numbers. Agency A got the project; six months later the client came to us with a demo that couldn't handle their real PDFs. The first call predicts the engagement: if they start with your problem, they'll solve your problem. If they start with their technology, you'll get their technology — applied to something adjacent to your problem.
