The model is rarely why the project fails
Every failed AI project has a moment where someone says "the model isn't smart enough," and that sentence is often wrong. The model is fine. The data is a mess — duplicates, stale records, contradictory entries, critical context locked in someone's head — and no amount of prompt engineering fixes that.
Audit a hundred real records before promising a build
We've learned to audit data before promising anything. The audit is deliberately low-tech: sample 100 real records, try to answer the target questions from them manually, and count how often the data lets you down. If a human with full context can't get reliable answers from the data, the model won't either. This one exercise has killed more bad projects in week one than any technical spike — which is exactly the point. Better a dead project in week one than a dead project in month six.
The usual failures are ownership, labels, and coverage
The common data pathologies: recency without versioning (the "current" price list is actually three conflicting spreadsheets), implicit knowledge (the field everyone knows means something different than its label), and coverage gaps (the system works for the common case because the training examples are all common cases). Each has a fix, but the fixes are organizational, not technical — someone has to decide which spreadsheet is canonical, document what the field means, collect the edge cases.
Here's the part vendors won't tell you: data cleanup is the actual project, and the AI is the easy part at the end. Budget accordingly. We typically spend the first phase of an engagement on data readiness — not because we love ETL, but because everything downstream depends on it.
Known gaps beat data that was only assumed clean
The good news: you don't need perfect data. You need known data — documented gaps, measured quality, clear ownership. An AI built on honestly-assessed data outperforms one built on assumed-clean data every time.
Data readiness is where our AI engagements start — because the fastest way to a working AI product is refusing to build on a foundation you haven't inspected.
Field note
The three-spreadsheet price list is real: sales maintained one, finance another, and the "official" one was six months stale. The AI quoting tool built on top quoted confidently from whichever file it found first. The 100-record audit caught it in week one — a human trying to answer "what's the current price?" from the data failed 34 times out of 100. Fixing it took two workshops and one executive decision about canonical ownership. Unglamorous, organizational, and worth more than any model upgrade: quote accuracy went from "untrustworthy" to 99%+ on the eval set, because the data finally meant one thing.
Related service AI products
