There is a familiar shape to a stalled AI project. The model was fine. The demo went well. Then it met the actual data, and the accuracy nobody could reproduce turned out to depend on a hand-cleaned extract somebody prepared once and cannot prepare again.
The uncomfortable part is that this is rarely a modelling failure. It is a data failure that arrived wearing modelling clothes.
The same person, three different ways
Almost every organisation of any size has the same underlying condition: an entity that matters — a customer, an asset, a member of staff — exists in several systems, recorded slightly differently in each, with no agreement about which is authoritative.
Nobody designed it that way. It accumulates through acquisitions, system migrations that were never finished, a spreadsheet that became load-bearing, and the entirely reasonable decision, ten years ago, to get something working quickly.
The result is that a question as simple as how many customers do we have has several defensible answers, and the difference between them is not noise. It is the project.
What that means for a model
A model trained on inconsistent entities learns the inconsistency. Worse, it learns it silently:
- Duplicates inflate confidence. The same underlying case appearing three times looks like corroboration.
- Missing data is rarely missing at random. It usually correlates with something — a channel, a region, a period when a system was down — and the model will find that correlation.
- Definitions drift over time. A field that meant one thing before a migration and another after it produces a model that is quietly wrong on one side of the boundary.
None of these show up as errors. They show up as a system that is confidently wrong in production, which is considerably more expensive than one that is obviously broken.
Fix collection, not just the extract
The instinct is to clean the data before training. That works once. The pipeline then decays, because the source keeps producing the same problem and somebody has to keep cleaning it.
The durable fix is further upstream:
- Decide what the authoritative record is for each entity that matters, and make everything else defer to it.
- Fix collection at source. A field that is optional, free-text and unvalidated will contain everything eventually.
- Reconcile once, deliberately, with a record of the rules used, rather than repeatedly and informally.
- Capture the event, not just the state. Knowing something changed, and when, is usually worth more than the current value.
This is unglamorous work and it is very hard to get funded on its own. It is also the difference between a model that survives contact with production and one that does not.
The order of operations
We put the data strategy first, before any model, for a straightforward reason: it determines whether the rest is possible. In practice that means:
- Find where the records actually live — not where the architecture diagram says they live.
- Agree the entities and the authoritative sources.
- Fix collection, so the problem stops being regenerated.
- Then ask what is worth predicting.
Doing step four first is common, and it is why so many projects produce a convincing pilot and nothing else.
A test worth running
Take one number your business acts on — exposure, utilisation, churn, risk — and ask two teams to produce it independently.
If the answers differ, the gap is not a reporting problem to be reconciled in a meeting. It is the specification for the work that has to happen before any model is worth training.
We do this work before the modelling rather than around it. See how we work, or read about AI in regulated environments.