Governance is a build practice
Most AI governance material describes what should be true of a system. Very little describes what a team has to do differently on a Tuesday to make it true.
The gap matters, because almost every governance property — explainability, traceability, oversight, reversibility — is cheap to build in and expensive or impossible to add afterwards. Governance applied at the end is documentation about a system rather than a property of one.
1. The data strategy is the AI project
The most common failure is not a modelling failure. It is that the same customer, account or counterparty appears three ways across four systems, and nobody wanted to own the reconciliation.
No model survives that. Worse, in a regulated setting it produces a specific hazard: a model trained on inconsistently-identified entities will make systematically different decisions about the same person depending on which record it saw, and that is the shape of an outcome you will have to explain.
Do first:
- Entity resolution, with one authoritative identity referenced by everything else.
- Collection fixed at source, not reconciled downstream forever.
- Lineage recorded. For any field feeding a decision, you should be able to say where it came from and when.
- Known-quality assessment. Which fields are reliable, which are not, and which are reliable only for records after a certain date.
2. Decide what the model is allowed to do
Governance conversations go badly when the model’s role is vague. Be explicit, per use case, about which of these applies:
Informing — output shown to a person who decides. Lowest burden, and where most production value sits.
Recommending — output carries a default action a person accepts or rejects. Requires override capture and monitoring, because “accept” becomes the path of least resistance.
Deciding — output takes effect without a person. Highest burden. Requires reversibility, monitoring and a defined route to challenge, and should be reserved for decisions whose consequences are bounded.
Most organisations discover they need far less of the third category than the business case assumed.
3. Explainability is not a post-hoc layer
An explanation produced by a second system is that system’s opinion about the first. The two can disagree, and you will be asked about the disagreement.
The requirement is that the explanation comes from the same machinery as the answer:
- Contributing factors and their weights recorded at inference time.
- Input values as they stood then, not as they stand now.
- The version of the logic that ran, recorded per decision.
- What the person did with the output, and why if they departed from it.
This is why model choice is a governance decision. On structured financial data, the accuracy gap between a directly interpretable model and an opaque one is often small — and the difference in what you can defend is not.
4. Documentation that is worth having
Model documentation tends to be written for an audit and read by nobody. The version that earns its place answers questions the team itself will have in eighteen months:
- What decision does this support, and what happens if it is wrong in each direction?
- What data was it trained on, over what period, and what known biases exist in that period?
- What was deliberately excluded, and why? (Protected characteristics and their proxies belong here explicitly.)
- What performance was accepted at approval, measured how, on which population?
- What are the known failure modes and the conditions under which it should not be used?
- Who owns it, and when is it next reviewed?
Keep it with the code, versioned with the code. Documentation in a separate repository describes a system that no longer exists.
5. Human oversight that is not a rubber stamp
Oversight exists only if the reviewer can see the reasoning, can disagree without friction, and has the time to do so.
Two anti-patterns worth naming:
Volume without capacity. A reviewer given 400 decisions a day approves them. That is not oversight, it is a queue.
Automation complacency. Where the model is usually right, checking degrades. The counter is targeted verification — a sampled subset reviewed properly — rather than nominal review of everything.
Capture every override: who, when, model output, action taken, reason. Review the distribution quarterly. Overrides clustering in one product, region or customer segment are the most informative signal the system produces.
6. Fairness, concretely
Two things are commonly conflated. Excluding protected characteristics from the inputs is necessary and not sufficient, because proxies exist and models find them.
What is required is outcome testing: measure decision rates and error rates across groups, on real production data, on a schedule, and record the result whether or not it is comfortable. Define in advance what threshold triggers action, because a threshold set after seeing the number is not a threshold.
This is also where the data foundation returns. Disparities in outcomes frequently originate in disparities in data quality — a group whose records are less complete receives worse decisions, through no property of the model at all.
7. Monitoring, drift and the kill switch
A model in production is a system that will stop being correct at some point.
- Monitor inputs as well as outputs. A distribution shift in the inputs is an alert in its own right, and usually arrives first.
- Monitor override rates. A rise means the people closest to the work have stopped trusting it, and they are usually right.
- Define the rollback. Which version is reverted to, who authorises it, how long it takes, and what happens to decisions made in the interim.
- Test the rollback. An untested rollback is a plan, not a control.
8. Third-party and foundation models
Where a model is not yours, the governance obligation does not move. It changes shape:
- What can you evidence about training data and evaluation? Usually less than you need — record that gap rather than papering over it.
- Does the provider version the model, and are you notified? A silent upgrade changes your decisions without a change on your side.
- What leaves your estate, and is that compatible with your data protection position?
- What is the fallback if the service is unavailable or withdrawn?
9. What a reviewer actually asks for
In our experience, the requests are specific and practical:
- Show me one decision from six months ago, with the inputs as they stood.
- Show me the version of the logic that produced it.
- Show me who reviewed it and what they did.
- Show me the last outcome-disparity test and what you did about it.
- Show me your monitoring, and the last time it triggered.
If any of those needs an engineer to write a query, the evidence exists in principle and not in practice.
A sequence that works
- Fix identity and collection. Expect this to dominate.
- Build the deterministic version and instrument it fully.
- Introduce the model only where it beats the rules on the same instrumentation.
- Approve it against documentation that will still be true next year.
- Monitor inputs, outputs and overrides; review outcomes on a schedule.
We build this way in safety-critical rail work, where a fatigue score has to be defensible factor by factor — RailGard AI — and across nine regulated operational service areas in ECS Group Command. The regulator differs; the discipline does not.
Putting AI into a regulated process? Get in touch, or read about our financial services work.