The question arrives eventually, and it is always some version of the same thing: why did it say that?
It might come from a regulator, an auditor, a board member, or the customer on the wrong end of the answer. What they want is not a description of the architecture. They want to know which facts about this particular case produced this particular result.
Systems that cannot answer that get switched off, regardless of how accurate they were.
Post-hoc explanation is not the same as an audit trail
There is a well-established set of techniques for interrogating a model after the fact and producing an account of which inputs mattered. They are useful, and they are not the same thing as a record.
An after-the-fact explanation tells you what the model would weigh, given the model as it stands today. An audit trail tells you what actually happened: the inputs as they were at the time, the version of the logic that ran, the output, and who saw it.
The distinction matters because the model changes. If the explanation is regenerated later, against a model that has since been retrained, it is an explanation of a decision nobody made.
Log the contributing factors, not just the answer
The practical version of this is much less exotic than it sounds. When a score is produced, record alongside it:
- Each input value as it was at the time, not a reference to a record that will later be updated.
- The contribution of each factor, in whatever terms the model supports.
- The version of the logic, the model and the configuration.
- The outcome, including anything a human did with it afterwards.
That record is what turns “the system calculated it” into an answer. It also makes the thing debuggable, which is a benefit nobody sells but everybody wants by month three.
Prefer models you can account for
There is a real trade-off between accuracy and accountability, and in regulated work it usually points one way. A simpler model whose behaviour can be explained line by line is frequently more valuable than a marginally better one that cannot be defended.
This is not an argument against sophisticated methods. It is an argument for being honest about where they belong: a complex model is easier to justify when it informs a human decision than when it makes one.
Where we have built scoring into safety-critical work, the design constraint was explicitly that a person should be able to disagree with the result and see exactly why it said what it said. RailGard AI assesses fatigue risk against the expectations of the ORR, HSE and Network Rail, weighing time of day, shift length, cumulative hours, recovery time and night working — with every contributing factor recorded, so a score can be taken apart afterwards rather than taken on trust.
Human oversight has to be real
“A human is in the loop” is easy to claim and easy to make meaningless. It only counts if the person:
- can see the reasoning, not just the answer;
- has the time and standing to overrule it;
- and leaves a record when they do.
If overruling the system is inconvenient, socially awkward or invisible, the oversight is decorative — and that will become apparent at exactly the wrong moment.
Build it in at the start
None of this is difficult if it is designed in. All of it is difficult to retrofit, because the information you need was not captured at the time and cannot be reconstructed afterwards.
The test is simple enough: pick a decision from six months ago and try to explain it. If that requires anybody’s memory, the explanation does not exist.
Putting a model somewhere it will be challenged? Get in touch, or read more about AI in regulated environments.