Why this is a different problem
Putting a model into a marketing function is a commercial decision. Putting one into a safety-critical operation is a regulatory one, and the difference is not the accuracy required — it is that someone outside your organisation may need to understand, months later, how a specific output was produced and what was done about it.
That single requirement changes almost every design decision. It is also, in our experience, the requirement that gets discovered last.
This paper sets out what we have found necessary in practice, based on building RailGard AI — a fatigue risk platform scoring every shift against the expectations of the ORR, HSE and Network Rail — and ECS Group Command, which carries audit trails and certification evidence across nine operational service areas.
1. The data foundation decides the ceiling
Almost every stalled AI programme we have seen stalled below the model, in the data. The pattern is consistent: the same person, asset or site appears three different ways across four systems, and no amount of model selection survives it.
Before any modelling work is worth starting:
- Entity resolution. One authoritative identity for each person, asset and location, with the other systems referencing it rather than redefining it.
- Collection fixed at source. If a field is unreliable where it is captured, it is unreliable everywhere downstream, forever. Reconciliation after the fact is a treadmill.
- Historical outcomes recorded honestly. A model learns from what happened. If failures were closed as “repaired” with no cause, or incidents were recorded inconsistently, there is nothing to learn from.
- Timestamps that mean one thing. In operational data, when something was recorded and when it happened are routinely different, and confusing them produces models that appear to predict the past.
This work is unglamorous, is usually the majority of the effort, and is the part most likely to be cut when a programme is under pressure. Cutting it does not save time; it moves the failure later.
2. Score, do not classify
In a regulated setting, a binary output is hard to defend and hard to act on. “High risk” invites the question “compared with what?” and offers no route to proportionate action.
A score with published factors is more useful and more defensible. For fatigue, that means time of day, shift length, weekly hours, recovery time, night working and consecutive duties, each contributing a visible amount. The person receiving it can see not just the number but which part of the roster drove it — and therefore what to change.
The design rule: every output should carry the inputs that produced it and their individual contributions. Not an explanation generated after the fact, but the actual machinery.
3. Explainability comes from the same machinery as the answer
A post-hoc explanation is a second model’s opinion about the first model. In a regulated environment that is a liability, because the two can disagree and you will be asked about it.
Practical consequences:
- Prefer model families whose contributions are directly readable where the accuracy difference is small — which, on structured operational data, it usually is.
- Record the contributing factors at inference time, not on request. You cannot reconstruct them later if the inputs have changed.
- Version the logic and record which version ran. “The rules changed in March” must be answerable per record.
- Keep the input values as they stood at the time. A record that reads today’s roster to explain a score from June is wrong and confidently so.
4. Human oversight has to be real
Every framework requires meaningful human oversight. In practice, oversight only exists if three things are true:
The person can see why. Oversight of a number nobody can interrogate is a rubber stamp with extra steps.
The person can disagree, and it is recorded. Overrides are the most valuable signal an operational model produces about itself. If disagreement happens in a phone call, you have neither the improvement loop nor the evidence.
Disagreeing is not punished. If overriding the model creates paperwork or suspicion, people stop doing it, and you lose the control you claimed to have.
Record, for each override: who, when, what the model said, what they did instead, and why. Review the pattern quarterly. A cluster of overrides in one depot or one shift type is telling you something the model does not know.
5. The audit trail, specifically
Most systems log changes. The questions that actually arrive are different:
| What gets asked | What a change log holds |
|---|---|
| What did the system know at the time? | The current values |
| Which version of the logic applied? | Nothing |
| Was this automatic or chosen? | A user ID, sometimes a service account |
| Who could see this, and did they? | Nothing — reads leave no trace |
| Why was the model overridden? | Nothing |
Build the trail to answer the first column. That means capturing inference inputs and outputs, logic version, the action taken, and read access on sensitive records.
6. Failure modes worth designing against
Alert fatigue. A system that flags too much gets muted, and the muting is permanent — a second attempt inherits the reputation of the first. Start conservative. Trust is built by being right about small things.
Drift. Operational conditions change: new rosters, new sites, a new client with different patterns. Monitor input distributions as well as outputs, and treat a distribution shift as an alert in its own right.
Automation complacency. Where a model is usually right, people stop checking. This is a known and well-documented risk in safety-critical settings, and it argues for occasional deliberate verification rather than for removing the human.
The convenient silence. Doing this properly will sometimes produce evidence that a compliant arrangement is a risky one. Some organisations conclude it is safer not to look. That reasoning has never held up: the pattern is reconstructible afterwards from records you were already required to keep. The only question is whether you saw it in time.
7. A workable sequence
- Establish the record. Entity resolution and collection at source. Expect this to be most of the programme.
- Build the deterministic version first. A transparent rules-based score, agreed with the people who will be held to it. It is often good enough, and it creates the evidence trail everything else needs.
- Instrument it. Inputs, outputs, versions, actions and overrides captured from day one, before anyone needs them.
- Introduce the model where it demonstrably beats the rules, on the same instrumentation, with the same explanation surface.
- Review the overrides, not just the accuracy. That is where the operational truth is.
What to ask a supplier
- Can you show me the contributing factors for a single output from six months ago?
- Which version of the logic produced it, and where is that recorded?
- What happens to that record when the underlying data is corrected?
- How is an override captured, and who reviews the pattern?
- What is monitored to detect drift, and what does it trigger?
If those answers require an engineer to go and query something manually, the evidence exists in principle and not in practice — and that distinction is the whole subject of this paper.
We build systems that show their working. Get in touch, or read more about our rail and infrastructure work.