Designing an ISO-ready AI system: what to prepare before development starts

The moment an AI project discovers it has to answer to ISO/IEC 42001 almost always arrives too late, usually once the system is already in staging and somebody asks whether a risk assessment was ever done. What follows is two or three weeks of retroactive reconstruction, with a team trying to document decisions made months earlier and a launch date slipping.

The awkward part is that some of that work cannot be recovered at all, however long anyone spends on it.

What cannot be reconstructed after the fact

Policies, procedures and user manuals can be written at the end of a project. They describe a state, and a state is verifiable at any point. Four things, by contrast, exist only if they were produced while the system was being built.

The first is the provenance of the data used to train or tune the model. If nobody tracked where each dataset came from, on what basis and with what usage rights, six months later that chain cannot be reconstructed, only asserted. An auditor knows the difference.

The second is the alternatives that were rejected. The question of why this architecture, this model, this vendor was chosen has a credible answer only if the options were on the table at the same time. Reconstructing it later produces a justification, which is a different thing.

The third is the baseline of the process before the system was introduced. How long it took, how many errors it produced, what it cost. Without that number measured up front, later monitoring has nothing to compare against and nobody can say whether the system is improving the situation or degrading it. It's also why so many AI projects can't demonstrate their own return: nobody recorded the starting point.

The fourth is the acceptance criteria. Deciding what has to be true for the system to count as successful, before seeing the results, is one thing. Deciding it afterwards, looking at the metrics that came out, is an exercise that accommodates any outcome.

Technical decisions with compliance consequences

Some choices that look purely architectural in a project plan determine whether the system can be governed at all. Changing them later means going back into the code, not into the documentation.

Logging. Article 12 of the AI Act requires high-risk systems to technically allow the automatic recording of events over the lifetime of the system, and Article 26(6) requires the deployer to retain those logs for at least six months. The operational question, though, isn't switching logging on, it's deciding what gets recorded. A log holding only the output won't let anyone reconstruct a single decision months later. That needs the input received, the model version live at that moment, the outcome produced, and any human intervention along with who made it. Adding those fields afterwards touches the data schema, often latency, sometimes the whole pipeline.

Human oversight. Article 14 requires high-risk systems to be designed so people can effectively oversee them. In practice that comes down to three design questions: who can stop the system, how quickly, and with what information in front of them at the moment they have to decide. An operator who sees only the outcome, without the elements that produced it, is in no position to challenge it, and Article 13 on transparency toward those using the system exists precisely because oversight without understanding stays formal.

The level of explainability. How well the system has to justify its outputs depends on what it decides and who it affects. This constrains the architecture: choosing a non-interpretable model for a use case that requires a reason is a decision no amount of later documentation undoes.

The fallback path. What happens when the system is unavailable, or produces an outcome the operator rejects. If no workable manual path exists, the system effectively cannot be switched off, and a system that cannot be switched off is a risk no control compensates for.

When a technical choice changes your legal role

In the piece on the inventory we covered how Article 25 of the AI Act sets out three situations where an entity using a system becomes its provider, taking on the obligations under Article 16. Two of those three arise from decisions made at design time.

Fine-tuning on proprietary data and changing a decision threshold can both fall within the notion of substantial modification. Changing the purpose of a system not classified as high-risk so that it becomes high-risk has the same effect. The reclassification requires no notification or registration, it happens when the conditions are met.

The practical consequence is that "let's start from the vendor's model and adapt it to our data" isn't only an implementation choice. It needs making with an understanding of what follows, and the phase to raise the question is design, when changing course costs one meeting.

The questions to ask, and what to do with a vague answer

A list of questions is worth little without a way to recognise an insufficient answer. These six each carry an exit criterion.

• Is there a non-AI alternative, and why was it rejected? If nobody considered the alternative, the project doesn't have a case yet, it has a hypothesis. This question stops more projects than all the others combined, which is why it goes first.
• What decision does the system produce, and who does it affect? The answer drives the risk classification and therefore everything downstream. If the answer is "it supports the operator", ask how often the operator departs from the suggestion: below a certain rate, the system is the one deciding.
• Who owns it in production? A name is needed, not a department. If nobody accepts the role before release, nobody will accept it afterwards.
• What is the current baseline, measured? Without a number collected beforehand, later monitoring has nothing to compare against.
• What has to be true for the system to count as successful? Written down in advance, with numeric thresholds, otherwise a metric confirming the decision already taken will always be found downstream.
• What happens when it gets something wrong? Who notices, how quickly, and what they do. If the answer is that the end customer will notice, the project has a design problem, not a documentation problem.

The gates, and what clears them

Few checkpoints are needed, but each needs explicit exit criteria, otherwise they turn into meetings that acknowledge decisions already made.

At kickoff, the first two questions above get answered and a position is taken on the expected risk classification and on the role the organisation will assume. It's the cheapest gate and the one skipped most often.

At design review, the choices with compliance consequences get checked: what the logs record, how oversight is designed, what level of explainability is required, what fallback path exists. Leaving this gate with any of those open means it will be closed badly and in a hurry later on.

Before production, what gets checked is evidence rather than intent: the measured baseline, logs active and verified against one real case reconstructed end to end, a named owner, an open row in the AI system inventory, and a monitoring plan with its thresholds.

Thirty or ninety days after launch, actual behaviour gets compared against the baseline and the acceptance criteria set at the start. This is the gate almost nobody puts in the calendar, and it's the one that says whether the system is doing what it was built for.

If you're not the one building it

Most organisations don't develop from scratch, they buy or integrate. The same questions apply, but they need putting to the vendor during selection, because once the contract is signed the negotiating leverage is gone.

Five things are worth asking for in a tender: the system's technical documentation, the role the vendor states it takes on under the AI Act, which logs the system produces and in what format they're accessible, what validation testing was carried out and with what results, and how much notice is given before the underlying model changes. A vendor that won't answer these during a tender won't answer afterwards either, and that silence is itself useful input to the decision.

Why a policy isn't enough here

Across much of ISO/IEC 42001's Annex A, a written and approved procedure counts as sufficient evidence. Across area A.6, the one covering the AI system life cycle, it doesn't.

The controls in that area deal with documenting design choices, verification and validation, deployment, operational monitoring and event logging. For each, an auditor asks for the record the system produced while running, not the document describing how it ought to run. A test plan counts if the test results exist. A release procedure counts if there's a trace of the checks performed before the last release.

That distinction changes what has to be built, and above all when. Evidence in this area is only produced while the system is being developed and used, which brings the argument back to where it started.

The most frequent mistakes

The most common of all is the proof of concept that slides into production without passing any gate, because it worked and there was pressure. It begins life as an experiment, therefore without documentation, and ends up handling real cases with a level of control designed for a demo.

The second is nominal human oversight, the operator approving three hundred cases a day who in practice is rubber-stamping. Formally the control exists, substantively the system decides on its own, and the gap surfaces at the first incident.

The third is documentation written for the auditor rather than for whoever will maintain the system. It's recognisable because nobody on the team ever opens it, and because it goes stale without anyone noticing.

How this connects to the rest of the series

Designing with the knowledge that the system will have to answer these questions isn't extra work, it's the same work in a different and cheaper order. The information is needed either way: gathered during, it costs hours; reconstructed afterwards, it costs weeks; and in four cases it can't be gathered at all.

The next piece in this series leaves technical documentation behind and turns to people, with what the standard requires on competence and why it's as auditable a requirement as any risk register.