Every organisation that takes governance seriously ends up with a framework for it. A way of making architectural decisions, a review board, a set of standards, a change process, a professional code the practitioners sign up to. The whole apparatus exists to answer one question: how do we make sure the things we build are the things we should have built?
So when AI arrives, the instinct is comforting and almost universal: we already have a governance framework — we’ll add AI to it. Fold it into the review board’s remit, tick it on the change form, mention it in the standards.
You can’t, though. Not because the framework is bad, but because it’s silent in exactly the places AI is loudest. The companion to this post argued that using AI well is a personal discipline — keep the judgement, don’t outsource it. This is the organisational version of the same problem, and it’s harder, because a discipline you can hold in your own head doesn’t scale to a whole team. For that you need governance — and the governance you have has a hole in it.
What the old frameworks quietly assume
Traditional governance — enterprise-architecture review, change control, the professional codes — governs how decisions get made and how change is run. It’s very good at that. But it was built for a world of conventional software, and it quietly assumes three things about the systems it’s governing.
- Determinism. The same input produces the same output. You can test the thing once, sign it off, and trust it to keep behaving.
- Explainability. When it does something, you can say why — trace it to a line of code, a rule, a configuration.
- A known, static risk surface. The ways it can go wrong are enumerable at the gate, and they don’t shift once it’s live.
A model breaks all three. Its outputs are probabilistic, not deterministic — the same prompt can give you different answers, and “we tested it” no longer means “it will do that again.” It’s opaque: a model that declines a customer often can’t tell you which factor tipped it, only that the number came out below a threshold. And its risk surface is neither known nor static — it drifts as the world the model was trained on moves out from under it. These aren’t my framing; they’re the properties that governance bodies themselves now list as the reason AI needs its own treatment: opacity, autonomy, scale, data dependency, and probabilistic rather than deterministic behaviour.
So the framework you already trust wasn’t wrong. It was built for a world the model doesn’t live in.
The gap is wider than bias
The obvious hole is fairness. A model trained on biased data will launder that bias into a number that looks objective — I’ve argued before that this is designed in at the whiteboard, not acted out later, and I won’t go over that ground again here. But fairness is only the most-discussed part of the gap. There are two others your existing governance barely touches.
The first is security, and it’s a genuinely new surface. A language model wired into your systems can be talked into things by its own input — the industry’s working catalogue of these risks reads nothing like a traditional threat model: prompt injection, leaking data it was given in confidence, being manipulated into taking actions it was never meant to take. Your penetration test doesn’t cover the case where the attacker simply asks the system nicely.
The second is the lifecycle. Conventional governance checks the thing once, at the gate, and moves on. A model is never done at go-live: it needs continuous monitoring, periodic red-teaming, retraining as it drifts, and a documented record of what it was trained on and how it behaves — the “model card” that didn’t exist as a concept when your change process was written. A review board built to approve a release has no mechanism to ask “and who is watching this in March, when the data has moved and nobody has looked?”
Regulators are already in the gap
The people least surprised by all this are the regulators. Ahead of comprehensive legislation, they’ve been moving into the gap from two directions at once. Some are writing AI-specific law that classifies systems by risk — the EU’s AI Act, the most developed example, tiers them from prohibited, through high-risk, down to limited and minimal, with obligations that scale accordingly. Others have pointedly declined to write new AI-specific rules. The UK’s pro-innovation approach leaves AI to the regulators that already exist, and that puts the onus squarely on the firm: in financial services the FCA points to Consumer Duty, the Senior Managers regime and existing operational-resilience rules (the same resilience expectations the EU has codified in DORA). Nobody is going to hand you an AI rulebook. You’re expected to work out what your existing obligations mean for a model, and to show your working.
The EU’s version also hands out roles, and they’re easy to skim past. The Act names a chain of operators: the provider, who builds a system or puts it on the market under their own name; the importer, who brings one in from outside the EU; the distributor, who passes it along the supply chain; and the deployer, who uses it. Each role carries its own obligations, and the provider’s are by far the heaviest. Most organisations assume they’re deployers — they bought the model, they didn’t build it. But the roles aren’t fixed. Put your own name on a high-risk system, modify one substantially, or repurpose a system so that it becomes high-risk, and a deployer, importer or distributor can find it has become the provider in the Act’s eyes, obligations and all. Nothing in a conventional change process asks which role a change leaves you in.
Whichever route a regulator takes, the direction of travel is unmistakable, and it lands on the same principle a courtroom reached years ago: “the algorithm did it” has never been a defence, and “the model did it” won’t be either.
What’s emerging to fill it
The good news is that you don’t have to invent the response from scratch. A small stack of public frameworks now exists specifically for this: the OECD’s principles for trustworthy AI, the US NIST AI Risk Management Framework — organised around four plain-spoken functions, govern, map, measure and manage — and ISO/IEC 42001, the first certifiable standard for an AI management system, built like the quality and security management standards before it.
They’re serious, well-made, and worth reading. They’re also heavy — designed for organisations with the scale to stand up a management system and staff it. Which is where most of the honest difficulty lives.
The proportionate version
This blog has made this argument before in a different costume: a heavyweight framework imposed on an organisation too small or too immature to carry it doesn’t produce governance — it produces shelfware and theatre, and it teaches everyone that the process is something to route around. A sole architect cannot operationalise a full NIST programme, and pretending otherwise is how you end up with an impressive policy nobody follows.
The frameworks themselves concede the point — they say governance should scale to an organisation’s size, maturity and risk tolerance. So the proportionate move isn’t to build a parallel AI-governance bureaucracy. It’s to borrow the structure of those frameworks without the ceremony, and embed it in the architecture tools you already own. What that looks like at a size a real team can lift is a post of its own. The point here is narrower: you already own the tools. What’s missing is pointing them at a target the frameworks didn’t tell you to aim at.
Why this lands on the architect
It lands here because it sits on the seam the architect already stands on — between the technology and the business, between “it works in the demo” and “it’s defensible when it doesn’t.” The data scientist can tell you the model’s accuracy. The compliance team can tell you the rule. But only the architect is positioned to decide how the whole thing is structured so that the fair, explainable, monitored path is the default rather than the thing someone has to fight for — and to write down why while that choice is still cheap.
That’s the same move as the personal one in the companion post, scaled up. Individually, the temptation is to let the tool do your thinking. Organisationally, the temptation is to let the framework do your governing — to assume that because you have a process, you’ve covered this. You haven’t. The frameworks you already trust won’t cover AI for you, and the regulators are telling you as much.
The gap doesn’t close by waiting for the standard to mature, the regulator to prescribe, or the vendor to hand you a certificate. It closes when the architect decides — now, while it’s cheap — what “good” looks like for this organisation at this size, and writes it down where the next person will find it. That the frameworks don’t cover it yet isn’t a reason to wait.
It’s the job.