AI governance is not paperwork

There is a version of AI governance that everybody recognises and nobody respects. A policy document lands in the intranet, a mandatory training module gets assigned, a form appears in the procurement process, and the organisation declares itself governed.

Six months later a team is running a model that nobody in the risk function has heard of, connected to a data source nobody assessed, producing outputs that feed a decision about a real person.

The policy was never wrong. It was just never load-bearing.

The test that actually matters

I have started asking one question when people tell me they have AI governance in place. Not whether they have a policy. Whether they can answer this, about any specific system, in under a day:

Who approved this, on what evidence, and what happens when it is wrong?

It is a deliberately unglamorous question. It is also the question an auditor, a regulator, a journalist or a plaintiff's lawyer will ask, and the one that separates governance from documentation.

Most organisations can produce the policy. Far fewer can produce the answer.

A control you cannot evidence is not a control

The most common failure I see is treating a vendor claim as a control.

A vendor states in a sales deck that customer data is not used for training. Someone writes that down in the risk assessment. The risk is marked as mitigated. Nobody records where the claim came from, when it was made, or what happens if the vendor changes the position in a product update six months from now.

That is not a control. It is a note.

Every third-party AI assessment I sign is grounded exclusively in the vendor's own published documentation, cited and dated. Not because the sales team is lying, but because a claim you cannot re-check has a shelf life you cannot measure. When a vendor quietly updates a data-handling page, the assessment should break loudly rather than remain technically true and practically obsolete.

The same logic applies internally. If a safety measure can be removed by a customer with API access and a modest dataset, it was never a control. It was a default. That distinction is not academic, and it is exactly what security researchers demonstrated when they showed that fine-tuning access could be used to work around a frontier model's safety behaviour.

The register that does not match reality

The second failure is quieter and more widespread.

Most organisations maintain an application register for cyber and AI risk. Most organisations also have staff signing up to free tools with a work email, building agents in low-code platforms, and connecting things to the tenant that will never appear in that register.

So the register is accurate and useless at the same time. It is an accurate account of what was declared. It is useless as an account of what is running.

This is why discovery has to be automatic. A register maintained by asking people to self-report their AI use is a survey, not an inventory. The frameworks agree on this point even where they differ elsewhere. NIST's AI RMF requires organisations to maintain an inventory of their AI systems with a named individual or team responsible for keeping it current, and that ownership requirement is the part most often skipped.

Governance is a design-time activity

The third failure is timing.

Retrofitting governance onto a live system leaves you two options: switch it off, or accept the risk. Neither is a decision anyone wants to take to an executive, so in practice the risk gets accepted with a note about future remediation that nobody schedules.

Constraints belong in the architecture review, not the incident review. That means the governance question is asked when someone proposes the system, not when someone notices it.

The Australian public sector has landed in the same place. The Digital Transformation Agency's pilot framework centres on an AI impact assessment tool designed to help agencies identify, assess and manage potential AI use case impacts and risks, and the reasoning is explicit: AI systems operate with elevated complexity, speed and scale which can amplify potential harmful outcomes in ways that existing technology governance frameworks and processes may not fully address.

Note what that says. Not that AI creates entirely new categories of harm. That it amplifies existing ones faster than existing processes can respond.

What good actually looks like

Nothing in this is exotic. The organisations doing it well share a few habits.

Assessments cite sources. Every finding traces to a document, with a date, that someone else can open.
Decisions have names attached. Not a committee, a person.
The register is populated by discovery, not by self-report.
Risk ratings use the organisation's existing scale, so an AI risk can be compared against every other risk on the board's page rather than living in its own vocabulary.
Failure modes are written down before deployment, including what the organisation will do when the system is wrong, because it will be.
Somebody owns it. Governance distributed across everyone is owned by no one.

None of that requires a new framework. It requires the existing one to be used.

The uncomfortable part

Governance has a reputation problem inside technical teams, and mostly it has earned it. Governance written by people who have never deployed anything becomes paperwork, and paperwork gets routed around by people with a deadline.

The fix is not more documentation. It is governance that answers a real question a builder also wants answered: can I ship this, and will it still be defensible in twelve months?

Framed that way, governance is not the brake. It is the thing that lets you go faster, because the alternative to a documented yes is not speed. It is an undocumented yes that somebody unwinds later, expensively, in public.

Twelve weeks to a working pilot. Twelve months to an answer for the regulator. Only one of those is optional.

Primary sources worth your time, rather than vendor commentary:

National framework for the assurance of AI in government — Department of Finance. Agreed and released by the Data and Digital Ministers Meeting on 21 June 2024, it establishes the cornerstones and practices of AI assurance and shows how governments can practically apply Australia's AI Ethics Principles.
Australian Government AI assurance framework: findings and recommendations — DTA. The results of the pilot, including what agencies found difficult.
Use of enterprise Generative AI tools in the Victorian public sector — OVIC. Sets out minimum expectations to work through before purchasing and integrating an enterprise generative AI tool.
Artificial Intelligence: understanding privacy obligations — OVIC. Explains when the Information Privacy Principles of the Privacy and Data Protection Act 2014 apply, and why a privacy impact assessment belongs at design time.
NIST AI Risk Management Framework — the Govern, Map, Measure, Manage structure, and the inventory requirement most programmes skip.
ISO/IEC 42001 — the management system standard, for organisations that want certifiable rather than principles-based.

If you are standing up AI governance from nothing and want to talk it through, get in touch.