AI governance must-haves (According to me)

I get asked some version of this question most weeks. We are standing up AI governance, where do we start, what do we actually need.
The honest answer is that the list is shorter than most people expect, and almost none of it is a document. What follows is my list. It is opinionated, it is drawn from doing this inside a regulated organisation rather than writing about it, and you should argue with any part of it that does not fit your context.
1. A named owner, not a committee
The most common structural failure I see is governance owned by a forum. A steering group meets monthly, reviews a paper, notes the risk, and disperses. Nobody in that room can be called at 4pm on a Friday and asked to make a decision.
You need a person. One name, with the authority to say no and the obligation to explain why. Committees are useful for escalation and for the genuinely contested calls. They are useless as the default owner, because accountability distributed across twelve people is accountability held by none of them.
This is not a novel opinion. It is the first thing NIST asks for in the Govern function of the AI Risk Management Framework, and it is the part organisations skip most often.
2. An inventory built by discovery, not self-report
Every organisation I have worked with maintains an application register. Every organisation I have worked with also has staff signing up to free tools with a work email, building agents in low-code platforms, and connecting things to the tenant that will never reach that register.
So the register is simultaneously accurate and useless. It accurately records what was declared. It tells you nothing about what is running.
An inventory maintained by asking people to self-report their AI use is a survey. If your discovery is not automatic, at least partly, you do not have an inventory. You have a sample, and a biased one, because the people least likely to fill in the form are exactly the people doing the thing you most want to know about.
3. Assessments grounded in evidence somebody else can re-open
A vendor states in a sales conversation that customer data is not used for training. Someone writes that down. The risk is marked as mitigated.
That is not a control. It is a note.
Every third-party assessment worth signing is grounded in the vendor's own published documentation, cited and dated, so that another person can open the same page and check. Not because vendors lie, but because a claim you cannot re-check has a shelf life you cannot measure. When the vendor quietly changes a data-handling page, you want your assessment to break loudly rather than stay technically true and practically obsolete.
The test: could a colleague who has never met you reconstruct why this was approved, from the artefact alone, in an hour?
4. The gate sits before the build, not after it
Retrofitting governance onto a live system gives you two options. Switch it off, or accept the risk. Nobody wants to take the first to an executive, so in practice the second happens with a note about future remediation that never gets scheduled.
Constraints belong in the architecture review, not the incident review. That means the assessment is triggered when someone proposes the system, not when someone notices it.
Australian governments landed in the same place. The national framework for the assurance of AI in government, agreed by the Data and Digital Ministers Meeting in June 2024, is built around exactly this: assurance as something applied through the lifecycle rather than bolted on at the end.
5. Your existing risk scale, not a new one
There is a strong pull towards inventing a bespoke AI risk taxonomy. Resist it.
If your AI risks are rated on a scale nobody else in the organisation uses, they cannot be compared against any other risk, which means they cannot be prioritised, which means they will not be funded. An AI risk that sits in its own vocabulary sits in its own corner.
Rate AI risk on the same scale as everything else on the board's page. The novelty is in the failure modes, not in the arithmetic.
6. A written answer for when it is wrong
Not if. When.
Before deployment, someone should have written down what the system does when it produces a wrong output, who notices, how quickly, what the person affected can do about it, and what the rollback looks like. Half a page is enough.
Most programmes document the intended behaviour in great detail and the failure behaviour not at all. That asymmetry is where the actual harm lives, because the intended path is the one that was tested.
7. A review date on every approval
An approval without an expiry is a permanent decision made on temporary information. Models change under you. Vendors ship features you did not assess. Use cases drift from what was described in the form.
Every approval should carry a date at which somebody looks again. Twelve months is a reasonable default, shorter for anything touching personal information or a decision about an individual.
Three things you can probably skip
A bespoke framework. There are good ones already. NIST's AI RMF is free, structured around Govern, Map, Measure and Manage, and deliberately sector-agnostic. ISO/IEC 42001 exists if you need something certifiable. In Victoria, OVIC has published minimum expectations for enterprise generative AI tools that are more practical than most internal documents I have read. Writing your own from scratch is usually displacement activity.
An AI ethics board. Not because ethics does not matter, but because in most organisations the board becomes a place where hard questions go to be discussed rather than decided. Put the ethical questions into the assessment itself, where they have to be answered before the gate opens.
A tooling purchase, first. Governance platforms are useful once you know what your process is. Bought before that, they encode somebody else's process and you spend a year fighting it.
The one question
Here is how you find out whether you have built governance or documentation. Pick any AI system in your organisation, at random, and ask:
Who approved this, on what evidence, and what happens when it is wrong?
If you can answer all three in under a day, you have a programme. If you can only produce the policy that says such systems must be approved, you have a document.
The gap between those two is the entire job, and it is not closed by writing more.
Read more
- National framework for the assurance of AI in government - Department of Finance. Agreed by the Data and Digital Ministers Meeting on 21 June 2024, setting out the cornerstones and practices of AI assurance.
- AI assurance framework pilot: findings and recommendations - DTA. What agencies actually found difficult when applying an impact assessment tool in practice. More useful than the framework itself.
- Use of enterprise Generative AI tools in the Victorian public sector - OVIC. Minimum expectations to work through before purchasing and integrating a tool.
- Artificial Intelligence: understanding privacy obligations - OVIC. When the Information Privacy Principles apply, and why the privacy impact assessment belongs at design time.
- NIST AI Risk Management Framework - the Govern, Map, Measure, Manage structure, plus the inventory and named-ownership requirements most programmes skip.
- ISO/IEC 42001 - the AI management system standard, for organisations that need certifiable rather than principles-based.
If you are standing up AI governance from nothing and want to talk it through, get in touch.