AI governance: policies, roles and controls for responsible AI

Overview

AI governance is the set of policies, roles, processes and technical controls an organisation uses to make sure its AI systems are safe, lawful, fair, secure and accountable throughout their lives. It answers practical questions. Which AI systems do we have? Who owns each one? What could go wrong, and how bad would it be? What checks must a model pass before release? Who can approve an exception? How do we notice when a system starts behaving badly, and what do we do then?

An everyday analogy is food safety in a restaurant chain. Individual cooks know how to cook, but the chain also needs rules about suppliers, fridge temperatures, allergen labelling, cleaning schedules, inspections, incident reporting and who is responsible in each kitchen. Without that system, one careless branch can poison customers and damage the whole brand. AI governance is the food-safety system for AI: it does not replace skilled engineers, but it makes good practice consistent, visible and checkable across many teams.

Governance matters more as AI moves from experiments to products that make or shape decisions about people, money and safety. Laws such as the EU AI Act, GDPR and KVKK create legal duties; customers and auditors ask for evidence; and generative AI and agents introduce new risks such as prompt injection, hallucination and over-broad tool access. A working governance programme turns these pressures into routine engineering and management practice.

What it is

AI governance has organisational and technical building blocks.

Building block	What it covers
Principles and policy	The organisation's commitments, such as fairness, transparency, privacy, safety and human oversight, translated into binding rules
Roles and accountability	Executive sponsor, governance committee, system owners, risk, legal, privacy and security functions, with clear decision rights
AI inventory	A register of every AI system and model, with purpose, owner, data, vendor, users, markets and risk level
Risk classification	A method to rate systems by impact and map them to legal categories, such as high-risk under the AI Act
Lifecycle controls	Required steps from idea to retirement: impact assessment, data review, evaluation, approval, monitoring, decommissioning
Technical safeguards	Evaluations, red teaming, guardrails, access control, logging and monitoring
Documentation	Model cards, data sheets, risk assessments, decision records and audit trails
Third-party management	Due diligence and contracts for model providers, data suppliers and AI-enabled tools
Incident management	Reporting, investigating and learning from AI failures and near misses
Training and literacy	Making sure staff understand AI capabilities, limits and their responsibilities
Assurance	Internal audit, external audit or certification, and metrics reported to leadership

Where it came from

AI governance grew out of older disciplines. Banks have long applied model risk management to credit and trading models, for example under the US supervisory guidance SR 11-7 issued in 2011. Data protection law brought privacy impact assessments and accountability duties, strengthened by the GDPR from 2018. From the mid-2010s, research on algorithmic bias and high-profile failures pushed organisations to adopt AI ethics principles.

International frameworks followed. The OECD adopted its AI Principles in May 2019, updated in 2024. The US National Institute of Standards and Technology published the AI Risk Management Framework (AI RMF 1.0) in January 2023, organised around four functions: Govern, Map, Measure and Manage. In July 2024 NIST added a Generative AI Profile (NIST AI 600-1). ISO and IEC published ISO/IEC 23894 on AI risk management in 2023 and ISO/IEC 42001, the first certifiable AI management system standard, in December 2023. The Council of Europe's Framework Convention on Artificial Intelligence, an international treaty on AI and human rights, opened for signature in September 2024. The EU AI Act entered into force in August 2024, making several governance practices, such as risk management, documentation, human oversight and AI literacy, legal requirements for covered systems.

How it works

1. Set direction and ownership

Leadership approves an AI policy and risk appetite, names an accountable executive and creates a cross-functional body, often including engineering, product, legal, privacy, security, risk and ethics, with clear decision rights.

2. Build the inventory

Every AI system, including purchased tools and features that call external models, is registered with an owner, purpose, data sources, model provider, users, markets and status. Shadow AI, unregistered use of AI tools, is brought into view through procurement and network controls.

3. Classify risk

Each system is rated by potential impact on people, finances, safety and reputation, and mapped to legal categories. A low-risk internal summariser gets a light process; a model that affects hiring or credit gets the full process.

4. Apply lifecycle controls

Risk tier determines required steps. Typical gates include an impact assessment at design, a data review for provenance, consent and bias, pre-release evals for accuracy, robustness, fairness and safety, red teaming for generative and agentic systems, security review, and sign-off by named approvers.

5. Build in human oversight

For consequential decisions, define who reviews outputs, with what information, authority and time. See human-in-the-loop patterns.

6. Monitor in production

Track performance, drift, fairness metrics, incidents, complaints and costs with model monitoring. Set thresholds that trigger review, rollback or retraining.

7. Handle incidents and change

Report and investigate failures, notify regulators or customers where required, and feed lessons into policy. Material changes, such as a new model version or new use, re-enter the review process.

8. Assure and report

Internal audit checks that controls work. Some organisations certify against ISO/IEC 42001. Leadership receives regular metrics: inventory coverage, systems by risk tier, open findings and incidents.

Key concepts and terms

Term	Meaning
Responsible AI	Developing and using AI in ways that are safe, fair, transparent and accountable.
AI inventory	A register of AI systems and models with key attributes.
Risk tiering	Assigning systems to risk levels that determine required controls.
Impact assessment	Structured analysis of a system's effects on people and the organisation.
Human oversight	Meaningful human ability to monitor, intervene and override.
Model risk management	Discipline for validating and controlling models, originating in banking.
AI management system	Organisation-wide policies and processes, as defined in ISO/IEC 42001.
NIST AI RMF	Voluntary US framework with Govern, Map, Measure and Manage functions.
Shadow AI	AI tools used without approval or visibility.
Red teaming	Adversarial testing to find failures and misuse paths.
Model card	Documentation describing a model's purpose, performance and limits.
Accountability	Clear assignment of responsibility for outcomes and decisions.

A worked example

This scenario is illustrative. A mid-sized insurance company has 300 employees and is adopting AI quickly. A survey finds 41 AI uses: 12 built in-house, 18 vendor tools with AI features and 11 cases of staff using public chatbots with customer data. Nobody can say which are high-risk.

The company appoints its chief risk officer as accountable executive and forms an AI governance group meeting fortnightly. It adopts a two-page AI policy and a three-tier risk scheme. Tier 3 covers systems that influence decisions about customers, such as claims triage and pricing. Tier 2 covers customer-facing generative AI. Tier 1 covers internal productivity tools.

Within three months all 41 uses are in an inventory. Four are Tier 3, including a claims fraud-scoring model. Public chatbot use with customer data is stopped and replaced with an approved enterprise assistant whose contract forbids training on company data. For the fraud model, the team runs an impact assessment, finds that the false-positive rate is 1.8 times higher for one postcode group, retrains with adjusted features, and sets a monitoring threshold to alert if the ratio exceeds 1.25. Every flagged claim is reviewed by a trained handler before any action. A model card, evaluation report and approval record are stored with the model in the registry. After a year, an internal audit tests 20 controls and finds three gaps, mainly incomplete monitoring for vendor tools, which become next quarter's priorities.

Strengths and limitations

Strengths:

Reduces the chance and impact of harmful or unlawful AI outcomes.
Gives leadership, regulators and customers evidence of responsible practice.
Makes good engineering practice consistent across teams.
Speeds approvals by making requirements clear in advance.
Prepares the organisation for laws such as the EU AI Act.

Limitations:

Can become bureaucratic paperwork that slows teams without reducing risk.
Hard to cover vendor tools and shadow AI completely.
Metrics for fairness, safety and robustness are imperfect and context-dependent.
Requires scarce skills across law, ethics, data science and security.
Frameworks and laws change, so programmes need regular updating.

How it compares with alternatives

Framework or approach	Type	When it fits best
NIST AI RMF	Voluntary risk framework	Structuring risk activities flexibly, especially in the US
ISO/IEC 42001	Certifiable management system standard	Organisations wanting external certification and integration with ISO 27001
EU AI Act	Binding regulation	Legal compliance for AI in the EU market
Model risk management (SR 11-7 style)	Supervisory guidance for financial models	Banks and insurers with existing model validation functions
Ethics principles only	High-level commitments	A starting point, but insufficient without processes and controls

Practical guidance and best practices

Start with an inventory; you cannot govern AI systems you do not know about.
Use risk tiers so effort matches impact, and keep the low-risk path light.
Name a single accountable owner for each system.
Turn principles into concrete, testable requirements per tier, such as required evaluations and approvals.
Embed controls in existing tools, such as pipelines, model registries and ticketing, rather than separate forms.
Cover vendors: due diligence, contract terms on data use and incident notification, and periodic review.
Monitor deployed systems with thresholds that trigger action, and rehearse incident response.
Train staff in AI literacy appropriate to their role.
Map your controls to the frameworks and laws that apply, so one set of evidence serves several purposes.

Common mistakes

Publishing ethics principles without any process to enforce them.
Making governance a one-time approval instead of a lifecycle activity.
Ignoring purchased tools and employee use of public AI services.
Applying the same heavy process to every use case, pushing teams to avoid it.
Treating human oversight as a formality, with reviewers lacking time or authority.
Leaving governance to legal teams alone, without engineering involvement.

Frequently asked questions

Is AI governance a legal requirement?

Parts of it are. The EU AI Act requires risk management, documentation, human oversight and AI literacy for covered systems, and data protection laws require accountability. Other parts are good practice expected by customers, investors and auditors.

Which framework should we use?

Many organisations use the NIST AI RMF to structure risk activities, ISO/IEC 42001 for a certifiable management system, and map both to laws such as the EU AI Act. The choice depends on markets, sector and whether certification is needed.

Who should own AI governance?

An accountable executive with a cross-functional group. Day-to-day ownership of each system belongs to a named business or product owner, supported by risk, legal, privacy, security and engineering.

How does governance apply to generative AI and agents?

The same lifecycle applies, with added focus on prompt injection, data leakage, hallucination, content safety, copyright, tool permissions and monitoring of agent actions. NIST's Generative AI Profile and the OWASP Top 10 for LLM Applications are useful references.

Does governance slow innovation?

Badly designed governance does. Good governance makes requirements clear, keeps low-risk paths fast and prevents costly incidents and rework, which often speeds delivery overall.

What evidence do auditors look for?

An inventory, risk classifications, impact assessments, evaluation results, approvals, monitoring records, incident logs, training records and evidence that controls actually operate.

Glossary

AI governance: policies, roles, processes and controls for AI.
Responsible AI: the goal of safe, fair, transparent and accountable AI.
AI RMF: NIST's AI Risk Management Framework.
ISO/IEC 42001: the AI management system standard.
Risk tier: a category determining required controls.
Impact assessment: analysis of effects on people and the organisation.
Model registry: a store of model versions with metadata.
Shadow AI: unapproved AI use.
Human oversight: the ability of people to monitor and intervene.
AI literacy: the knowledge needed to use AI appropriately.

Summary

AI governance is the organisational system that keeps AI safe, lawful and accountable: principles turned into policy, clear ownership, an inventory, risk tiers, lifecycle controls, human oversight, monitoring, incident handling and assurance. It draws on model risk management, data protection and frameworks such as the OECD AI Principles, the NIST AI RMF of 2023 and ISO/IEC 42001 of 2023, and it now underpins compliance with laws such as the EU AI Act. Start with an inventory and risk tiers, embed controls in everyday engineering tools, cover vendors and generative AI, and keep the process proportionate so it reduces risk without blocking useful work. This page is general information, not legal advice.

It belongs to Security and governance in my stack, under MLOps and AI infrastructure. Others in the same group:

OAuth 2.0 OIDC JWT mTLS Zero Trust OWASP GDPR KVKK EU AI Act