Guide · Updated September 2026

AI governance: a working framework for companies without a compliance department

AI governance is how an organization decides where AI may be used, under what rules, with what checks, and who answers for the result. Most of what is written about it assumes a bank’s risk function. This guide assumes you have an IT lead, outside counsel, and a business to run, and it reflects three changes to the rules in 2026 that most published guidance has not caught up with.

What AI governance is, and what it is not

Strip away the vocabulary and governance answers four questions. Where are we using AI? Which of those uses could hurt someone or the business? What do we check before and after we switch one on? And who is accountable when it goes wrong? If your company can answer all four in writing, it has AI governance. If it cannot, no committee or ethics statement makes up the difference.

It is not a statement of principles, though it may start from one. It is not a committee, though it may use one. And it is not the same as security or privacy, though it borrows from both: an AI system can be perfectly secure and still discriminate, fabricate, or drift.

The gap between having AI and governing it is wide. In ISACA’s 2026 poll of more than 3,400 audit, security, and privacy professionals, only 38% said their organization had a formal, comprehensive AI policy. That was up from 28% a year earlier, and a quarter still had none. Among the breached organizations in IBM’s 2026 study, 35% had no AI governance policy and another 33% were still writing one, and only 19% coordinated their governance and security teams. Employees see the same thing from below: in the KPMG and University of Melbourne study, only 40% said their workplace had a policy or guidance on generative AI.

The six parts of a working framework

Every serious framework, whatever its diagram looks like, reduces to these six. The label on each is the function it corresponds to in the NIST AI Risk Management Framework, so you can map your work to the standard when a customer or auditor asks.

An inventory NIST: Map

A list of every place AI touches the business: what you built, what you bought, and the AI features that arrived inside software you already had. For each one, record what it does, what data it sees, who owns it, and who the vendor is. Most governance efforts that fail never finished this step, and everything after it depends on it.

Risk tiers NIST: Map

Not every use deserves the same scrutiny. Sort by consequence: a tool that drafts marketing copy is low; one that touches customer data is medium; one that influences hiring, lending, pricing, medical, or legal outcomes for a person is high. Three tiers are enough. The tier decides how much of the rest applies.

Rules people can follow NIST: Govern

An acceptable use policy, a data classification that says what may go into which tools, and minimum contract terms for AI vendors: no training on your data, retention limits, breach notice, audit rights. Two or three pages in total. If staff cannot recall the rule while they work, it is not a control.

Review before launch NIST: Measure

A checkpoint sized to the tier. Low-tier uses register themselves in the inventory. High-tier uses need evidence: evaluation results on realistic cases, testing for biased outcomes where people are affected, a security review, and a decision about where a person stays in the loop. The reviewer needs the authority to say “not yet.”

Monitoring after launch NIST: Manage

Models drift, vendors change their models underneath you, and usage spreads beyond the case you approved. Decide what gets measured, who looks, how an incident is reported, and how a system is switched off. That last item is not hypothetical: in ISACA’s 2026 poll, 39% of respondents did not know whether their organization had a documented process for shutting down an AI system.

Named accountability NIST: Govern

One executive owns the program. Every system in the inventory has a named business owner who answers for its behavior. The board or leadership team sees a short report a few times a year: what is in use, what is high-tier, what went wrong, and what changed. Accountability that belongs to a committee belongs to no one.

Part three is where most companies should start writing, and the AI acceptable use policy template gives you the structure. Part one is where most should start working, and it is usually where they find their shadow AI.

NIST AI RMF and ISO 42001: which one you need

The NIST AI Risk Management Framework is free, voluntary, and the common language of AI governance in the United States. NIST released version 1.0 on January 26, 2023. It is organized around four functions: Govern (the culture, roles, and policies that cut across everything), Map (understand the context and risks of each system), Measure (test and track them), and Manage (act on what you find). A companion Generative AI Profile, NIST AI 600-1, followed on July 26, 2024. NIST has said the framework is being revised, so treat 1.0 as current but not final.

ISO/IEC 42001, published in December 2023, is a different kind of document. It is a management-system standard, in the family of ISO 27001 for security, that specifies requirements for establishing and continually improving an AI management system. It applies to organizations of any size that develop, provide, or use AI. Unlike the NIST framework, you can be certified against it, by accredited third-party auditors working under a companion standard, ISO/IEC 42006, published in 2025.

The practical answer for most companies: use NIST as the thinking tool, because it is free, readable, and what American customers and regulators reference. Look at ISO 42001 when enterprise customers start asking for a certificate in security questionnaires, or when you sell AI into Europe. Pursuing certification before you have an inventory is the expensive way to discover you needed an inventory.

The rules that apply, as of September 2026

This is the section where published guidance ages fastest. Three of the four items below changed materially during 2026. What follows was checked against the official texts, and it is a map rather than legal advice.

EU AI Act

Applies to: Anyone placing AI on the EU market or whose AI output is used in the EU, wherever they are based

Regulation (EU) 2024/1689 entered into force on August 1, 2024 and sorts AI into four tiers: unacceptable risk (prohibited), high risk, transparency risk, and minimal risk. Prohibitions have applied since February 2, 2025 and obligations for general-purpose AI models since August 2, 2025. The high-risk dates moved in 2026. The AI Omnibus, in force since July 27, 2026, pushed the rules for high-risk systems in areas such as employment, education, and biometrics to December 2, 2027, and for AI embedded in regulated products to August 2, 2028. It also added a ban on AI that generates non-consensual intimate imagery, eased the AI literacy duty, and extended relief for small companies to small mid-caps. Fines for prohibited practices run up to €35 million or 7% of worldwide annual turnover, whichever is higher. Any guide telling you high-risk obligations begin in August 2026 predates the change.

New York City Local Law 144

Applies to: Employers and employment agencies using automated tools to screen candidates or employees in NYC

In force since January 1, 2023 and enforced since July 5, 2023. An automated employment decision tool may not be used unless it has had an independent bias audit within the past year, a summary of the audit is public, and candidates receive notice at least ten business days beforehand. Penalties run from $500 to $1,500 per violation, and each day of use counts separately. If a vendor’s resume screener ranks your New York applicants, this law applies to you, not only to the vendor.

Colorado SB26-189

Applies to: Developers and deployers of automated decision-making technology affecting Colorado consumers

Colorado’s 2024 AI Act (SB24-205) was the first broad US state AI law, and it never took effect as written. It was delayed in 2025 and then repealed and re-enacted in May 2026 as SB26-189, which goes into effect on January 1, 2027. The new law covers automated decision-making in consequential decisions such as employment, housing, lending, insurance, health care, and education. It requires notice to consumers, disclosure after an adverse outcome, and gives people the right to correct their data and ask for human review. The Attorney General enforces it.

US banking: SR 26-2

Applies to: Banking organizations, chiefly those with more than $30 billion in assets

For fifteen years the reference point for model governance in US banking was SR 11-7. On April 17, 2026 the Federal Reserve, OCC, and FDIC replaced it with SR 26-2. One footnote matters more than the rest of the document: generative and agentic AI are explicitly outside its scope, with banks told to apply their own risk management to them. The largest regulated users of AI in New York are therefore building governance for their newest systems without a dedicated supervisory template.

Two points hold whatever jurisdiction you are in. First, laws that never mention AI still apply to it: a discriminatory lending decision is unlawful whether a person or a model made it, and privacy law covers personal data in a prompt as much as in a database. Second, your customers are regulating you faster than governments are. AI questions increasingly appear in vendor security reviews, and they tend to ask about the same six things.

Who should own it, by company size

Under about 200 people

One executive owner, usually the COO or whoever runs security, and a working group of three: IT or security, legal or operations, and a business lead who uses AI every day. Two hours a month. The inventory is a spreadsheet. This is enough, and a committee of twelve would be worse.

200 to 2,000 people

The same structure plus someone for whom this is a real share of their job. Many companies this size bring in a fractional AI leader to build the program over a quarter and train an internal owner to run it. A review board meets monthly for high-tier uses only.

Larger, or regulated

A dedicated function with its own lead, integrated with existing risk, privacy, and model validation teams rather than built beside them. Here a certifiable standard and governance tooling begin to pay for themselves.

There is no settled answer on which title should hold it. In McKinsey’s survey of organizations using AI, 28% said the CEO oversees AI governance and 17% said the board does. What matters more is that it is one person, that they can stop a launch, and that leadership hears from them on a schedule. If no one inside has the capacity, a fractional AI leader is a common way to stand the program up and hand it over.

A 90-day plan

Days 1 to 30

See it and own it

Name the executive owner and the working group.
Build the inventory: survey staff without blame, pull sign-on and expense data, and read your vendors’ release notes for AI features you never approved.
Publish a two-page acceptable use policy and a three-tier data classification.

Days 31 to 60

Sort it and set the gate

Assign every inventoried use a risk tier and a named business owner.
Review contracts for the vendors behind medium and high-tier uses: training on your data, retention, breach notice.
Define the pre-launch review for high-tier uses and run it once on a real system, so you find out what evidence you cannot yet produce.
Check the high-tier list against the laws above. Hiring tools used in New York are the most common surprise.

Days 61 to 90

Keep it running

Set up monitoring and an incident route for high-tier systems, including how each one gets switched off.
Train everyone on the rules, and owners on their duties.
Deliver the first leadership report. Then put the inventory refresh and the policy review on a quarterly calendar.

For the training in the last phase, see AI governance training. To find out where you stand before you begin, the AI readiness assessment scores governance as one of its five dimensions.

Do you need AI governance tools?

Eventually, and later than vendors will suggest. The category has four kinds of product: registries that hold the inventory and route reviews; evaluation and monitoring tools that test models and watch them in production; compliance tools that map your controls to NIST, ISO 42001, or the EU AI Act; and access and data controls that enforce your rules at the point of use.

A spreadsheet is the right registry until it has a few dozen rows and more than one editor. Buy monitoring when you have a high-tier system in production, and compliance mapping when a customer or regulator asks for it. Software bought before the inventory exists tends to end up governing nothing.

AI governance as a job market

Governance is also a career, and an undersupplied one. In the IAPP’s AI Governance Profession Report, fielded in 2024, 77% of surveyed organizations were working on AI governance and nearly a quarter named finding qualified people as a challenge. The AIGP certification from the IAPP, whose exam launched in April 2024, is the best-known credential for individuals.

Our own data shows where the demand is not. As of September 20, 2026, only 6 of the 47 New York AI companies we track name AI governance in an open role. The practices governance depends on are asked for far more often: 25 companies name evals and 9 name human-in-the-loop design. Startups hire for the engineering of trustworthy AI and rarely for the title. In our research the governance titles sit with banks, insurers, law firms, and health systems, which is where a governance job search in New York should point. The live figures are in the AI skills glossary.

For what the work looks like inside a large regulated company, Alayna Kennedy, Director of AI Governance at Mastercard, covers it in Episode 17 of the AI in NYC Show, including the distance between publishing principles and embedding them in a data pipeline.

Four ways it goes wrong

Principles with no process. A values statement on the website changes nothing in a product review. If your governance has no checkpoint where something can be stopped, you have a press release.

Governance as the department of no. When review is slow and the answer is usually refusal, people stop asking. Unapproved use goes up, and your inventory becomes fiction.

Governing only what you build. Most companies buy far more AI than they build. The resume screener, the support bot, and the AI features in your CRM belong in the inventory too.

Doing it once. A one-time audit is out of date within a quarter. The inventory is a living document, or it is a historical one.

AI governance: common questions

What is AI governance?

AI governance is the set of decisions, rules, and checks an organization uses to control how AI is built, bought, and used, together with the evidence that they are followed. In practice it has six parts: an inventory of where AI is used, risk tiers, rules people can follow, review before launch, monitoring after launch, and named accountability.

What is an AI governance framework?

A framework is a structure for organizing that work. The two most used are the NIST AI Risk Management Framework, a free and voluntary US framework built on four functions (Govern, Map, Measure, Manage), and ISO/IEC 42001, an international management-system standard that an organization can be certified against. Most companies use NIST to decide what to do and consider ISO 42001 when customers begin asking for proof.

Is AI governance legally required?

It depends on what your AI does and where. The EU AI Act imposes binding duties that phase in through 2028. New York City requires bias audits and notice for automated hiring tools. Colorado’s automated decision-making law takes effect January 1, 2027. Beyond AI-specific law, existing privacy, consumer protection, and anti-discrimination rules already apply to decisions made with AI. This guide is not legal advice; confirm your obligations with counsel.

Who should be responsible for AI governance?

One named executive, supported by a small cross-functional group covering technology, legal or compliance, and the business. In McKinsey’s survey of organizations using AI, 28% said the CEO oversees AI governance and 17% said the board does. Which title holds it matters less than three things: that it is one person, that they can stop a launch, and that they report to leadership on a schedule.

What is the difference between AI governance and responsible AI?

Responsible AI describes the goals: systems that are fair, safe, transparent, and accountable. AI governance is the machinery that gets you there: who decides, what is checked, what is written down, and what happens when something goes wrong. A company can publish responsible AI principles and have no governance at all, which is the most common failure in the field.

Do small companies need AI governance?

Yes, in proportion. A 50-person company needs a named owner, a list of the AI tools in use, a two-page policy, and a rule about which data may go into which tools. That is a few days of work, and it gives you an answer when an enterprise customer’s security review asks how you use AI.

Is there an AI governance certification?

For individuals, the best known is the IAPP’s Artificial Intelligence Governance Professional (AIGP) credential; training launched in 2023 and the exam in April 2024. For organizations, ISO/IEC 42001 is the certifiable standard, with audits carried out by accredited third-party bodies under ISO/IEC 42006.

Sources

Regulatory dates were read in the official texts in September 2026. Survey figures describe the populations noted: ISACA and IAPP respondents are governance and security practitioners, IBM’s are organizations that suffered a breach, and McKinsey’s governance figures come from its July 2024 survey. This guide is general information, not legal advice.

Need someone who has built this before?

We match companies with vetted AI leaders and consultancies who have stood up governance programs in production, and we will tell you when a policy and a spreadsheet are all you need.

Start a project brief Fractional AI leadership