Human review is work that must be designed, staffed, and measured. Its purpose is to detect and resolve consequential uncertainty before it causes an unacceptable outcome. A generic requirement to “have a human in the loop” leaves too much unspecified.
Begin with what the reviewer must decide. Then give that person the evidence, time, expertise, and authority to make the decision.
NIST identifies excessive deference to generated content as an automation-bias risk. The review arrangements here are proposed controls to evaluate, not a guarantee of detection. NIST Generative AI Profile, section 2.7
The procedure at a glance
| Step | Decision it settles | Record |
|---|---|---|
| 1. Review contract | What the reviewer decides, with what evidence and authority | Contract per review point |
| 2. Routing | Which cases reach a person, and which person | Routing rules with their owner |
| 3. Interface | What the reviewer sees and can do | Interface requirements, tested with reviewers |
| 4. Capacity | How many reviewer-hours the design needs, including bursts | Staffing plan and queue-age target |
| 5. Learning loop | How corrections become approved rules, or do not | Change log with scope and approver |
| 6. Escalation and fallback | What happens when review cannot resolve a case, or the system is down | Named owners and a rehearsed fallback |
| 7. Evaluation | Whether the combined human and system process works | Errors surviving review, effort, and rework |
Each step is proposed practice. The worked contract at the end applies them to the order-intake chapter's illustrative example.
Choose the review pattern first
Review can take several forms, and a workflow may need more than one. Choose deliberately; each catches different failures at a different cost.
| Pattern | How it works | Fits when | What it can miss |
|---|---|---|---|
| Approval of every case | A person approves each proposal before anything leaves the system | Consequences are hard to reverse and volume is manageable | Reviewer fatigue leads to approval without checking; capacity limits volume |
| Exception review | Only routed cases reach a person; the rest proceed under rules | Validation can reliably identify the cases that need judgment | Errors in cases the routing did not flag |
| Sampling after release | A share of released cases is checked afterward | Errors are reversible and a delay in detection is acceptable | Individual errors until the sample finds a pattern |
| Two-person review | A second qualified person checks critical items | A single error has serious consequences | Doubles the cost; both reviewers may share the same blind spot |
Every pattern still needs Steps 1 to 7. In the guide's workflows, sending a customer reply, approving an order, and publishing a catalog record each sit behind approval or exception review, because the action is external or hard to undo; the authority chapter explains how consequence and recovery inform that choice.
Step 1: Create a review contract
| Field | What to specify |
|---|---|
| Review object | The draft, proposed field change, recommendation, or action |
| Required checks | Identity, source support, applicability, completeness, or authorization |
| Evidence shown | Source passages, original values, validation results, and uncertainty |
| Reviewer | Role with the relevant domain skill and authority |
| Allowed decisions | Accept, edit, request information, reject, or escalate |
| Escalation destination | Named role and a usable queue |
| Time requirement | How long the case can wait before a fallback is needed |
| Record | Decision, material edits, reason, and configuration version |
A reviewer should not need to guess why a case reached them. Present the unresolved issue and relevant evidence clearly. A large transcript or document dump can make review nominal rather than effective.
Write one contract per review point, not one per system. A support pilot has at least two: the agent deciding whether a draft may be sent, and a quality reviewer sampling sent replies. They check different things and need different evidence.
Where this step fails. The contract names a reviewer without the authority to act on what they find, such as an agent who notices a pricing error but cannot change the price. Or it lists "check accuracy" as the required check, which leaves the reviewer to decide what accuracy means case by case.
Step 2: Route in a fixed order: rules, conditions, consequence, confidence
For whoever runs the pilot
A missing optional description and an uncertain shipping quantity deserve different treatment. Define critical fields and actions with the process owner. Route missing evidence, conflicting sources, unfamiliar cases, or failed validation to the appropriate expert.
Use confidence estimates only after evaluating what they mean for the actual cases. Explicit conditions such as an unrecognized customer or conflicting product identity may be more useful than a general score. Even a high-scoring output must satisfy permission and business rules.
Route by rules, conditions, and consequence before score
1. Do permission and business rules pass?
↓ No →
Block. A high score does not override permission or business rules.↓ Yes
2. Is an explicit condition present?
Missing evidence · conflicting sources · unfamiliar case · failed validation · unrecognized customer · conflicting product identity
↓ Yes →
Route to the appropriate expert, with the unresolved question and its evidence.↓ No
3. Does it touch a critical field or action?
Defined in advance with the process owner
↓ Yes →
A reviewer checks it against the review contract before it proceeds.↓ No
- Routine handling. Example from the chapter: a missing optional description.
Apply the routing in a fixed order. Deterministic rules come first: permission limits, required fields, and business validations, which no score can override. Observable conditions come next: an unknown customer, conflicting sources, a value outside its permitted range. Consequence comes third: a case touching a critical field or action goes to a qualified reviewer even when nothing else flags it. Only after those does a calibrated confidence estimate, if one exists and has been tested on local cases, decide between ordinary review and lighter handling.
Write routing rules as data the process owner can read and change, with a version and an owner. A rule hidden in a prompt cannot be audited.
Where this step fails. Everything is routed to review because no one defined critical fields, so reviewers learn to approve in bulk. Or a confidence threshold is chosen from a vendor default and never checked against the team's own cases.
Step 3: Give the reviewer a practical interface
For whoever runs the pilot
Show the original value beside the proposed value. Link material claims to the supporting passage. Highlight changes to quantities, dates, commitments, and permissions. Preserve missing or conflicting information instead of silently filling it in.
Make rejection and correction as accessible as acceptance. Do not measure reviewer performance solely by acceptance rate or speed. Those incentives can encourage approval without substantive checking, the automation-bias risk that NIST AI 600-1 names among human–AI configuration risks; the authority chapter lists the signs that approval has become a formality and how to test for them. NIST AI 600-1, section 2.7
Where reviewers disagree, determine whether the source is ambiguous, the rubric is incomplete, or expertise differs. Resolve the operating rule and preserve the disagreement as evidence rather than treating one person's response as automatically correct.
Test the interface with the reviewers before the pilot, using cases that contain known problems. Requirements to verify:
- The reason the case was routed is visible without scrolling.
- Every proposed value can be traced to its source in one step.
- Units, dates, and quantities are shown as received and as normalized.
- Rejecting, requesting information, and escalating take no more effort than accepting.
- The decision, any edit, and the reason are recorded without retyping.
- The configuration version that produced the proposal is recorded with the decision.
Step 4: Plan capacity before rollout
Review load can erase a pilot's benefits. An illustrative queue receives 300 cases per day, with 20% escalated for review at six minutes each. That requires 360 minutes, or six reviewer-hours daily, before meetings, interruptions, and other duties.
If the escalation rate reaches 40%, the same process requires 12 reviewer-hours. The system may still be valuable, but staffing and response expectations must change. These averages also hide bursts and difficult cases; monitor queue age and the distribution of review times.
Count this labor in the economics once. If it is already included in assisted handling time, do not add it again as a separate expense.
Plan for queue length as well as hours. Little's Law states that the average number of items in a queuing system equals the average arrival rate multiplied by the average time each item spends in it. Little, Operations Research, 2011 If escalations arrive faster or wait longer, more cases sit open, and each open case is a customer, order, or record waiting. The operations leaders guide works this through for an exception queue. The law describes averages over a stable period; a morning burst can produce a long queue even when the daily total is manageable.
Set a target for the oldest acceptable waiting case, and decide in advance what happens when it is exceeded: add reviewers, narrow eligibility, or fall back to the existing process.
Step 5: Separate correction from learning
For whoever runs the pilot
A reviewer correcting one case does not necessarily establish a rule for future cases. A customer's unusual product shorthand may apply only to that account. Route reusable feedback through an owner who can approve its scope and test the change.
Keep model, prompt, source, and rule updates traceable. Re-evaluate relevant cases after material changes. This prevents a local correction from silently changing behavior elsewhere.
Give each proposed rule change a record: what triggered it, its scope (one account, one category, everyone), who approved it, which held-out cases were rerun, and the result. Corrections that stay local remain in the case record and do not change behavior.
Step 6: Define escalation and fallback
Specify who takes over when the reviewer cannot resolve a case, how that person is notified within the operating process, and what happens if no one responds in time. A queue without ownership merely relocates the bottleneck.
For action-taking systems, identify who can suspend execution and how pending work is reconciled. Practice a fallback using actual staff and interfaces. A written rollback plan is less useful if no one can carry it out during a busy period.
Rehearse three situations before live use: the system is unavailable, the system is available but producing a pattern of bad proposals, and an escalation owner is absent. For each, write who notices, who decides, and how cases already in flight are handled.
Step 7: Evaluate the combined process
For your analyst
Test reviewers under realistic conditions, including plausible but incorrect output, missing evidence, and cases with no valid answer. Measure errors that survive review, correction quality, review time, escalation usefulness, and downstream rework.
Do not claim that human review eliminates risk. Its value depends on whether people can detect the relevant problems before the consequence occurs. When they cannot, narrow the task, change the interface, add stronger validation, or keep the action outside the system's authority.
Seed the evaluation with known problems so detection can be measured, guarding the seeds on the write path as the authority chapter describes, and report the combined process with denominators:
| Measure | What it shows |
|---|---|
| Seeded problems caught, of seeded problems presented | Whether reviewers detect the failures that matter |
| Errors reaching a customer or record, of cases released | What the combined process lets through |
| Review time distribution, not only the average | Whether bursts and hard cases fit the staffing plan |
| Escalations resolved usefully, of escalations | Whether the escalation route works |
| Accepted proposals later corrected, of accepted proposals | Whether acceptance reflects checking |
Where some cases bypass review, sample them for quality at an agreed rate and record what the sample can and cannot establish. A clean sample bounds the error rate only loosely; the pilot design playbook shows how loosely for small samples.
Worked contract: exception review in order intake
For your operations lead
This applies Steps 1 and 2 to the illustrative order-intake pilot; the operations leaders guide sizes the Step 4 exception queue for the same example. The roles and controls come from the order-intake chapter; the numbers are its illustrative assumptions.
| Field | Entry |
|---|---|
| Review object | A staged order: header, lines, and proposed SKU and unit mappings, with flagged exceptions |
| Required checks | Customer and document identity; each blocking exception resolved or rejected; quantity and unit against the source line; ship-to against the account record |
| Evidence shown | Source location for every field; original and normalized units side by side; mapping version; duplicate and revision status |
| Reviewer | Order-entry specialist |
| Allowed decisions | Approve, correct with reason, hold and query the customer, reject |
| Escalation destination | Catalog and mapping owner for unknown mappings; the owner of the customer's account record for a ship-to not on the approved list (a proposed assignment) |
| Time requirement | Set by operations from the service commitment to the customer, not from the queue's convenience |
| Record | Decision, corrected fields, reason, and configuration version |
Routing follows Step 2: rules first (required fields, approved master data, duplicate checks), then conditions (unknown customer, conflicting pack size, amended revision), then consequence (quantity, unit, price basis, and ship-to treated as critical). In the chapter's illustration, routine orders take 2 assisted minutes and exception orders 7, so the exception share drives review capacity directly; the chapter's economics show the result moving from the base case as that share rises.
Related chapters: authority and autonomy, pilot design, economics worksheet, document extraction, operations leaders.