Assistance, approval, and autonomy

Autonomy describes what a system is permitted to do without a person approving each action. It is not a measure of intelligence. A highly capable model can operate as a read-only assistant; a simple rule can automatically trigger a consequential action.

Choose authority separately from capability. The relevant questions are what can happen, who is affected, how errors are detected, and whether the action can be reversed in time.

This chapter describes four operating arrangements, the contract that defines each, what makes human approval real rather than ceremonial, how recovery limits authority, and what evidence would justify changing it. It ends with the starting arrangement for each of the guide's nine workflows. The framework is editorial guidance, not a standard.

Four operating arrangements

The latter two can overlap in practice. The important requirement is an explicit operating contract, not a maturity ladder. More autonomy is useful only when it improves the workflow within acceptable constraints.

Authority belongs to a step, not to a system

One system can work under several arrangements at once. In the order-intake chapter, the assistant extracts and proposes freely (assistance), the order waits for a specialist (approval before action), and the integration then writes it under deterministic rules designed to prevent a duplicate, using an identifier the order system must be verified to honor. In meeting follow-up, the transcript and candidate items are assistance, while sending the recap and creating each task need approval. Assigning one arrangement to "the AI" hides these differences and invites the most permissive step to set the tone for the rest.

A misreading to avoid

"We will start with assistance and move up as trust grows" treats the arrangements as rungs. The chapter's position is different: the right arrangement depends on the action's consequence and recoverability, and some actions may never warrant less than approval before action, however good the system becomes. Releasing a payment, for example, is an authorization decision with control implications; the finance leaders guide explains why a confidence score is not an authorization. Conversely, a low-consequence, easily undone action may not need case-by-case approval even in a first pilot.

Define the operating contract

Record the allowed inputs, eligible cases, permitted tools, maximum scope, conditions requiring escalation, and authority to suspend the system. Specify what the system must never infer: identity, consent, spending authority, or a commercial commitment cannot be invented from ambiguous text.

For a purchase-order pilot, the contract might allow extraction and proposed mappings while prohibiting order submission until an employee approves the fields. For a knowledge assistant, it might allow retrieval from documents the employee can access but no modification of records.

Permissions should be enforceable by the application. Instructions in a prompt are useful descriptions of desired behavior but should not be the only barrier to an unauthorized action.

What a contract records

As a proposed template:

Enforce the contract in the application

Prompt injection exploits untrusted input combined with a higher-trust prompt; the retrieval chapter cites the definition and the standard mitigations, including least privilege and human approval for high-risk operations. Neither source measures how well any control works. Together they explain why the guide's workflows withhold tools rather than instruct the model not to use them: an incident brief cannot close alerts because it has no credential to do so, not because it was told not to.

Human approval must be workable

An approval button does not establish effective review. The reviewer needs enough information, time, skill, and authority to detect and correct consequential errors. Show the proposed change, its supporting evidence, missing information, and the result of validation checks.

Avoid making reviewers reconstruct everything from scratch. If they must repeat the original task to know whether the output is correct, the system may still be useful, but its economics should include that effort. Evaluate the combined person-and-system process.

NIST identifies automation bias and over-reliance as human–AI configuration risks. This is a reason to test review effectiveness, not an estimate of how often a reviewer will fail. NIST Generative AI Profile, section 2.7 Design evaluation to include realistic workload and disagreement, not only quiet demonstrations where every output receives unusual attention.

Signs that approval has become a formality

These are things to watch for, proposed as practice rather than reported observations:

• Approval time collapses while volume rises. If a specialist approves a staged order in seconds, check whether the evidence is being read.
• Edits stop. A reviewer who never changes anything may be reviewing excellent output or not reviewing. Seeded test cases with known errors, mixed into ordinary work during a pilot, distinguish the two. Guard them on the write path, not only through the reviewer: use a test customer, a sandbox account, or a flag the integration refuses, invisible to the reviewer, so that a seed approved in error cannot reach a real order, client, or record.
• The evidence is not on screen. If the approver must open three systems to check a value, the interface invites approval without checking. The support chapter asks for the precise passage behind each consequential claim, not a link to a help center.
• Volume outruns reviewers. The human review playbook shows how escalation share drives reviewer-hours; when review cannot keep up, approval degrades before anyone records it.

Approval that adds nothing is a design problem

If a reviewer cannot detect the errors that matter (because the evidence is unavailable, the time is too short, or the reviewer lacks the expertise), approval before action provides little protection, and labeling the arrangement "human in the loop" overstates it. The remedies are to change the interface, narrow the scope, route difficult cases to someone qualified, or keep the step in assistance until review can work.

Reversibility has limits

Deleting a draft is straightforward. Reversing a shipped order, a sent message, or a disclosed confidential fact can be difficult or impossible. Even technically reversible actions can have consequences before the reversal completes.

Define the available recovery action and the maximum acceptable delay. Identify what information is needed to reconcile a partial write or determine whether an external action happened. Test recovery during the pilot rather than assuming a vendor's undo feature covers the business process.

Three kinds of irreversibility

The third kind has no undo at all, which is why the knowledge-retrieval design puts its control before retrieval. A reviewer who sees restricted material has already been exposed.

Uncertain outcomes

Some actions end in doubt: the request may have succeeded while the confirmation was lost. An uncertain response does not establish that the original operation failed, and caller-provided identifiers can make retries safe; the order-intake chapter cites the engineering source. Its retry sequence shows the difference between a retry that creates a duplicate and one that finds the existing order. Recovery testing should include this case, not only clean failures.

A worked boundary decision

Suppose an onboarding assistant identifies that a customer needs an integration. It can explain approved setup steps and show verified account state. Connecting the integration may expose data or authorize future activity. That step requires the existing account authority and confirmation mechanism, not merely conversational agreement interpreted by the model.

The system can still make onboarding easier while keeping that action boundary intact. Conversely, automatically retrieving a public help article may need no case-by-case approval because its effect and permissions are limited.

Walking the decision through

The same reasoning, applied step by step to the onboarding example (an editorial illustration):

1. What can happen? Connecting an integration can expose customer data to a third party and authorize activity that continues after the conversation ends.
2. Who is affected? The customer, their own customers whose data flows, and the software company's security obligations.
3. How would an error be detected? Possibly late, by the customer, after data has moved.
4. Can it be reversed in time? Disconnecting stops future flow; it does not recall data already shared.

The answers place the action among those that keep a person, using the customer's own authorization mechanism, in front of it. The onboarding chapter follows this: configuration changes stay with existing permissions, and custom integrations leave for the implementation team rather than being drafted as setup steps. Explaining the setup steps, by contrast, is assistance with a reviewer who can catch mistakes before the customer acts on them.

Decide what evidence permits expansion

Before increasing authority, evaluate unsupported actions, scope violations, uncertain cases, and recovery under failure. Determine whether the monitoring team can detect problems quickly enough and whether the exception queue has an owner.

Retain explicit reasons to reduce authority after deployment: a changed source, unfamiliar input class, degraded quality, or a new integration can invalidate the evidence supporting the previous boundary. Autonomy is a continuing operating choice, not a one-time graduation.

Evidence that would support a change

As a proposed checklist, before moving a step from approval to monitored execution:

• Results on held-out cases, including the hard ones, with errors counted by severity, not as one accuracy figure. The pilot design playbook explains why zero observed failures in a small test says little about rare ones.
• A record of what reviewers changed during the approval period, and why. If reviewers caught consequential errors, removing them removes that catch.
• A tested recovery for each failure the step can cause, with its delay measured.
• Monitoring that would detect the failures that matter, with an owner and a response time.
• A narrower scope for the expanded authority than for the pilot, so the change can be evaluated before it widens.

Evidence that should reduce authority

• A source or model change since the evidence was gathered.
• A new kind of input the evaluation set did not cover.
• An incident in the expanded scope, even one caught before harm.
• Monitoring that has stopped being read.

How authority maps across the guide's workflows

Each workflow chapter sets a starting arrangement. These are the guide's proposed designs for fictional pilots, not observations of deployed systems.

None of the nine starts with monitored or bounded execution. That reflects the guide's choice of first pilots, where the evidence needed to justify less oversight has not yet been gathered. It is not a claim that those arrangements are never appropriate. A candidate for monitored execution would be a low-consequence, recoverable step inside one of these workflows, such as sending a customer an acknowledgement for an order a specialist has already approved, after the approval period has produced the evidence listed above. Retrieval itself stays assistance throughout: it prepares material for a person and acts on nothing.

What to do with this chapter

As a proposed sequence for any new system:

1. List the steps where output becomes action, using the workflow-change chapter's map.
2. For each, answer the four questions: what can happen, who is affected, how errors are detected, whether recovery is possible in time.
3. Choose an arrangement per step and write the operating contract, including what is never inferred.
4. Enforce permissions in the application; withhold tools the step does not need.
5. Make approval workable, and test whether it catches seeded errors.
6. Test recovery, including uncertain outcomes.
7. Record the evidence that would justify a change, and the conditions that would reverse one.

Related chapters: human review playbook, pilot design, workflow change, capabilities, onboarding, order intake.