A pilot should end with a decision supported by evidence: expand, revise, simplify, defer, or stop. Define that decision before building the system. Otherwise a demonstration can consume the budget without answering whether the workflow is useful.
The pilot evaluates a complete operating arrangement, including people, sources, integrations, and review. It should not measure only the model while assuming the rest will work.
This playbook is a procedure. Each step below says what to decide, what to record, and how the step can fail. The method is proposed practice from this guide; the one external study cited is used only for what it shows about comparison design. The worked charter at the end uses the illustrative support example and invents no results.
The procedure at a glance
| Step | Decision it settles | Record | Companion |
|---|---|---|---|
| 1. Charter | What question the pilot answers and who decides | Signed charter | Pilot kit |
| 2. Comparison | What the assisted process is compared against | Baseline definitions and assignment method | Pilot kit |
| 3. Case set | Which cases are tested, and which are held out | Case inventory with categories and sources | Evidence checklist, case library |
| 4. Acceptance | What counts as passing, set before results | Requirements, severities, thresholds | Evaluation scorecard |
| 5. Stages | How the system moves from offline to live | Stage entry and exit evidence | Scorecard |
| 6. Measurement | Which numbers are collected, with denominators | Case records and effort logs | Scorecard and workbook |
| 7. Result | What happened, including failures | Results memo | Pilot kit |
| 8. Decision | Expand, revise, simplify, defer, or stop | Decision memo with owner | Pilot kit |
Step 1: Complete the pilot charter
| Field | What to write |
|---|---|
| Workflow owner | Person accountable for the operational result |
| Eligible population | Cases included and excluded, with reasons |
| Current process | Baseline system and complete unit of work |
| Proposed change | Assistance, inputs, outputs, and authority boundary |
| Business hypothesis | Improvement expected and how value would be realized |
| Quality requirements | Acceptable output and errors by severity |
| Evaluation design | Cases, comparison, reviewers, and observation period |
| Operating limits | Permissions, capacity, latency, and recurring cost constraints |
| Stop conditions | Events requiring suspension or rollback |
| Decision date | When evidence will be reviewed and by whom |
| Configuration versions | Model, prompt, retrieval, and source versions under test |
| Sample-size rationale | Why the case counts are enough for the thresholds, and the uncertainty they leave |
The charter is a working agreement. Record changes during the pilot so a revised scope is not presented as if it were the original experiment.
Three fields carry most of the weight. The unit of work must be the completed outcome, not an intermediate artifact: a resolved request rather than a sent reply, an accepted order rather than an extracted field. The authority boundary must be enforceable by the system's permissions, not only described in a prompt. The business hypothesis must name the mechanism by which value would be realized, so the economics can be tested rather than asserted later; the economics worksheet supplies the arithmetic.
Where this step fails. A charter written after the build can end up describing what was built rather than what should be tested. A charter without a decision owner produces a report but no decision. A charter that leaves the observation window open lets follow-up work fall outside the measurement.
Step 2: Establish a comparison
For whoever runs the pilot
Measure the current process or a credible simpler alternative. Record effort, quality, volume, and exceptions using the same definitions planned for the assisted process. Avoid comparing polished assisted examples with an unusually poor baseline.
Choose a comparison design deliberately; each answers a different question with different limits.
| Design | How it works | What it can support | Main limitation |
|---|---|---|---|
| Concurrent control | Cases or people assigned at random to assisted and current processes during the same period | The strongest within-pilot comparison of effect | Assignment must be agreed in advance and kept; contamination between groups |
| Staggered rollout | Groups start using the system at different times; later starters serve as comparison for a while | Comparison under real rollout conditions | Groups may differ for reasons unrelated to the system |
| Before and after | The same team measured before and after the change | A descriptive change in results | Seasonality, staffing, case mix, and other changes can explain the difference |
| Simpler alternative | A template, better search, or deterministic rule tested alongside the assisted process | Whether the AI component adds value beyond simpler fixes | Requires building the alternative well enough to be fair |
Published research shows why the design must be recorded. In the support study cited elsewhere in this guide, a larger staggered rollout followed a small randomized pilot, and the authors analyze rollout effects with difference-in-differences methods; they also report lacking the original randomized pilot's control-group identities, so the pilot analysis compares identified treated workers with other untreated colleagues. Brynjolfsson, Li, and Raymond, version 2 The point for a pilot owner is narrow: say which comparison was actually run, and describe its limits in the result.
When feasible, assign cases or people to the two processes at random, record the unit of assignment (case, agent, or team), and analyze at that unit. A comparison whose assignment was chosen by hand supports a descriptive result only, because easy cases drift toward whichever process someone favors. A staggered rollout is stronger when the start order is also drawn at random. Account for learning and case mix. If the pilot uses a before-and-after comparison, document other changes that could explain the outcome. A descriptive improvement can still be useful, but label attribution limits.
Where this step fails. Volunteers who adopt early are compared with everyone else, so enthusiasm is measured instead of the system. The simpler alternative is skipped, so improvements that better search or templates would have delivered are credited to generation.
Step 3: Build the case set
For whoever runs the pilot
Collect ordinary cases, important variants, difficult cases, and cases outside scope. Use only material the organization is permitted to use. Keep development examples separate from held-out evaluation cases. For time-dependent work, ensure later information does not leak into an earlier decision.
Keep two sets apart and report them separately:
- A challenge set chosen to exercise hard behavior: missing evidence, conflicting sources, permission boundaries, embedded instructions, and cases with no valid answer. Its error rate says nothing about everyday traffic, because it is built to be difficult.
- A representative sample drawn from the eligible population by an agreed method, which supports statements about ordinary performance.
Prevent leakage in four ways. Reserve held-out cases before any prompt or retrieval tuning. Split near-duplicates by family so one variant is not in development and another in evaluation. For historical replay, use only information available at the time of the original decision; a later resolution note or postmortem must not answer an earlier case. Version the case set so a later change to an answer key can be traced.
Sample size depends on variation, the decision, and the rarity of important failures. There is no universal number of examples that makes a system safe or proves ROI. A small sample with no observed critical failures does not establish that the true critical-failure rate is zero. For illustration, if independent cases each have a fixed 1% failure probability, observing zero failures in 20 cases has probability 0.99^20, about 82%. Those assumptions are illustrative; correlated cases need different analysis. A rough guide runs the other way, known as the rule of three: with zero failures in n independent cases, the 95% upper bound on the failure rate is about 3 ÷ n. After 20 clean cases the true rate could still be about 14%; after 120, about 2.5%. Seek appropriate statistical expertise when the decision depends on estimating rare events or small effects.
A clean test run says little about rare failures
- True failure rate 1%
- True failure rate 5%
Show the data
| True failure rate | Cases tested | Chance of zero observed failures |
|---|---|---|
| 1% | 20 | 81.8% |
| 1% | 50 | 60.5% |
| 1% | 100 | 36.6% |
| 5% | 20 | 35.9% |
| 5% | 50 | 7.7% |
| 5% | 100 | 0.6% |
Where this step fails. Cases are collected from whatever was easy to export, so the hard categories are missing. Development examples reappear in evaluation under a different ID. The case library offers authored fixtures for rehearsing the scoring; they are not a representative sample of anyone's traffic.
Step 4: Define acceptance before reviewing results
Use a small set of explicit requirements. For support drafting, these might include source-supported material claims, no cross-account information, acceptable resolution quality, and a meaningful reduction in complete handling effort. The actual tolerances belong to the process owner and context.
Do not choose thresholds after seeing which ones the system passes. If a requirement changes for a legitimate reason, record why and evaluate against the revised requirement transparently.
| Decision dimension | Evidence needed |
|---|---|
| Useful output | Qualified review against a defined rubric |
| Workflow improvement | Comparable end-to-end effort or outcome measure |
| Acceptable consequences | Errors by severity and scope handling |
| Operational feasibility | Review load, integration reliability, and recovery |
| Economic plausibility | Observed inputs replacing key assumptions |
Separate requirements into two kinds. Mandatory requirements are pass or fail: an unauthorized action, a disclosure across an access boundary, or a mishandled excluded case stops the pilot regardless of averages. Performance requirements are thresholds on rates and effort, set with their denominators. Do not average a mandatory failure into a score.
Define severity categories with examples drawn from the workflow, and agree who classifies a disputed case. Each workflow chapter in this guide proposes a severity table; use it as a starting point, not a standard. Count a correct abstention, such as asking for missing information, as a correct outcome rather than a failure.
Where this step fails. Thresholds are written as single numbers without denominators, so "95% accurate" later turns out to exclude the hard cases. The denominator is set too small for the threshold: 57 passes in 60 cases is 95% observed, but the exact 95% interval around it runs from about 86% to 99%, so 60 cases cannot show that a true rate is at least 95%. Size the sample to the tolerance, not the tolerance to the sample. Severity is assigned after the fact by the person who built the system.
Step 5: Run in stages
For whoever runs the pilot
First evaluate offline cases. Then run in shadow mode alongside the existing process where that is useful. Move to supervised use only when the earlier evidence supports it. Each stage should answer a specific uncertainty; do not create stages merely as ceremony.
Each stage answers one uncertainty
Stage 1
Offline cases
Does it produce acceptable outputs on known cases, including hard ones?
Evidence: Held-out cases scored against agreed criteria
Stage 2Where useful
Shadow
Does it hold up on live inputs without affecting anyone?
Evidence: Outputs compared with the existing process
Stage 3
Supervised use
Does the whole workflow improve, with review, fallback, and real workload?
Evidence: End-to-end effort, errors by severity, review load
Stage 4
Decision
Expand within the tested scope, revise, or stop?
Evidence: Results against criteria set before the trial
At any stage: a failed criterion returns the pilot to the previous stage or stops it, and the existing process remains the fallback.
Keep the scope and permissions bounded. Ensure employees know how to fall back and where to report problems. Include realistic workload and the time needed to inspect evidence.
For each stage, write down in the charter what must be true to enter it and what evidence ends it. Offline replay ends when held-out cases have been scored against the agreed criteria and access problems are fixed. Shadow mode, if used, ends when outputs on live inputs have been compared with the existing process without affecting anyone. Supervised use ends at the observation window's close, with all assigned cases followed through. A failed mandatory requirement at any stage sends the pilot back or stops it.
Where this step fails. Stages are run in sequence but the evidence that should gate them is not collected, so supervised use begins because the calendar says so.
Step 6: Plan the measurement
For whoever runs the pilot
Decide in advance what is recorded for every case, and record it for the comparison group too.
- Effort to completion, including review, correction, failed attempts, follow-up, and escalation, over the same observation window for both groups.
- Errors by severity, counted both before release (caught by reviewers) and after (reaching a customer or record). A strong reviewer can hide a weak system if only released errors are counted.
- Outcome measures tied to the unit of work, such as resolution, repeat contact, or order correctness.
- Operational events: outages, invalid outputs, timeouts, and fallback use.
- Configuration: model version, prompt revision, source snapshot, and settings, so a result can be tied to what produced it.
Report every rate with its numerator and denominator, and keep eligible, attempted, and completed counts distinct. The evaluation scorecard lists case-record fields, and the pilot kit links the companion workbook, whose Evaluation sheet keeps cohorts separate.
Step 7: Record the result honestly
For whoever runs the pilot
Report the number of eligible cases, attempted cases, accepted outputs, escalations, failures, and exclusions. Include human effort and downstream rework. Explain which cases benefit and which do not.
Preserve disagreements among reviewers where they reveal ambiguity in the task. Do not silently remove difficult cases from the denominator. Treat system outages and invalid outputs as part of the operating result when they would affect users.
A negative or mixed result is a finding. Record what was learned about the sources, the interface, the review burden, and the simpler alternative. A pilot that shows better search delivers most of the value without generation has answered its question.
Step 8: Make the expansion decision
Expansion requires more than a positive average. Confirm that mandatory requirements were met, handling of critical failures is understood, review can scale, and the business case survives observed costs. Identify what remains untested at larger volume or with a different population.
If the system falls short, choose a bounded revision with a clear hypothesis, a simpler alternative, deferral until a named dependency is resolved, or stopping. Record a decision owner and the next evidence required. This makes the pilot useful even when deployment is not justified.
Write the decision memo in a fixed shape so decisions can be compared:
| Section | Contents |
|---|---|
| Decision | Expand within tested scope, revise and retest, simplify (adopt the simpler alternative), defer, or stop |
| What was tested | Population, comparison design, stages completed, configuration |
| Mandatory requirements | Each one, passed or failed, with the cases involved |
| Performance | Rates with denominators, by category and cohort |
| Economics | Observed inputs replacing assumptions; capacity and cash kept separate |
| What is untested | Volumes, populations, or conditions not covered |
| Owner and next evidence | Who decides next, and what observation would change the decision |
Worked charter: the support-reply pilot
The following applies the procedure to the illustrative support pilot in the support-reply chapter. Every value is one of that chapter's illustrative assumptions; no result is reported, because no pilot has been run.
| Field | Entry |
|---|---|
| Workflow owner | Support manager; knowledge owner signs off on sources; engineering owns retrieval boundaries and fallback |
| Eligible population | Documented product questions; billing disputes, account access changes, contractual commitments, and security incidents excluded. Illustrative eligible share: 60% of 5,000 monthly requests |
| Current process | Agent researches and drafts; unit of work is a resolved request, followed through a fixed observation window |
| Proposed change | Read-only assistant retrieves permitted sources and drafts; the agent approves and sends with existing permissions |
| Business hypothesis | Assisted effort falls from 8 to 5 minutes including review; value is realized only through a named avoidable expense |
| Quality requirements | No cross-account content or unauthorized action (critical); no wrong-version instruction, invented action, or false promise (major) |
| Evaluation design | Offline replay of an illustrative 120-case coverage plan (60 straightforward, 20 ambiguous, 15 stale or conflicting, 15 excluded actions, 10 adversarial), then a supervised trial with comparable work assigned to assisted and current processes |
| Comparison | Current process, and improved search and templates, on the same categories |
| Stop conditions | Any critical failure pauses the integration; a draft-quality failure pattern triggers review |
| Decision date | After the observation window closes for the last assigned case |
The chapter's illustrative offline gate is zero observed critical failures, every excluded-action case routed correctly, and at least 95% of ordinary answerable cases free of major errors. The figure in Step 3 is the reminder that a clean result on a small set bounds risk only loosely, and the same holds for the 95% gate: on 60 straightforward cases, a 57-of-60 result meets it while leaving the true rate anywhere from about 86% to 99%. The decision memo should say so, or the sample should grow. The economics, including how review time and adoption move the result, are in the chapter and the economics worksheet.
Related chapters: evidence, choosing a workflow, economics worksheet, human review, model selection.