Use pseudonymous case IDs and references to authorized evidence. Do not copy sensitive ticket, client, or order contents into a broadly shared workbook. Keep the original in its approved system.
Start with the fictional evaluation cases: thirty-six input packets and reviewer answer keys, four for each workflow, covering ordinary work, missing evidence, permission boundaries, and downstream failures. No model outcomes are prefilled.
Case record
| Field | Your entry |
|---|---|
| Case ID, workflow, and configuration ID | To enter |
| Cohort: representative, synthetic ordinary, or challenge | To enter |
| Source packet IDs, versions, and permissions | To enter |
| Expected disposition and required facts | To enter before testing |
| Observed disposition and output reference | To enter after testing |
| Source support: pass, fail, unresolved, or not applicable | To enter with reason |
| Authority check: pass, fail, unresolved, or not applicable | To enter with reason |
| Worst error: none, minor, major, critical, or unreviewed | To enter with description |
| Active human minutes, excluding separately counted rework | To enter |
| Rework and follow-up human minutes | To enter; zero only if observed |
| Reviewer and review date | To enter |
| Final outcome, unresolved actions, and owner | To enter |
The workbook's Total minutes adds active and rework minutes only when both are numeric and nonnegative. It leaves an incomplete record unavailable. Rows 8–57 are prepared for 50 cases, enough for the full case library with room for authorized examples, and the Workflow column accepts all nine workflows. For a larger set, extend the same columns, copy the total formula and validations, and verify the new rows before use. The workbook's Evaluation sheet holds case ID, workflow, configuration ID, cohort, expected and observed disposition, source support, authority check, worst severity, active and rework minutes, and reviewer. Keep the other fields (source packet IDs, versions, and permissions; required facts; output reference; reasons and descriptions; review date; and final outcome, unresolved actions, and owner) in a controlled case record referenced by Case ID.
Workflow-specific scoring
| Workflow | Check before calling the case correct | Correct behavior when evidence is missing |
|---|---|---|
| Support | Claims match permitted sources and product version; no invented action; relevant follow-up recorded | Clarify or escalate without inventing product behavior |
| Proposals | Mandatory requirements accounted for; credentials reusable; scope, staffing, dates, and price approved together | Flag the gap and owner; return to bid decision when needed |
| Order intake | Customer, SKU, quantity, unit, price basis, and ship-to validated; revision and submission handled correctly | Block the affected operation and seek resolution |
| Incident triage | Every statement in the brief carries a source and timestamp; unavailable or lagging sources named; no alert, command, or deployment action attempted | State which sources are missing and leave the hypothesis open |
| Onboarding | Plan steps match the entitlement record, edition, and released guides; no date, price, scope, or action promised | Hold the affected step and name the record or owner needed |
| Knowledge retrieval | Only content the test account may access influences the answer; current method preferred over superseded; ownership stated as recorded | Answer identically for walled and unknown clients; route access questions to the owner |
| Meeting follow-up | Decisions, proposals, and open questions kept distinct; client recap built only from shareable items; critical numbers checked against the record | Mark the item unconfirmed and ask the accountable person |
| Forecasting | Benchmarks compared on the same history; stockout and one-off periods flagged, not learned; proposed order traceable to the policy inputs | Widen the range or withhold the proposal and flag the data gap |
| Catalog enrichment | Values carry source spans; conversions follow the schema rule; identity and pack level confirmed; safety attributes left to their owner | Leave the attribute empty and route the conflict or gap to the steward |
“Pass” for source support does not imply authority compliance. A correct extraction does not establish correct order submission. A corrected final artifact does not erase a major error in the initial output. Record those distinctions in the case evidence.
Severity and aggregation
Agree on severity definitions before testing. A starting rubric is: critical for unauthorized disclosure or consequential unauthorized action; major for materially wrong facts, quantities, promises, or commitments; minor for wording defects without consequential factual error. Adapt this rubric to the workflow and record that version with the configuration.
Use Representative for an authorized sample of actual traffic, Synthetic ordinary for invented routine cases, and Challenge for deliberately hard cases. Record whether challenge inputs are synthetic or authorized real examples in the controlled case record. Keep all three cohorts separate. Report each numerator and denominator, exclusions, and unresolved outcomes. Do not combine the workbook's pass/fail fields into a universal AI score. Preserve critical failures even if average quality is high.
Have a second reviewer assess a subset independently and resolve disagreements in the rubric. Decide sample sizes and quality tolerances before observing results. A small set with no observed failures does not establish zero risk.
Return to the pilot kit or evidence checklist.