Evidence: what counts as a demonstrated result

A demonstration shows that a system can produce a result under the conditions shown. A deployment shows that someone put a system into use. Neither fact, by itself, establishes dependable performance or a financial return.

Useful evidence connects the claim to a defined population, measurement method, time period, and comparison. The stronger the decision you want to make, the more closely the evidence should match the workflow and conditions you will operate.

This chapter is the guide's working method made explicit. The research examples below are real studies, described within the limits their authors state. The illustrative numbers are invented and labeled as such.

Five kinds of claim

Before judging the support for a claim, say which kind of claim it is.

Each claim needs more than the one before it. A capability claim can rest on a demonstration. A performance claim needs a defined case set and a denominator. An effect claim needs a comparison. An economic claim needs a realization mechanism, which the economics chapter treats in detail. A transfer claim needs a reason to think the population, task, and system resemble yours. A vendor deck that moves from "it can" to "it will save you money" in one slide has skipped three steps.

Match the evidence to the claim

There is no single ranking that answers every question. Technical documentation may be the best source for a supported API parameter. A local experiment may be the best source for review effort. A field study can inform expectations while remaining a poor substitute for the data needed to authorize a particular integration.

The guide uses the same distinction in its own sourcing. Where it cites vendor documentation, as in the knowledge retrieval chapter's permission-filtering sources, it labels the citation as vendor guidance and uses it for what the product is documented to do, not for whether any deployment does it correctly.

Ask about the denominator

“Most answers were correct” is incomplete without knowing which answers were counted. Were difficult cases excluded? Were only accepted suggestions measured? Did the evaluation count each field, each document, or each completed transaction?

A system that correctly extracts many easy fields can still fail on a small number of critical values. Report severity and case mix, not only a blended success rate. Include the cases that escalated or produced no output; those outcomes affect labor and usefulness.

Three denominator questions to ask:

• Which cases were eligible, and which were evaluated? In the illustrative figure, 54 correct results are 90% of the 60 cases evaluated and 54% of the 100 requests that arrived. Both numbers are true; only one describes the workload.
• What unit was counted? Field-level accuracy overstates document readiness: if fields are independently correct 98% of the time, a 20-field document is fully correct about 67% of the time. The document extraction chapter shows this compounding in a figure. The calculation assumes field errors are independent; check that assumption on your own documents.
• Whose behavior is in the denominator? A rate measured only on the cases where staff accepted a suggestion leaves out the rejected ones, which carry the review cost. The support chapter counts rejected drafts among all assisted cases for this reason.

Separate association from effect

A team may become faster after adopting a tool while also receiving training, handling easier work, or changing staffing. An observed improvement is real as a description, but attributing all of it to AI requires a comparison that addresses those alternatives.

Three studies show how design shapes what a result can mean.

A staggered rollout at one company. Brynjolfsson, Li, and Raymond study 5,172 support agents and report a 15% average increase in issues resolved per hour, with effects that differ across workers. The larger rollout was staggered and followed a small randomized pilot; the authors analyze it with difference-in-differences methods, and report lacking the original pilot control-group identities. Generative AI at Work, version 2 That design supports a stronger reading than a before-and-after comparison, for that deployment and population. It does not establish what a different support team, product, or model would see, and some details of how its productivity outcome was constructed are not public.

A small randomized trial. METR randomly allowed or disallowed early-2025 AI tools across 246 tasks completed by 16 experienced open-source developers in mature projects. Allowing the tools increased completion time by 19%; the developers had forecast a 24% reduction beforehand and estimated a 20% reduction afterward. Becker, Rush, Barnes, and Rein, arXiv 2507.09089 Randomization makes the effect estimate credible for that setting. The sample is small and the setting specific, and the authors state that they do not show AI fails to speed up most developers. The finding that matters for this chapter is the gap between what people believed and what was measured.

A large survey. DORA's 2025 report surveyed nearly 5,000 technology professionals: 90% use AI in their work and more than 80% believe it increased their productivity; adoption is associated with higher software delivery throughput and higher delivery instability. DORA 2025 report The report states that it speaks in terms of comparisons rather than causal effects, and the productivity measure is self-reported.

None of these settles the others. The survey is large and observational; the trial is small and causal; the rollout study is causal-leaning for one company. The engineering leaders guide shows them side by side. Read together, they argue for local measurement of effort, not for adopting any one study's number.

For a practical pilot, document what else changed and compare similar work. Where feasible, use an agreed allocation between the current and assisted process. Where that is not feasible, describe the limits of a before-and-after comparison rather than hiding them.

Do not translate a task-level effect directly into organization-wide savings. Improved drafting affects only part of a job; staffing and cash outcomes depend on what happens to the released time.

Self-report is a measurement too

Surveys and interviews measure what people say, which is evidence about perception, adoption, and experience. They are weaker evidence about effort or output. The METR trial's forecast and estimate, set against its measured result, show why a pilot that asks staff whether they feel faster has not measured whether they are.

Use self-report for what it measures well: where people find the system useful, what they distrust, which cases they route around it. Measure effort and quality directly.

The same care applies to official statistics. The Census Bureau's Business Trends and Outlook Survey asks firms whether they used AI "in any business function" in the last two weeks, a wording adopted in November 2025 that is not comparable with earlier cycles. Census Bureau, May 2026 Its estimates are shares of firms with standard errors, and a 90% confidence interval spans 1.645 standard errors either side. BTOS methodology The industry guides present one cycle's estimates with their standard errors, from which a reader can form that interval, and none reads a single cycle as a trend.

Record the version of the evidence

For your analyst

Model behavior, product configuration, and research papers can change. Preserve the source version and the conditions behind a claim. A later paper may revise its sample or estimates; combining an old sample count with a new percentage produces a claim no source actually made.

The support study above is an example. Its earlier versions report different figures from version 2, so this guide cites version 2 alone and never mixes its numbers with older ones. The consulting experiment is cited through its 2026 journal article rather than the earlier institutional summary.

For system evaluations, retain model identifier, prompts, tools, source collection, permissions, and relevant settings. “We tested the assistant” is not enough to reproduce a result after the system changes.

Use explicit labels

For your analyst

The guide distinguishes editorial analysis, illustrative assumptions, reported deployments, and observed measurements. A hypothetical calculation can teach a useful method, but it should never acquire the appearance of a case study because it contains realistic numbers.

The guide uses one controlled list of evidence statuses, applied to every chapter:

• Editorial analysis: original reasoning and proposed practice. Most chapters carry this label; any worked example inside them is still illustrative analysis.
• Illustrative analysis: reasoning and arithmetic with invented inputs. Every worked example in this guide is in this category, and the workflow chapters built around one carry the label.
• Provider documentation: specifications a provider publishes about its own product, checked on a stated date and not tested by the guide.
• Vendor-reported deployment: a provider's account of a customer result. Identify who measured it and whether the customer corroborated it.
• Independently documented deployment: an account from a party without a commercial stake, with a stated method.
• Neuronify-observed data: data Neuronify measures directly, such as the Hiring Index's daily snapshot of posted roles and pay ranges. It describes job postings, not AI outcomes, and carries its method and limits like any other source.
• Neuronify-observed pilot: a pilot Neuronify measured directly. None exists yet in this guide. For a reader, the equivalent is your own measurement, on your work, with your comparison.

When interpreting research, separate the authors' finding from your own inference. When a source supports only part of a paragraph, place the citation next to that part. The guide's figures carry the same labels on their frames, so a chart of invented numbers cannot pass for a measurement.

Reading someone else's evaluation

A vendor, a consultant, or an internal team may hand you a results table. Before using it, ask what would have to be true for the numbers to describe your situation. The questions below are proposed practice, drawn from the evaluation designs in the workflow chapters.

• Were the cases held out? Results on examples used to tune prompts or select configurations overstate performance on new work. The model selection chapter asks for held-out cases reserved before tuning.
• Could the answer leak in? A historical replay that lets the system see a later resolution, a postmortem, or a corrected master record is grading against the answer key. The incident triage and forecasting chapters describe the leaks specific to their data.
• Were difficult cases included? Ask for the challenge set separately from ordinary traffic, and for the counts in each.
• Were errors weighted by consequence? One invented account action matters more than many awkward sentences. Ask for errors by severity, not one accuracy figure.
• Was the configuration recorded? A result belongs to a model version, prompt, retrieval method, source snapshot, and settings, not to a product name.
• Were repeated runs reported? Where output varies between runs, a result from the best attempt hides the cost and inconsistency of the others.
• Who scored it, and did scorers agree? A rubric applied by one person, or by the system being evaluated, needs a check.

A report that answers all of these may still not transfer to your work. One that cannot answer them is a demonstration, whatever its table says.

A claim record

For your analyst

For each material claim, capture the wording, chapter, source, publication and access dates, evidence type, population, metric, comparison, limitations, and review status. Record missing information explicitly: a blank field reads as checked, so write down what you have not verified.

The review status field is where honesty about partial work lives. A source checked only at its landing page should say so; so should one read through a secondary summary because the primary text was not available, or a working paper read only in abstract. Those records are still useful; they say exactly how far the support goes.

A reusable question set is: What happened? To whom? Compared with what? Measured how? For how long? Under which system version? Who reported it? What would make it fail to transfer here?

Make negative evidence useful

A pilot that saves no time because review is expensive identifies a real constraint. A model that succeeds on ordinary cases but fails on amended documents defines a boundary for eligibility. Publish those boundaries as part of the result.

Several of the guide's own worked examples end in negative or marginal results, on purpose. Incident triage's base case barely clears its recurring budget and every stress case falls below it. Onboarding guidance is positive as capacity and negative as cash at its stated realization. The METR trial measured a slowdown. Each of those results tells a reader something a success story would hide.

The practical objective is a decision readers can assess. It is acceptable to conclude that evidence supports a narrow supervised use, that a business benefit remains unmeasured, or that the simpler process is preferable. Precision about limits makes the resource more useful than an unqualified success story.

Related chapters: pilot design, model selection, choosing a workflow, economics, engineering leaders.