Select a model as part of a system that completes a defined task. A benchmark, provider brand, or impressive demonstration can help identify candidates, but it cannot establish which configuration works best for your documents, users, permissions, and operating costs.
The output of selection should be a dated decision record: the accepted configuration, the cases it was tested on, the requirements it met, and the conditions that would trigger reconsideration.
This chapter describes a method. It contains no model ranking, provider price, or benchmark result. The worked figures are invented arithmetic. Provider specifications such as prices, limits, and model versions change often, so check each one at its source on the day you evaluate and record that date with the result.
Write requirements before choosing candidates
Specify the inputs, output form, acceptable errors, latency, throughput, data constraints, and action boundary. Distinguish hard requirements from preferences. If a deployment arrangement cannot satisfy a required data condition, better writing quality does not compensate for that gap.
Describe the task in operational terms. “Extract these order fields with supporting evidence and preserve missing values” is more testable than “understand purchase orders.” “Draft a supported answer or escalate” is more useful than “be helpful.”
A requirements sheet for selection can be short. The point is to decide, before any candidate is run, which conditions are gates and which are trade-offs.
| Requirement | What to state | Gate or trade-off |
|---|---|---|
| Task and output | Input types, output schema or format, what counts as a complete answer | Gate for format; trade-off for quality above the minimum |
| Errors that matter | Which mistakes are critical, major, or minor in this workflow | Gate for critical errors; trade-off for the rest |
| Scope behavior | What the system must do with missing evidence, conflicts, and excluded requests | Gate where a wrong answer has consequences |
| Data handling | Which data may be sent where, retention, residency, and contractual limits | Gate |
| Access and action | What the system may read, and whether it may act | Gate |
| Latency | Acceptable response time for the user or batch window, including slow cases | Gate for a hard ceiling; trade-off within it |
| Throughput | Peak volume and how far requests can be queued | Gate if peaks cannot be absorbed |
| Cost ceiling | Maximum recurring cost per completed unit of work, review included | Gate at the ceiling; trade-off below it |
| Operability | Monitoring, logging, version control, and fallback the team can actually run | Gate where a missing control is unacceptable |
Write the cost ceiling in the workflow's own unit, such as a resolved request or an approved order, not in tokens. A ceiling stated per token says nothing about how many attempts, retries, and reviews a completed unit needs.
Treat the configuration as the unit of comparison
For your technical lead
A model used with retrieval and validation may outperform another model used without them. Record the complete configuration so the comparison is meaningful: model version, prompt, source set, retrieval method, tools, validation, and review process.
Changing any one element can change the result. A better retrieval step can make a less capable model adequate; a stricter validation rule can turn a fluent but careless configuration into one that escalates correctly; a poorly designed review screen can double the minutes a reviewer spends on the same output. When a comparison changes two elements at once, the result cannot be attributed to either.
Keep a configuration record for each candidate:
| Element | Record |
|---|---|
| Model | Exact model identifier as called, provider, and date tested |
| Instructions | Prompt or instruction revision, with its version identifier |
| Evidence | Source collection snapshot and retrieval method, if any |
| Tools | Which tools the system may call, and with what permissions |
| Settings | Generation settings that affect variation and length |
| Validation | Deterministic checks applied to the output |
| Review | Who reviews what, and the interface they use |
Include the current process and a simpler alternative
Include the current process and a simpler alternative. A structured import, rule-based extractor, or search interface may meet the need with less maintenance. If one candidate receives extensive tuning and another receives none, state that difference rather than attributing the entire result to the underlying model.
The simpler alternative is not a formality. Google's machine-learning guidance recommends starting with a simple system and establishing metrics before adding complexity; that is engineering guidance, not evidence about any particular workflow. Google, Rules of Machine Learning In this guide's workflow chapters, the comparison with improved search, templates, or deterministic rules is what shows whether the model adds value beyond better information access. If it does not, the model is a cost without a benefit.
Construct a representative evaluation set
For your analyst
Use permissioned cases covering typical work, important variants, difficult cases, missing information, and requests outside scope. Reserve held-out cases before tuning. Avoid near-duplicate documents split between development and evaluation when that would make results misleading.
For each case, define what an acceptable answer requires and which errors matter. Some tasks have an exact reference value; others need a rubric and qualified reviewers. Record disagreement rather than forcing an artificial single answer when the domain permits several valid outcomes.
Three practices keep the set honest:
- Separate the ordinary sample from the challenge set. A sample drawn from real traffic estimates how the system will behave on typical work. A deliberately difficult set tests specific risks. Combining them into one score produces a number that describes neither.
- Freeze the held-out cases before tuning. Any case used to adjust instructions, retrieval, or validation becomes a development case. If a held-out case is inspected and the configuration changed in response, replace it.
- Use time-correct inputs for historical replay. When replaying past work, supply only the documents and context available at that time. A later resolution note or corrected record in the evidence lets the system answer from the future.
Have two qualified reviewers score a subset independently before relying on aggregate results. Where they disagree, decide whether the rubric is ambiguous or the case has more than one acceptable answer, and revise the rubric before scoring the rest. The companion case library shows the form these answer keys can take.
Run repeats where outputs vary
Where generation varies between runs, test enough repeated cases to understand whether important failures recur. Do not choose a system based on its best retry while ignoring the cost and inconsistency of the others.
Report the distribution, not only the best result. For each case run several times, record how many runs were acceptable and whether any produced a critical error. A configuration that is right on three runs and wrong on the fourth has a failure rate a reviewer will meet in production. Count retries in cost and latency: a system that needs two attempts to produce one acceptable output costs twice the attempt price and takes at least twice as long.
Use a decision table
| Dimension | What to record | Decision use |
|---|---|---|
| Task quality | Acceptable completed outcomes and errors by severity | Establish whether the workflow is usable |
| Scope behavior | Handling of missing evidence and excluded cases | Check whether escalation is dependable |
| Review effort | Human time including correction and follow-up | Determine full workflow cost |
| Latency | Distribution of completion times, including slow cases | Match the user experience requirement |
| Reliability | Failed calls, invalid output, retries, tool errors | Estimate operational burden |
| Cost | Total recurring cost per acceptable outcome | Compare economic feasibility |
| Deployment fit | Required data handling, access, and integration conditions | Apply non-negotiable constraints |
Do not average a hard failure into a weighted score. A configuration that violates a required access boundary is ineligible even if it is inexpensive and fluent.
Report each dimension with its denominator: critical errors among cases evaluated, correct escalations among cases requiring escalation, and the slowest tenth of response times, not only the average. A weighted total hides exactly the trade-offs the process owner needs to see.
A hypothetical comparison
Suppose configuration A costs $0.04 per attempt and produces 90 acceptable outcomes from 100 attempts without a retry. Configuration B costs $0.02 per attempt and produces 70. In this simplified example, model cost per acceptable outcome is about $0.044 for A and $0.029 for B.
That calculation still does not select a winner. If B requires much more review, misses critical cases, or fails a quality requirement, the lower model cost is irrelevant. Include human effort, retrieval, tool calls, retries, and operation before drawing a financial conclusion. These figures are illustrative and are not provider prices or benchmark results.
Requirements are gates; everything else is a trade-off
1. The configuration
- Model version
- Prompt
- Source set
- Retrieval method
- Tools
- Validation
- Review process
2. Pass or fail
- Required access boundary
- Data handling and deployment conditions
- Critical cases handled
Any failure: ineligible. Not a weight.
3. Compare those that pass
- Acceptable outcomes and errors by severity
- Escalation on missing evidence
- Review effort, including correction
- Latency, including slow cases
- Reliability: failed calls, retries
- Total cost per acceptable outcome
Illustrative: the denominator matters
A: $0.040 per attempt → $0.044 per acceptable outcome (90 of 100)
B: $0.020 per attempt → $0.029 per acceptable outcome (70 of 100)
Review effort and critical misses are not yet included.
Show the data
| Configuration | Cost per attempt | Acceptable outcomes per 100 | Model cost per acceptable outcome |
|---|---|---|---|
| A | $0.040 | 90 | $0.044 |
| B | $0.020 | 70 | $0.029 |
Add review and rework
Extend the same example with two further invented assumptions. A reviewer spends one minute checking every attempt, and a person spends four more minutes completing each request the system did not handle acceptably. Reviewer time is valued at $45 an hour. All 100 requests are then completed in both configurations, some by the system with review and the rest by people.
| Per 100 requests | Configuration A | Configuration B |
|---|---|---|
| Model cost | $4 | $2 |
| Checking every attempt (100 minutes) | $75 | $75 |
| Completing unacceptable attempts | 10 × 4 min = $30 | 30 × 4 min = $90 |
| Total | $109 | $167 |
| Cost per completed request | $1.09 | $1.67 |
Once people's time is counted, the order reverses. Model cost is under 4% of the total for A and about 1% for B; the difference between the configurations is almost entirely review and rework. Change the assumptions, for example a faster check or a cheaper fallback, and the result can move. The point is the method: compare configurations on the cost of a completed unit of work, with people's effort included, and only after both have passed the requirements.
Treat deployment as a requirements question
For your technical lead
Hosted access through a provider, a managed service inside a cloud account, and a model run on infrastructure the organization controls are different deployment arrangements, and the choice among them starts from the requirements sheet rather than from the model. The questions are the same for each arrangement: where the data goes and who can access it, which contractual and retention terms apply, how the system behaves at peak volume, who applies updates and when, and what the team must operate itself.
Open-weight models, whose weights can be run on infrastructure the organization chooses, add options and obligations. They can satisfy a data-handling or isolation requirement that a hosted service cannot, and they place serving, scaling, monitoring, security updates, and evaluation of each new release on the team. The open-weight deployment chapter works through these arrangements, the responsibilities each places on the team, and how their costs behave; treat any comparison here as a list of questions to answer, not a recommendation.
Whatever the arrangement, run the same evaluation set and record the same configuration details. A configuration that changes deployment has changed, and the evaluation applies only to what was tested.
Make the choice and preserve the evidence
Choose among configurations that meet the mandatory requirements. Explain the remaining tradeoffs in plain language. A slower system may be suitable for overnight document work and unsuitable for an interactive conversation. A lower-cost system may be sufficient for classification while a different configuration handles difficult drafting.
Record limits on eligible cases and the fallback. Keep the evaluation set and result summary available to the process owner, not only the engineering team. A decision record should distinguish observed results from expectations about scale.
A proposed decision record contains:
- the requirements sheet, with gates and trade-offs marked;
- each candidate's configuration record;
- the evaluation set's composition, the held-out portion, and how answer keys were reviewed;
- results by dimension with denominators, separated by ordinary sample and challenge set;
- the choice, the trade-offs accepted, and who accepted them;
- eligible cases, excluded cases, and the fallback;
- the re-evaluation triggers below, and the date of the next scheduled review.
Revisit after meaningful changes
For your technical lead
A model, prompt, source collection, integration, or operating population can change accepted behavior. Define which changes require re-evaluation. Track the exact version tested and the date of any pricing assumption. If a provider uses an alias that can move between versions, record that operational uncertainty.
Provider documentation shows why the identifier matters. Anthropic's vendor documentation states that for its recent models a dateless model ID maps to a single fixed snapshot, that updated versions ship under new IDs, and that earlier models also have convenience aliases resolving to the most recent dated snapshot. The same page notes that serving infrastructure around a model can change and occasionally produce minor differences in observable behavior even when the model ID and weights have not. Anthropic, Model IDs and versioning Conventions differ by provider, so record what the identifier you call actually guarantees.
| Change | Proposed response |
|---|---|
| New model identifier or a provider-announced update | Rerun the held-out set before switching |
| Prompt, retrieval, tool, or validation change | Rerun the affected cases; rerun all if the change is broad |
| Source collection change, such as new product version | Rerun cases touching the changed sources |
| New population, such as a new customer segment or document type | Extend the evaluation set before extending eligibility |
| Unexplained shift in production review rates or errors | Investigate, then rerun; infrastructure changes can alter behavior without a new ID |
| Price change | Recompute cost per completed unit; re-evaluation needed only if a ceiling is crossed |
The guide will maintain version-specific profiles separately from this method. Until a profile has dated sources and task evidence, it should not be presented as a current recommendation.
When a selection exercise is not worth running
Skip a formal comparison when a deterministic method already meets the requirements, when volume is too low for the result to matter, or when no one can define an acceptable output. In the first case, use the deterministic method. In the second, choose a reasonable configuration, keep it read-only, and measure. In the third, the problem is not model choice: define the work first, using the workflow-change and choosing a first workflow chapters.
Related chapters: evidence, pilot design, retrieval, document extraction, and engineering leaders.