Build, buy, or configure

Use this playbook when a workflow is worth improving and the remaining question is how to source the system: use a feature you already have, configure a packaged application, build on a model provider's API, build with open-weight models, or commission a services firm. The procedure compares those options by the effort and control each involves over the whole life of the system, not by the first invoice. It produces a written sourcing decision with its assumptions, its exit plan, and the conditions that would reopen it.

This chapter is the sourcing decision. The questions to put to a specific vendor, and the evidence to ask for, are in the companion vendor diligence playbook. Run this one first: diligence on a vendor is wasted if the workflow does not need what that vendor sells.

The options, procedure, and controls below are proposed practice. The worked comparison uses invented effort and cost assumptions; they are not prices, quotes, or measurements of any product.

The procedure at a glance

Steps 1 to 5 need no running system; their elapsed time depends on how quickly vendors and legal answer the gate and exit questions. Step 6 is the only one that needs a running system, and it can be skipped when step 2 leaves a single option standing.

The five options

Each option moves different work to different people. None removes the work of deciding what good output is, checking it, and keeping the sources it depends on current.

Two things in the table deserve attention before any estimate.

The evaluation set is yours under every option. Whoever builds or sells the system, the cases that represent your work, the definition of an acceptable outcome, and the record of how the system performed on them belong to you. An organization that relies on a vendor's demonstration instead has no way to compare options, check a vendor's update, or leave. Google's engineering guidance for machine learning puts establishing metrics and a simple first system ahead of model work, and that applies whether or not you write the model code yourself. Google, Rules of Machine Learning

Model change is a cost you schedule when you choose the model, and a risk you must detect when a vendor chooses it. When you call a provider directly, you decide when to move to a new model version and you pay for re-evaluating before you do. Provider conventions differ. Anthropic's documentation, for example, says that for its 4.6 generation and later a dateless model identifier maps to a single fixed snapshot with updates shipped under new identifiers, while noting that serving infrastructure changes can occasionally produce minor differences in observable behavior even with the same identifier and weights. Anthropic, model IDs and versions Whatever the convention, record the exact identifier you evaluated and treat a change as a trigger to re-run your cases. With an embedded feature or a packaged application, the vendor makes that choice for you, so the question becomes how you will notice.

Open-weight deployment moves serving and patching to your team; its responsibilities are set out in open-weight and hosted deployment.

Step 1: Define the workflow and requirements

Sourcing starts after the workflow is understood, not before. Write down the unit of completed work, the current effort and error pattern, and the boundary of the first pilot, using choosing a first AI workflow and what AI changes in a workflow. Then write the requirements sheet described in selecting models for a specific task: what the system must do, what it must never do, and which requirements are pass-or-fail gates rather than trade-offs.

Record: the unit of work, the eligible case types, the requirements sheet with gates marked, and the owner of the workflow.

Decision point: if the requirements cannot be written down, stop. Buying a product to discover what you want transfers the definition of the workflow to the vendor's defaults.

Where this step fails: requirements written as a feature list from a vendor's brochure. Requirements describe your work ("cite the article and revision that supports each instruction"), not a product ("has retrieval").

Step 2: List the hard constraints

For your technical lead

Some requirements eliminate options outright. Typical gates are data handling (where customer content may be processed and stored, and for how long), permissions (whether the system can enforce the same access boundaries your people work under), deployment (network isolation, region, on-premises), integration (which systems must be read or written, and how), and authority (which actions the system may take; see assistance, approval, and autonomy).

Apply each gate to each option as pass, fail, or unknown. An unknown is a question for vendor diligence or for your own engineering team, not a pass.

Record: a constraint table, options across the top, each cell pass, fail, or unknown with the evidence or the open question.

Decision point: remove options that fail a gate. If one option remains, step 6 is unnecessary; still complete steps 3, 4, and 5 for that option, so the memo can say whether it is worth doing and what it commits you to. If none remains, revisit the requirements or the workflow boundary before looking for more options.

Where this step fails: treating a gate as a weight. A packaged application that cannot enforce per-client access boundaries does not become acceptable because it is cheaper; the knowledge retrieval chapter shows why that boundary matters before retrieval, not after.

Step 3: Describe what each option leaves you owning

For whoever runs the pilot

For each surviving option, fill in the three ownership columns from the table above for your situation, with names. "Knowledge upkeep: support knowledge owner, about a day a week" is useful; "knowledge upkeep: us" is not.

The economics chapter lists the continuing work that the license price leaves out: knowledge and source upkeep, quality review and evaluation reruns, integration monitoring, access reviews, and handling model or provider changes. AI economics In its support example, half of the illustrative $2,000 monthly budget pays for knowledge upkeep and quality review rather than software. Those activities exist under every option; the options differ in who performs them and whether you pay for them in cash or in staff time.

Record: the ownership table, with a named owner or role for every continuing activity.

Where this step fails: an activity with no owner. It does not disappear; it becomes a slow decline in answer quality that nobody is measuring.

Step 4: Estimate lifecycle effort

For your finance lead

Estimate setup and continuing effort for each option over a stated horizon, such as 24 months. Use the same categories for every option so the comparison is fair:

• Setup cash: licenses, implementation fees, tooling, a services firm's fee.
• Setup staff time: mapping, integration, building, initial evaluation, training.
• Continuing cash: subscriptions, usage, hosting.
• Continuing staff time: knowledge upkeep, review and evaluation reruns, monitoring, access reviews, model-change testing.
• Exit cost: recorded separately in step 5, not added to the horizon total.

Value staff time at the rate your economics worksheet uses, and keep the cash and the staff time in separate columns as well as adding them; finance may need to see them apart. Count each activity once. If a vendor's fee includes evaluation reruns, do not also count your staff doing them.

Record: the estimate table, each input marked observed, estimated, or assumed, with its source. The companion workbook's Lifecycle sheet holds one option per column, keeps cash and staff time apart, and records exit cost without adding it.

Decision point: if the options are within the uncertainty of the estimates, do not let the arithmetic decide. Move to step 5 and let exit cost and control break the tie.

Worked comparison: the support-reply pilot

The following is an illustrative comparison for the fictional pilot in the support-reply chapter. Every figure is an invented assumption; none is a price or a measurement. Staff time is valued at $45 an hour, the rate used throughout that chapter.

Setup cost is setup cash plus setup hours at $45. Monthly cost is continuing cash plus continuing hours at $45. The horizon total is setup plus monthly cost times the number of months.

Three things are visible in this example that a comparison of license prices would hide.

First, staff time is a large share of every option's monthly cost: 39% for A, 75% for B, and 73% for C. Option A has the highest software bill and the lowest staff time, and it ends up cheapest under these assumptions because the vendor performs work that options B and C leave to the team.

Second, the ranking depends on assumptions that are hard to know before a pilot. If option A's continuing staff time were 40 hours a month rather than 20, its monthly cost would be $3,200 and its 24-month total $80,800, the highest of the three. The packaged option is cheaper only if it really absorbs the work it claims to.

Third, setup and running costs trade against each other slowly. Option C costs $10,300 more than B to set up and $180 a month less to run, so its lower running cost would take about 57 months to make up the difference, longer than most systems run unchanged.

None of these totals is a benefit. Compare them with the workflow's value, which the support chapter estimates at $5,400 of monthly capacity before costs, and which becomes money only through a named realization mechanism. AI economics

Where this step fails: estimating setup carefully and continuing work loosely. In this example the continuing staff hours decide the ranking, so they deserve the most scrutiny and are the first thing a pilot should measure.

Step 5: Assess switching cost and exit

For your technical lead

Every option should have a written answer to the question: if this stops working, or a better option appears, what would leaving cost and what would we keep?

Record: an exit estimate for each option, in the same units as step 4, kept separate from the horizon total.

Decision point: an option whose exit cost is unknown is not ready to choose. Ask the questions in vendor diligence or, for a build, write the export and documentation requirements into the plan.

Step 6: Run a bounded comparison

For whoever runs the pilot

When more than one option survives and the estimates do not separate them, test them on the same cases. Use the staged approach in pilot design: offline cases first, with the same held-out set, the same acceptance criteria, and the same scorecard for every option. Include the current process and the simplest alternative, such as improved search and templates, as the comparison in selecting models for a specific task recommends.

Measure the inputs that drove the estimate, not only output quality. In the worked comparison above, continuing staff time decides the ranking, so a pilot that records how long knowledge upkeep and review actually take under each option answers the question the estimate could not.

Record: scorecard results for each option on the same cases, plus measured staff time where available.

Where this step fails: letting each vendor run its own demonstration on its own examples. A comparison is only a comparison when the cases, criteria, and reviewers are the same and are yours.

Step 7: Record the decision

Write a short sourcing memo, using the decision-memo structure in the pilot kit:

• the workflow, unit of work, and requirements, with gates marked;
• options considered, and which gate removed each one that was removed;
• the ownership table, with named owners for every continuing activity;
• the lifecycle estimate and exit estimate, with each input's status;
• comparison results, if step 6 ran;
• the choice, and the assumption it is most sensitive to;
• reopening conditions: for example, a model or provider change, a measured continuing effort well above the estimate, a contract renewal, or a change in data-handling requirements.

Where this step fails: a decision recorded as a product name with no reasons. A year later, nobody can tell whether the conditions that justified it still hold.

Accounting is a separate question

For your finance lead

Whether setup costs are capitalized or expensed, and how a cloud arrangement is presented, depends on the reporting framework, the contract, and the facts of the arrangement, not on whether the system uses AI. The economics chapter cites the FASB and IFRS sources, including an internal-use software update whose effective date makes adoption status the first question.

This chapter gives no accounting opinion. Give finance the step 4 estimate with cash and staff time separated, and the contract terms, and let them classify it. The finance leaders guide lists the facts a qualified reviewer will need.

Common errors

Related reading

Vendor diligence · Selecting models for a specific task · Open-weight and hosted deployment · AI economics · Workflow economics worksheet · Pilot design · Engineering leaders