Vendor diligence

Applies to: buying an AI product or service for a defined workflow. Owners: the business owner of the workflow, with security, legal or privacy, finance, and engineering. Companion records: evidence checklist, evaluation scorecard, evaluation cases.

Vendor diligence is the work of finding out whether a specific product will do your work, under your constraints, at a cost you can sustain, and whether you can leave if it does not. Demonstrations, case studies, and certifications each answer part of that question. None answers all of it, and some answer a different question from the one being asked.

This playbook assumes the sourcing decision has been made: you intend to buy rather than build or configure what you already own. The build, buy, or configure playbook covers that earlier decision. Here the question is narrower: should we buy this product for this workflow, and on what terms?

The procedure below is proposed practice. The worked record uses a fictional vendor, Wexcombe Reply, evaluated against the guide's support-reply workflow. Nothing in it describes a real vendor, contract, or result.

The procedure at a glance

Steps 4, 5, and 7 can run alongside step 3. Step 1 cannot be skipped: without it, every later answer is evaluated against whatever the vendor chose to show.

Step 1: Write your requirements first

Write down the workflow, the unit of completed work, and what an acceptable outcome is before the first demonstration. The model-selection chapter sets out a requirements sheet that separates gates from trade-offs; use the same structure here. A product is a configuration someone else has assembled, so the same distinction applies.

Mark each requirement as one of two kinds:

• Pass or fail. Required access boundaries, data-handling conditions, deployment location, and handling of the cases that must never go wrong. A product that fails one is out, however good it is otherwise.
• Trade-off. Quality on ordinary cases, review effort, latency, reliability, and cost. These are compared among the products that pass.

Record who owns each requirement. A security requirement with no owner in security will be answered by whoever happens to be in the meeting.

Decision point. If you cannot state the unit of work and an acceptable outcome, stop. Diligence on a product for an undefined workflow measures the vendor's presentation, not your fit. Go back to choosing a first workflow.

Where this step fails. Requirements written after the demonstration can end up describing what the demonstration showed. Date the requirement sheet and keep it fixed; record any later change and the reason for it.

Step 2: Sort the vendor's claims by what they would need to prove

Vendor materials mix several kinds of claim. The evidence chapter distinguishes capability, performance, effect, economic, and transfer claims, and each needs different evidence. List every material claim the vendor makes and record its type and what was offered in support.

For each performance or effect claim, ask the questions the evidence chapter's checklist sets out: which population, which cases, which measure, which denominator, compared with what, and which version of the system. A percentage with no denominator is not yet a claim you can check. Record "not provided" when the vendor cannot answer; that is a finding, not a failure of the process.

Distinguish three sources:

• Vendor-measured. The vendor ran the test. Ask for the case set, the rubric, and the configuration tested.
• Customer-reported. A customer described its experience. Ask whether the customer measured it, against what, and whether you may speak to the person who did.
• Independent. Someone without a commercial interest measured it with a design you can read. Where it exists, it is the most useful of the three.

Published research can inform your questions, but its results do not transfer directly to your setting. The guide's account of a support-assistant study shows how much a study's design limits what it can claim: its larger rollout was staggered rather than randomized, and the authors analyzed it with difference-in-differences methods. Generative AI at Work, v2 A vendor citing that study, or any other, is citing a different system in a different setting. The engineering leaders guide shows two published studies of AI-assisted development reaching different pictures because they measured different populations with different designs.

Where this step fails. Case studies are selected. Ask how many customers deployed the product in the workflow you care about, how many are still using it, and why any stopped. If the vendor will not say, record the gap.

Step 3: Test on your own cases

For whoever runs the pilot

The only performance evidence that answers your question comes from your work. Build a small case set before the vendor sees it, following pilot design: ordinary cases in proportion to your traffic, plus a separate challenge set with missing information, conflicting sources, out-of-scope requests, and inputs that try to instruct the system. The evaluation cases give fictional fixtures for each of the guide's workflows; adapt the pattern to your own data.

Rules for a fair test:

• Hold the cases back. Do not share them during sales conversations or tuning. If the vendor configures the product, give it development examples and keep a held-out set for scoring.
• Fix the configuration. Record the product version, any model it names, settings, and the content it was given. A result on an undocumented configuration cannot be reproduced.
• Score with your rubric. Use the evaluation scorecard: outcome, severity of each error, reviewer effort, and whether the system correctly declined or escalated.
• Include review time. A fluent answer that takes longer to check than to write is not a saving. The model-selection chapter's worked example shows the cost ranking of two configurations reversing once checking and rework are counted.
• Run repeats where outputs vary. A single good run on a case is not evidence it will be good next time.

If the vendor will not allow a test on your cases before purchase, record that, and treat any commitment as a supervised pilot with explicit stop criteria rather than a deployment.

Decision point. If any pass/fail requirement fails on the challenge set, the product is out for this workflow, or the workflow's scope must change. Do not average a hard failure into an overall score.

Step 4: Verify data handling and access

For your technical lead

Ask for written answers, and map each one to a requirement from step 1. The questions that matter depend on the data, but these recur across the guide's workflows:

Permission inheritance deserves its own test. A vendor may state that its assistant only surfaces data a user can already see; Microsoft states this for its workplace assistant, for example. Microsoft Copilot data, privacy, and security Such a statement can be accurate and still reproduce existing oversharing, and it describes the vendor's design, not your configuration. Test it with accounts that mirror your real permission patterns, as the knowledge retrieval chapter proposes. OWASP lists misaligned access controls on embeddings and cross-context leakage as retrieval risks. OWASP LLM08:2025

For untrusted input, ask which of the prompt-injection mitigations cited in the retrieval chapter the product applies (least privilege, segregating untrusted content, human approval for high-risk operations), then include injection cases in your step 3 tests. No cited source measures how often attacks succeed against any product.

Where this step fails. A questionnaire answered by sales is a statement of intent. Ask which answers appear in the contract or the data processing terms, and record the rest as unverified.

Certifications and reports: what they cover

Vendors offer attestation reports and certifications as evidence of security and governance. Read them for what they cover.

AICPA describes SOC 2 as reporting on an examination of controls at a service organization relevant to security, availability, processing integrity, confidentiality, or privacy, and says SOC reports give users information needed to assess the risks of outsourcing services. AICPA & CIMA, SOC 2 That is useful for step 4. It is a report about controls at the vendor, not evidence that a model drafts accurate replies.

The same reasoning applies to certifications of a management system: they speak to how an organization manages a domain, within a stated scope. For any report or certificate, record:

• Scope. Which services, systems, and locations it covers, and whether the AI features you are buying are inside it.
• Period and date. When it was issued or the period it covers.
• Exceptions. What the auditor or assessor found.
• Subprocessors. Whether the model provider or hosting provider is inside the scope, or excluded.

None of these establishes task performance, and none replaces step 3.

Step 5: Establish change and monitoring terms

For your technical lead

An AI product can change underneath you: the vendor updates prompts, retrieval, or a model, or its provider does. A product that passed your test in March may behave differently in June without any change on your side.

Ask:

• What can change without notice? Models, prompts, retrieval, safety filters, and default settings.
• How are you told? Release notes, advance notice, and a way to stay on a prior version for a period.
• Can you pin a version? Providers differ. Anthropic's documentation, for example, describes model identifiers that map to fixed snapshots, and notes that serving infrastructure can still produce minor differences in observable behavior. Anthropic model IDs and versions Ask the vendor what it pins and what it does not.
• What monitoring does the vendor provide, and what must you own? Error reporting, usage, latency, and any quality signals. The vendor's dashboard measures what the vendor chose to measure.

Then write your own re-evaluation plan: which changes trigger a rerun of the step 3 case set, who runs it, and what happens if results fall below the acceptance criteria. The model-selection chapter lists re-evaluation triggers for configurations you operate; the same list applies to configurations a vendor operates on your behalf.

Over time, reviewers' trust in a product that has performed well can turn checking into a formality. The authority chapter lists the signs. The human review playbook proposes seeded test cases and sampling to keep review real; plan them before go-live, not after.

Step 6: Model the cost at your volume

For your finance lead

Ask for the pricing structure, then model it against your own volume using the economics worksheet. Questions that change the answer:

• What is the unit of charge? Seat, usage (requests, documents, minutes, tokens), outcome, or a mix. Usage pricing scales with adoption, which is what you are trying to increase.
• What happens at the edges? Overage rates, minimums, rate limits, and what the product does when a limit is reached.
• What is extra? Integration, additional content sources, premium models, support tiers, and storage.
• Which of your costs remain? Review, knowledge upkeep, integration monitoring, and evaluation. These belong in the worksheet's continuing-work line whether or not the vendor mentions them.

Compare on the cost of a completed unit of work, with people's time included, not on the vendor's price per seat or per call. Keep capacity value and cash separate, as the economics chapter sets out.

Step 7: Plan the exit before you enter

The easiest time to agree exit terms is before signing. Ask:

• What data and configuration can you export, in what format, and how promptly?
• Can you export the evaluation evidence the product holds about your cases, such as logs and reviewer edits?
• What happens to your data at termination, and how is deletion confirmed?
• What would it take to run the workflow without the product: the fallback the pilot already requires, or a replacement?

A product with no practical exit raises the stakes of every other answer.

Step 8: Decide and record

Write a diligence memo using the pilot kit decision structure. It should state:

• Each pass/fail requirement and how it was verified: tested, contractually committed, or stated but unverified.
• Step 3 results with denominators, separating ordinary and challenge cases.
• Open items and who owns them.
• Conditions on the decision: for example, a supervised pilot only, a contract term that must be added, or a re-test after a named change.

A decision to buy for a supervised pilot with explicit stop criteria is a legitimate outcome, and can be the right one when evidence on your own work is limited.

Worked record: a fictional support-drafting vendor

Fictional example. Wexcombe Reply is an invented vendor of a support-drafting assistant. The buyer is the support team from the support-reply chapter, whose product is the fictional HarborDesk 4. The questions follow this playbook; the answers and their status are invented to show how a record looks.

The 2,400 assisted cases come from the support chapter's illustrative volume (5,000 requests, 60% eligible, 80% adoption). A record like this is incomplete by design: it shows which answers rest on tests, which on contract terms, and which only on the vendor's word. In this fictional record the team would proceed to a supervised pilot only after the boundary and injection tests passed, with the training and provider-change terms in the contract.

If you are the vendor

This playbook is written for buyers, but founders can read it as a description of the evidence a careful buyer will ask for. The implications follow from the procedure itself, not from any survey of buyers:

• Expect to be tested on the buyer's cases. A product that cannot be configured and evaluated on a held-out set before purchase gives the buyer only demonstration evidence.
• State claims with their type and denominator. A performance claim with the case set, rubric, and configuration attached can be checked; one without them goes in the "not provided" column.
• Be specific about data handling in writing. Retention, training use, subprocessors, deletion, and permission behavior, mapped to contract terms rather than sales answers.
• Have a change policy. What changes without notice, how customers are told, and what can be pinned.
• Know your report's scope. If the AI feature or the model provider is outside the scope of a security report, say so before the buyer finds it.
• Make exit possible. Export of data, configuration, and evaluation history lowers the buyer's risk of starting.

The evidence chapter describes the same standards from the reader's side, and the build, buy, or configure playbook describes the comparison a buyer makes before a vendor conversation begins.

Common failure points

Related reading

Evidence: what counts as a demonstrated result, build, buy, or configure, selecting models for a specific task, open-weight and hosted deployment, pilot design, human review and escalation design, and the economics worksheet.