Applies to: hiring people or a firm from outside to build, configure, or run an AI system for a defined workflow. Owners: the business owner of the workflow, with whoever runs the pilot, a technical lead, finance, and legal counsel. Companion records: pilot kit, evaluation scorecard.
An AI partner is anyone outside your organization you pay to build or run the system. Hiring one differs from buying a product: a product exists and can be tested, while with a partner you evaluate people and a plan for something that does not exist yet.
Build, buy, or configure decides whether to commission outside people at all. Vendor diligence covers buying a product. This playbook starts once you have decided to bring in a partner and runs through the engagement to handover. If a partner proposes building on a particular product, run vendor diligence on that product as well.
The procedure is proposed practice. The worked record follows a fictional support team choosing between an independent engineer and a small firm. Nothing in it describes a real firm, person, contract, or result.
The procedure at a glance
| Step | Output | Owner |
|---|---|---|
| 1. Turn the charter into a brief | Partner brief and ownership split | Workflow owner |
| 2. Choose the partner shape | Shortlist with the reason for each | Workflow owner |
| 3. Run a bounded work sample | Scored results on your held-out cases | Whoever runs the pilot |
| 4. Check references and staffing | Reference notes; named staff list | Workflow owner |
| 5. Review the partner's evaluation plan | Plan annotated against the charter | Technical lead |
| 6. Check data handling, ownership, and exit | Data answers, ownership list, handover plan | Technical lead, counsel |
| 7. Choose a pricing structure | Fee structure and what it needs in the contract | Finance lead |
| 8. Write the statement of work | Statement of work mirroring the charter | Workflow owner, counsel |
| 9. Decide and record | Partner decision memo with conditions | Workflow owner |
| 10. Run the engagement | Weekly evidence reviews, change log, handover record | Whoever runs the pilot |
Steps 4 to 6 can run alongside step 3. Step 1 cannot be skipped: a partner asked to propose its own scope will propose the work it is best placed to sell.
Step 1: Turn the pilot charter into a brief
Write the pilot charter before the first partner conversation. Its fields are the brief. A partner that has read them can answer a specific question: can you deliver a system that meets these criteria, under these limits, by this date, and hand it back in a state we can run?
Then split the work. Staffing an AI pilot sets out the roles a pilot needs, and the ownership table in build, buy, or configure lists the continuing work every option leaves with you. As proposed practice:
- A partner can own: building and configuring the system, integration, knowledge preparation under your owner's direction, running the system during the engagement, and documentation.
- You must own: the decision, the charter and any change to it, the evaluation cases and the definition of an acceptable outcome, acceptance scoring, the authority boundary, source ownership after handover, and the realization of any value.
Record: the partner brief (charter plus ownership split) and the date it was fixed.
Decision point: if you cannot state the unit of work and the acceptance criteria, stop. Go back to choosing a first AI workflow, or hire a partner only for a bounded discovery task whose deliverable is the charter, priced and judged separately from any build.
Where this step fails: the brief is written after the first pitch and describes the partner's proposal. Date it and record any later change with its reason.
Step 2: Choose the partner shape
The table is editorial reasoning about structure, not a finding about any market or firm.
| Shape | Can be good at | Can be bad at |
|---|---|---|
| Independent freelancer | Direct access to the person doing the work; narrow, well-defined builds; little coordination | Key-person risk; limited capacity for security review, documentation, and cover; one person's judgment unchecked. Worker classification is a question for counsel |
| Boutique AI firm | Specialist depth; a small team that can cover for each other | Principals who pitch but do not deliver; capacity shifting to other clients |
| Large systems integrator or consultancy | Breadth, security and procurement processes, integration with large systems, capacity to scale | Coordination overhead; senior people who sell and junior people who build; process that can outweigh a small pilot |
| Embedded or forward-deployed team | Learning the workflow in place; transferring knowledge by working alongside your people | Consumes your staff time; dependency if handover is not planned; blurred lines between their staff and yours, a question for counsel |
| A software vendor's professional-services arm | Deep knowledge of its own product | An interest in more use of its product; evaluating its own product; a services contract that may not cover the product's behavior |
Choose the shape from the brief, not the reverse.
Record: a shortlist of two or three partners, each with its shape and the reason it fits the brief.
Where this step fails: the shape is chosen by whoever called first, and the brief is bent to fit what that partner sells.
Step 3: Run a bounded work sample on your own cases
For whoever runs the pilot
A demonstration on a partner's examples shows it can prepare a demonstration. Ask each shortlisted partner for a paid or explicitly bounded piece of work on your cases: a small offline build or design exercise with a fixed time limit and a written output.
- Use your cases, held back. Draw them from the case set in pilot design. Give development examples; keep held-out cases, including challenge cases such as conflicting sources and permission boundaries, for scoring.
- Handle permissions first. Use synthetic cases or material you are permitted to share, under confidentiality terms reviewed by counsel. A small sample is no reason to send customer data to a shortlisted partner.
- Score with your rubric. Your evaluator scores on the evaluation scorecard, with severity per error.
- Read the working. Ask for a short account of what they tried, what failed, and what they would decline to build. A polished output can hide weak judgment.
- State the bounds. Fix the time limit and any fee in advance.
Record: each partner's output, your scores with denominators by case category, and notes on the written account.
Decision point: a partner that fails a mandatory requirement on the sample, such as surfacing content across an access boundary, is out for this brief unless the failure traces to something you provided. Do not average it away.
Where this step fails: the partner sees the scoring cases, so the sample measures tuning to them.
Step 4: Check references and who will staff the work
References. Ask for references from a comparable workflow, and speak to the person who owned the result. Ask who measured success, what failed and how the partner responded, how scope changes were priced, and whether the team could run the system without the partner afterward. Record answers as reference statements: the partner chose the reference, so this is customer-reported evidence in the terms of the evidence chapter.
Staffing. Ask for the named people who will do the work, each person's role and share of time, and the rule for replacing anyone. Meet the delivery lead and the engineer who will build, not only the presenters. Ask about subcontractors; they will handle your data too.
How Neuronify approaches this (our own practice). This describes Neuronify's published practice; it is not evidence for this guide's procedure. Neuronify's page on how it vets AI engineers says individual engineers are assessed on open-ended scenarios and a live defense of their design, scored by practitioners against a published rubric, and that each match comes with a one-page report that includes a gaps section. For consultancies, the page says Neuronify calls references itself, reviews a real reference architecture, assesses the delivery lead directly, and maps team depth so a client knows who will do the work.
Record: reference notes marked as reference statements; a named staff list with roles, time shares, and the replacement rule.
Decision point: if the partner will not name delivery staff before signing, treat staffing as unknown: make named staff a contract condition or move on.
Step 5: Review how the partner evaluates its own output
For your technical lead
Ask for the partner's evaluation plan in writing and compare it with the charter. A plan you can check states the case set and how held-out cases stay separate from development examples; the rubric and severity categories; every rate with numerator and denominator, ordinary and challenge cases reported separately; the configuration under test (model identifier, prompt revision, retrieval settings, source snapshot); the changes that trigger a rerun, as in selecting models for a specific task; and how mandatory requirements such as access boundaries are tested and reported. One score across all cases is not yet checkable.
The partner's evaluation serves development; it is not the acceptance test. Acceptance is scored by someone on your side who did not build the system, on cases the partner has not tuned against, as pilot design proposes. Write that separation into the statement of work.
Record: the plan, annotated against the charter, with gaps listed.
Where this step fails: the partner's dashboard becomes the only record of performance, measuring what the partner chose to measure.
Step 6: Check data handling, ownership, and exit
For your technical lead
A partner touches your data in more places than a product does, including staff devices, development environments, and subcontractors. Start from the data-handling table in vendor diligence, and add:
- Whose accounts and environments? Accounts you own make the data path visible and the exit simpler.
- Who has access? Named people, how access is granted, and how it is removed at the end.
- What is retained? What the partner keeps afterward, including copies of your cases and logs.
Mark each answer tested, written into the contract, or only stated.
Build, buy, or configure warns that an engagement can end with the knowledge leaving with the firm. List every artifact the engagement will produce (code, prompts, retrieval and validation configuration, infrastructure definitions, evaluation cases, results and logs, reviewer edits, documentation) and who owns each. The evaluation set and its results belong to you under every option. Ownership terms, licenses for components the partner brings from earlier work, and what it may reuse elsewhere are questions for counsel.
Plan knowledge transfer as deliverables: documentation sufficient for someone else to run and re-evaluate the system, a session in which your people rerun the evaluation set unaided, and transition assistance for a stated period, including after an early end.
Record: data answers with status, the ownership list, and the handover plan with milestones.
Decision point: a partner that cannot work within your data requirements fails a gate. A partner that will not let you own the evaluation cases, results, and the configuration you run leaves the exit cost unknown, and an option with an unknown exit cost is not ready to choose.
Step 7: Choose a pricing structure
For your finance lead
Each structure creates an incentive. Choose the one that fits the uncertainty in the work, and write in what it needs. This playbook gives no market rates; get quotes against your brief and compare them on the same scope.
| Structure | What it rewards | Risk left with you | What the contract needs |
|---|---|---|---|
| Time and materials | Effort; no pressure to cut scope to protect margin | Open-ended cost; no pressure to finish | A cap, time reports tied to charter tasks, a right to stop at any stage gate |
| Fixed fee | Finishing within the price | Scope or quality narrowed to fit; disputes over what was included | Acceptance criteria with denominators, a change-request procedure, a definition of done |
| Milestone | Reaching each stage gate | Milestones defined as artifacts delivered rather than evidence accepted | Each milestone defined as evidence accepted by your evaluator, matched to the pilot's stages |
| Outcome-based | Moving a measured outcome | The measure moved in ways you do not want: a generous definition, easy cases, a short observation window | The measure's definition and denominator matched to the charter's unit of work; who measures; baseline; observation window; a cap |
An outcome fee works only if you can measure the outcome yourself with your pilot's definitions; the vendor diligence worked record shows how a word such as "resolved" can count work you would not count. Model any structure with your own staff time included, using the economics worksheet.
Record: the chosen structure, why it fits, and its contract needs.
Step 8: Write the statement of work
The statement of work should mirror the pilot charter, so the partner's commitments and your test are the same thing. The items below are questions; the contract terms are for counsel.
| Item | Question it should answer |
|---|---|
| Unit of work | Is it the charter's completed unit (a resolved request, not a sent draft)? |
| Scope | Which cases are in scope and which excluded, as in the charter? |
| Acceptance criteria | What are the mandatory requirements and thresholds, with denominators, and who scores them? |
| Evaluation set | Who owns the cases, who may see the held-out set, and how is it versioned? |
| Stop conditions | Which events pause or end the work, and what happens to payment? |
| Change requests | How is a change proposed, priced, approved, and recorded? |
| Deliverables | What arrives at each stage, including documentation, results, and handover sessions? |
| Staffing | Who are the named people, and what is the replacement rule? |
| Data terms | What may the partner access, where, for how long, and how is deletion confirmed? |
| Ownership | Who owns code, prompts, configuration, evaluation cases and results, and documentation? |
| Exit and transition | What assistance follows an early or scheduled end, and for how long? |
Record: the draft statement of work, each line traced to a charter field.
Where this step fails: the partner's standard template becomes the statement of work, and the charter's acceptance criteria appear nowhere in it.
Step 9: Decide and record
Write a partner decision memo using the pilot kit decision structure: each shortlisted partner's sample results with denominators, reference notes, named staff, evaluation-plan gaps, and data status; the ownership list and exit plan; the pricing structure; and the choice, its conditions, and open items with owners.
Hiring for one stage only, such as offline replay, with a further decision at its gate, is a legitimate outcome. It limits what you commit before you have seen the partner deliver.
Step 10: Run the engagement
For whoever runs the pilot
As proposed practice:
- Weekly evidence review. Cases run, results by category with denominators, failures by severity, open questions. A list of tasks completed is not evidence; a scored case is.
- Scope-change control. Every change to scope, criteria, or cases goes through the change-request procedure and is logged with its reason, so a drifted scope is never presented as the original.
- An independent evaluator. Acceptance scoring stays with someone on your side who did not build the system, and held-out cases stay away from the partner until scoring. The human review playbook describes seeded cases and sampling that keep review real.
- Knowledge transfer on schedule. Hold handover sessions during the engagement. If your people cannot rerun the evaluation set at the midpoint, they will not manage it at the close.
- Configuration record. Tie each result to the configuration that produced it.
Record: weekly reviews, the change log, and the handover record.
Where this step fails: the partner runs the weekly meeting from its own dashboard, and your evaluator first sees held-out results at the final gate.
Worked record: a fictional support-reply engagement
Fictional example. The company, people, and partners are invented. The buyer is the support team from the support-reply chapter, whose product is the fictional HarborDesk 4, with a charter like the pilot design worked charter and a technical lead with limited time. It shortlists an independent engineer, called here the freelancer, and Tamsin Row Applied AI, an invented small firm. The answers are invented to show how a record looks.
| Question | Freelancer | Tamsin Row | Status and next action |
|---|---|---|---|
| Work sample on held-out cases | Filtered the archived HD-19 article out at retrieval; clear account of a failed first approach | Handled HD-19; one cross-workspace case surfaced a passage, traced to a test account set up wrongly on our side | Rerun Tamsin Row's boundary case with corrected accounts |
| Who does the work | The freelancer alone | Pitch led by a principal; named delivery engineer not yet met | Meet the delivery engineer; ask the freelancer for a named backup |
| References | One, from a comparable support workflow: documentation arrived late | Two; one says the firm reported a single overall score | Ask both how failures were handled |
| Evaluation plan | Severity and denominators by case category | One accuracy figure across all cases | Our evaluator scores acceptance either way |
| Data and ownership | Works in our accounts; code, prompts, and cases ours | Works in its own environment first; wants to keep its prompt library | Require our accounts; counsel reviews ownership and confidentiality |
| Pricing | Time and materials | Fee per "deflected ticket" | Decline: deflection is not the charter's unit, a resolved request |
In this fictional record the team hires the freelancer for the offline-replay stage only, paid in milestones defined as evidence accepted by its own evaluator, with documentation and an unaided evaluation rerun by its technical lead due before the final payment, and a named backup agreed in writing. Tamsin Row remains an option if the pilot passes its gate and the supervised stage needs more capacity.
Common errors
| Error | What it looks like | Correction |
|---|---|---|
| Partner before brief | The partner's proposal becomes the charter | Fix and date the brief first |
| Demonstration taken as a work sample | "It worked on their examples" | A bounded sample on your held-out cases, scored by you |
| Pitch team mistaken for delivery team | The people you met do not build | Named staff and a replacement rule in the contract |
| Partner scores its own work | Acceptance read from the partner's dashboard | An independent evaluator on your side |
| Outcome fee on an unchecked measure | A fee per item defined by the partner | Match the measure to the charter's unit; measure it yourself |
| Ownership assumed | Prompts, configuration, or cases stay with the partner | Ownership list agreed before signing |
Related reading
Staffing an AI pilot · Vendor diligence · Build, buy, or configure · Pilot design · Human review and escalation design · Evidence: what counts as a demonstrated result
Next step: to scope the work before you talk to partners, scope your project with Neuronify, or read how Neuronify vets AI engineers.