Use this playbook once a workflow has been chosen and before the pilot charter is signed. It sets out who a pilot needs, defined by responsibility rather than job title; which roles one person may hold together and which must stay apart; how involved each role is at each stage; and whether each role should come from your own staff, a contractor, or a partner firm. It produces a staffing plan the pilot charter can name.
A pilot tests an operating arrangement, not only a model. The people who own the process, keep the sources current, check the output, and decide whether to expand are part of what is tested. A pilot staffed only with builders can show that a system runs; it cannot show that your organization can operate it.
Everything below is proposed practice unless a source is cited. The only sourced figures, posted pay at New York AI companies, are market context with their limits stated.
The procedure at a glance
| Step | Decision it settles | Record |
|---|---|---|
| 1. Name the roles | Which responsibilities this pilot needs | Role list |
| 2. Decide which roles may be combined | Who may hold more than one role | Combination check |
| 3. Plan involvement by stage | How much each role is needed at each pilot stage | Role-by-stage table |
| 4. Choose in-house, contractor, or partner | Where each role comes from | Sourcing choice and handover terms |
| 5. Test candidates with work samples | Whether the people proposed can do the job | Work-sample results |
| 6. Price the outside work | What the bought roles cost | Quotes by role, stage, and deliverable |
| 7. Record the staffing plan | Who is named, with backups and post-pilot owners | Roster attached to the charter |
Step 1: Name the roles
Define each role by what it decides and produces. The nine workflow chapters in this guide each list who their pilot needs, and the same responsibilities recur across them.
| Role | Responsible for |
|---|---|
| Executive sponsor | Funding the pilot, approving the charter, and making the expansion decision |
| Process owner (workflow owner) | The operational result of the workflow before and after the change; the baseline and the expansion memo |
| Knowledge or data owner | The sources the system reads: accuracy, versions, access scope, conflicts, and retirement |
| Builder | The system: prompts, retrieval, validation, interface, and test harness, with versions recorded |
| Integration engineer | Connections to systems of record, permissions enforced in the application, fallback, switch-off, and monitoring |
| Evaluator | The case set, severity rubric, and scoring, independent of the builder |
| Risk, compliance, or security reviewer | Whether the design meets legal, regulatory, and security obligations, where the workflow touches regulated data or consequential actions |
| Change or frontline lead | How the people doing the work use, check, and report on the system |
AI engineer or forward-deployed engineer. Both write the software that makes a model useful against your work; the difference is who employs them. An AI engineer, as this guide uses the term, builds for you, as your employee or under your contract, and the system is the deliverable. A forward-deployed engineer is employed by an AI vendor and placed with a customer to make that vendor's product work in the customer's environment; Neuronify's description of the role covers it further. A forward-deployed engineer can be a sensible part of a vendor option, but their continuity depends on the vendor contract, and their work is evaluated as build, buy, or configure says to evaluate any vendor's: on your cases, scored by your evaluator.
The risk reviewer is conditional, not optional. If the workflow reads personal, financial, health, or client-confidential data, or if any step can change a record, commit money, or reach a customer, name this reviewer before the charter is signed. The authority chapter sets out which actions need approval; the reviewer checks that the design enforces it. Whether a particular rule applies to your workflow is a question for counsel or your compliance function.
Record: the role list, with any role marked not needed and the reason.
Where this step fails: roles named by department ("IT", "the AI team"). When a source goes stale or a draft fails, nobody can tell whose job it was.
Step 2: Decide which roles may be combined
In a small firm one person may hold several roles, as long as the combination does not remove a check the pilot depends on. Workable combinations include sponsor and process owner, process owner and change lead, process owner and knowledge owner when the sources are few, and builder and integration engineer on a small system, provided the fallback and switch-off are documented.
Three pairs must stay apart:
| Must stay apart | Why |
|---|---|
| Builder and evaluator | A builder who writes the cases and scores the results is grading their own work. Pilot design lists severity "assigned after the fact by the person who built the system" as a failure |
| Requester and approver | Whoever asks for a change, to the acceptance criteria, a source, the system's permissions, or the scope of expansion, does not also approve it. An approver needs the information, time, skill, and authority to catch errors, as the authority chapter sets out |
| Builder and risk reviewer | Reviewing your own design adds no independent check |
If the organization is too small to separate builder and evaluator internally, bring the evaluator in from outside, or have the process owner score a held-out set the builder never sees.
Record: each person's roles, checked against the three pairs.
Decision point: if a prohibited pair cannot be separated, fix staffing before the charter is signed. A result scored by its builder cannot support an expansion decision.
Where this step fails: separation on paper only. An evaluator who works for the firm that built the system, or reports to the builder, is not independent.
Step 3: Plan involvement by stage
For whoever runs the pilot
Pilot design runs a pilot through a charter, offline evaluation on held-out cases, a supervised pilot on live work, and an expansion decision. The table uses qualitative levels rather than hours, because hours depend on the workflow and must be estimated locally. Lead owns the stage's output; regular means recurring work throughout the stage; review means sign-off at defined points; none means not needed.
| Role | Charter | Offline evaluation | Supervised pilot | Expansion decision |
|---|---|---|---|---|
| Executive sponsor | Review (signs) | None | Review (stop conditions) | Review (decides) |
| Process owner | Lead | Regular | Lead | Lead (writes the memo) |
| Knowledge or data owner | Regular | Regular | Regular | Review |
| Builder | Review (feasibility) | Regular | Regular | Review |
| Integration engineer | Review (feasibility) | Regular | Regular | Review |
| Evaluator | Regular (cases, rubric, thresholds) | Lead | Regular | Regular (results) |
| Risk or security reviewer, where needed | Review | Review | Review | Review |
| Change or frontline lead | Review (baseline timings) | Review | Lead (training, fallback, reporting) | Regular |
Note that the evaluator works before anything is built, because thresholds must precede results, and the frontline lead leads the supervised pilot, when reviewers' habits form; human review covers keeping review effective as volume rises.
Record: the table for this pilot, with a local hours estimate for each lead and regular cell, marked as estimated.
Where this step fails: planning only the build. If the knowledge owner and evaluator are not budgeted for the supervised stage, their work is absorbed silently and nobody maintains the system's inputs.
Step 4: Choose in-house, contractor, or partner
The ownership table in build, buy, or configure separates "What you own over the lifecycle", which under every option includes evaluation, review, and knowledge upkeep, from "What you depend on others for", which can be bought.
Anything you must own after go-live should not sit only with an outside party. That covers process ownership, knowledge upkeep, the evaluation set and the ability to re-run it, and approval authority. An outside advisor can help design a rubric or run a first evaluation round, but someone on your staff must own the cases and results and be able to re-run them when the model or a source changes.
Build and integration skills can be bought, if knowledge transfer is contracted.
| Role | In-house | Contractor or partner firm |
|---|---|---|
| Executive sponsor, process owner, change lead | Required | No |
| Knowledge or data owner | Required | May assist with a one-time cleanup |
| Builder | Possible if the skills exist | Possible, with knowledge transfer |
| Integration engineer | Preferred, since integrations need upkeep | Possible, with knowledge transfer |
| Evaluator | Owns the cases and re-runs | May design or run the first round, if independent of the builder |
| Risk or security reviewer | Your existing function, where one exists | As advisor; approval stays with you |
A knowledge-transfer clause should name the in-house recipient and list: code and configuration in a repository you control; prompts, retrieval settings, and validation rules held outside any vendor's interface; the case set, rubric, and every result; and runbooks for fallback, switch-off, and model-version changes. The exit checks in build, buy, or configure are the same list from the sourcing side. Choosing and contracting a partner is covered in choosing and working with an AI partner.
Whether a person should be engaged as an employee or a contractor, and what a contract must say about intellectual property, confidentiality, and worker classification, are questions for employment counsel.
Record: the source for each role; for every bought role, the handover deliverables and the named recipient.
Decision point: if nobody on your staff can receive the handover, name and prepare that person before the engagement starts, or treat the system as a vendor product and apply vendor diligence.
Where this step fails: a partner who builds, evaluates, and operates the pilot, then leaves. The organization holds a result it cannot reproduce and a system it cannot run.
Step 5: Test candidates with work samples
For whoever runs the pilot
A résumé shows what someone has worked on; a short work sample shows how they would approach this pilot. Use a non-sensitive slice of the workflow, such as a fictional case like those in the case library, and score each answer against the "good looks like" column before choosing.
| Role | Good looks like | Work sample or question | Warning signs |
|---|---|---|---|
| Executive sponsor | Can name the decision, the date, and what would stop the pilot | "What would make you stop this, and would you accept a result that says better search was enough?" | Describes success only; treats the pilot as a commitment to deploy |
| Process owner | Knows the unit of completed work and its effort from measurement | Time three recent cases from start to resolution, including follow-ups | Describes the procedure manual, not the work as performed |
| Knowledge or data owner | Knows which sources are current, which conflict, and who decides | Given two sources that disagree, explain how the conflict is resolved, recorded, and the old version retired | "Everything is in the shared drive"; no review dates |
| Builder | Treats evaluation as part of the build | "Walk through how you would test this system on cases where the right answer is 'I don't know' or 'this needs a person'." | Leads with model choice; relies on a demonstration; cannot say how they would notice a model version change |
| Integration engineer | Enforces permissions in the application and designs the fallback first | "If the call times out halfway, what state is the record in, and how does the person finish the case?" | Relies on instructions to the model to keep data separate; no switch-off |
| Evaluator | Sets criteria with denominators before results; keeps challenge cases apart from a representative sample | Design a severity rubric for this workflow, with one example per level, including a correct abstention, and say who settles disputes | Proposes a single accuracy percentage; reluctant to report a negative result |
| Risk or security reviewer | Asks what the system can read and do, not what the brochure says | "Which test would show that one customer's data cannot reach another customer's draft?" | Reviews a policy document instead of the running configuration |
| Change or frontline lead | Plans for reviewers to stay skeptical | "How would you know if approval had become a formality?" | Training covers only how to use the tool |
A strong builder answer includes cases built for missing and conflicting evidence, counts a correct abstention as a pass, and records the model identifier and prompt revision with each run. A strong change-lead answer names signals like those in the authority chapter.
Record: each candidate's work sample or answer, scored before the choice.
Where this step fails: letting candidates demonstrate on their own examples. As with vendors, the test is a test only when the case is yours.
Step 6: Price the outside work
For your finance lead
Price bought roles from quotes for defined deliverables by stage, with knowledge transfer as its own line, and ask every bidder for the same breakdown. Value in-house time at the rate your economics worksheet uses, and carry the continuing roles from step 4 into the lifecycle estimate in build, buy, or configure as continuing staff time.
Posted pay as market context. Neuronify's daily snapshot of postings at the New York AI companies it tracks recorded, on October 3, 2026, 784 New York City postings, 715 with a posted annual base range. Taking the midpoint of each range, the median for engineering roles was $215,000 (n=160); by title seniority, it was $200,000 for mid-level (n=70), $207,500 for senior (n=42), and $253,500 for staff and principal (n=33). Product roles had a median of $200,000 and operations roles $155,000. Neuronify Hiring Index, October 3, 2026 snapshot
Read those figures with their limits. They are posted base ranges, not offers or accepted pay, and exclude equity, bonus, and commission. They come from AI vendors in New York, not from the firms that staff pilots, and a firm in another industry or city may pay differently. They are one day's snapshot, seniority is classified from job titles, and ranges posted under New York's pay transparency law can be wide. They answer one question: whether a full-time builder hire is plausible at your budget. The AI engineer salary page shows the current version of the same data.
Contractor and partner rates vary with scope, seniority, engagement length, and who carries delivery risk. This guide cites no rate benchmark; get quotes for the actual work.
Record: quotes broken down by role, stage, and deliverable, with knowledge transfer priced separately, and in-house hours from step 3 valued at the worksheet rate.
Where this step fails: comparing a contractor's day rate with an employee's salary; one buys a deliverable, the other continuing capacity.
Step 7: Record the staffing plan
Attach a roster to the pilot charter with, for each role: one named person, not a team; the source (in-house, contractor, or partner); other roles held, checked against step 2; involvement by stage from step 3; a backup; for bought roles, the handover deliverables and recipient; and who holds the responsibility after the pilot ends.
Decision point: do not sign the charter until every role has a named person or a recorded reason it is not needed, and the after-pilot owner is filled in for process ownership, knowledge upkeep, evaluation, and approval.
Where this step fails: a roster that ends at the pilot. The expansion decision then approves a system nobody is assigned to run.
Worked example: staffing the support-reply pilot
This applies the procedure to the fictional HarborDesk pilot in the support-reply chapter. Roles and budget lines come from that chapter; the sourcing choices are illustrative and no figure is added.
The chapter assigns expansion to the support manager, source sign-off to the knowledge owner, retrieval boundaries and fallback to engineering, realization assumptions to finance, and investigation of any critical failure to security and engineering.
| Role | HarborDesk holder | Source |
|---|---|---|
| Executive sponsor | The executive who approves the $12,000 setup budget | In-house |
| Process owner and change lead | Support manager | In-house |
| Knowledge owner | Owner of the help articles | In-house |
| Builder | Support engineer, or a services firm | Depends on the sourcing option |
| Integration engineer | Support engineer: read-only access, retrieval boundaries, fallback, switch-off | In-house |
| Evaluator | A support team member who did not build the system, with a second reviewer scoring a subset independently | In-house |
| Security reviewer | Security function | In-house |
| Finance reviewer | Finance, for the realization assumption | In-house |
| Frontline reviewers | Trained agents who approve every reply | In-house |
Who comes from where depends on the sourcing choice. Build, buy, or configure compares three options for this pilot. Under option B, building on a model API, the support engineer is builder and integration engineer, so the evaluator must be someone else. Under option C, a services firm builds and HarborDesk operates; the contract names the support engineer as handover recipient for code, configuration, retrieval settings, and test results, and the evaluator must not come from the firm. Under option A, a packaged add-on, the support engineer configures and integrates, and any vendor staff, forward-deployed engineers included, are judged on HarborDesk's cases like any vendor claim.
The continuing-work budget shows which roles stay in-house. The chapter splits its invented $2,000 monthly budget into $800 for software and usage, $600 for knowledge upkeep, $400 for quality review, and $200 for integration monitoring. Only the first line pays a vendor under every option. Knowledge upkeep belongs to the knowledge owner, quality review to the evaluator and support manager, and integration monitoring to the support engineer. Absorbed by existing staff, that work is consumed capacity rather than cash, but it still needs a named person.
Combinations to check. The support manager holding process ownership and the change lead role is allowed. Under option B the support engineer may hold builder and integration but cannot evaluate. The support manager requests expansion through the memo; the sponsor approves. A change to the severity rubric after results arrive is requested by one person, approved by another, and recorded.
Common errors
| Error | What it looks like | Correction |
|---|---|---|
| Staffing only the build | The roster lists engineers and a vendor contact | Name the process owner, knowledge owner, and evaluator first |
| Builder grades their own work | Cases and scores come from whoever built the system | Separate builder and evaluator (step 2) |
| Outsourcing what you must own | A partner holds the cases, the rubric, and the only working configuration | Keep evaluation, knowledge upkeep, and approval in-house (step 4) |
| No handover recipient | Deliverables are listed but nobody on staff can operate them | Name and prepare the recipient before the engagement |
| Hiring from a demonstration | Candidates are judged on their own examples | Use a work sample from your workflow (step 5) |
| Pricing from salary surveys | Posted pay stands in for the cost of outside work | Get quotes by role, stage, and deliverable (step 6) |
| A roster that ends at the pilot | Nobody runs the system after expansion | Name after-pilot owners (step 7) |
Related reading
Choosing and working with an AI partner · Build, buy, or configure · Pilot design and acceptance criteria · Human review · Assistance, approval, and autonomy · Engineering leaders
Next step: if you need help scoping the pilot or finding the builder and evaluator, scope your project with Neuronify or read how Neuronify vets AI engineers.