AI-assisted support replies for a B2B software company

Industry: B2B software. Roles: support agent, support manager, knowledge owner, support engineer. Capabilities: retrieve, classify, generate. Human control: approve before sending. Benefit: service capacity and potentially avoided delivery expense. Costs: software, integration, review, evaluation, and knowledge maintenance.

The decision

A support manager wants to reduce repetitive research and drafting while keeping answers accurate. The first useful boundary is a draft assistant for documented product questions. The assistant proposes a response; an agent remains responsible for deciding what to send. Billing disputes, account access changes, contractual commitments, and security incidents follow separate procedures.

The unit of work is a resolved request, not a generated reply. A faster first response that creates three follow-ups may increase total work. Read this chapter to decide whether the assistant deserves a bounded trial, what that trial should measure, and what would justify stopping it.

All product names, tickets, document excerpts, time allocations, budgets, and pilot thresholds below are invented teaching examples. They are not customer records, measured performance, vendor promises, or industry benchmarks. The research section describes external evidence separately.

Map the work before choosing a model

A customer opens a ticket. An agent identifies the account and product version, searches documentation and previous cases, asks for missing information, drafts a response, and records the outcome. Delays can arise from outdated instructions, scattered knowledge, or uncertainty about the customer's permissions. A writing assistant will not necessarily resolve any of these upstream problems.

Follow a representative request through every handoff. Include specialist work and repeat contacts, even when they belong to another team. Define a consistent observation window for follow-up work and report unresolved cases separately; otherwise a team can appear faster simply by closing cases earlier.

Workflow map

Ticket → scope check → permission-filtered sources → draft with evidence → agent review → approved send → resolution follow-up

At the scope check, an excluded request leaves this path for the existing specialist queue. At retrieval, missing or contradictory evidence leads to a clarification or escalation. At review, an unsupported draft is edited or rejected. If the assistant is unavailable, the agent continues through ordinary search and templates. None of these branches grants the model permission to change an account.

This map has two separate gates: whether the request belongs in the pilot, and whether this particular response is suitable to send. Passing the first does not imply passing the second.

Compare against a simpler alternative

For a fixed instruction, a maintained response template may do most of the useful work. For hard-to-find instructions, better search and article ownership may be the higher-value intervention. Compare three configurations on the same task categories: current process, improved search/templates, and search/templates plus generated drafts. This separates the value of finding good information from the value of writing around it.

Walk through one ticket

For your support lead

The fictional product in this exercise is HarborDesk 4, a B2B reporting application. The policy facts below exist only for this example. Assume the normal support system has authenticated the customer and established access to their workspace. The assistant has read access to permitted context and approved help material; it has no account-action or message-sending tools.

Input: customer request

I used to export the weekly activity report, but after our move to version 4 the Export button is grey. I'm a Viewer in the workspace. Can you switch it on? We need the file this afternoon.

The ticket contains a documented product question and an account-change request. The agent can explain the feature and the approved route for requesting access. The request to change permissions is outside this pilot's authority. Urgency does not alter that boundary.

Input: approved evidence packet

These are original fictional excerpts, not quotations from a real help center.

The evidence packet also records the article owner, effective date, access scope, and retrieval timestamp. HD-19 is shown here as a test distractor: the production retrieval layer should filter it out for a known version-4 request. A model that chooses the familiar-looking older rule should fail this case.

Expected internal draft package

The assistant should give the agent more than a polished sentence. A useful proposed package contains:

• Disposition: answer the documented feature question; route the permission-change request through the existing access process.
• Relevant facts: the customer reports version 4 and Viewer status. No independent role check has been performed by the assistant.
• Evidence: HD-41 revision 3 supports the role requirement; HD-42 revision 2 supports the owner-managed access process.
• Unresolved point: if the authenticated product context contradicts the reported version or role, clarify before sending version-specific instructions.
• Authority boundary: no role was changed, no export was produced, and no response was sent.

A citation is a claim to check. The agent should be able to open the precise passage used, not merely a help-center homepage. The agent also needs to see what the assistant could not establish.

Expected customer-facing reply

In HarborDesk 4, activity-report exports are available to Editors and Workspace Owners. Viewer access does not include export, which is consistent with the role you described.

Please ask a Workspace Owner to review your access request through your organization's approval process. Owners manage roles under Settings → Members. I haven't changed your permissions. If you're already an Editor or Owner, let us know so we can investigate the disabled button further.

This is an editorially authored expected answer, not output from a tested model. In a real trial, the agent would verify the account context and attach the appropriate customer-accessible help links before sending. Internal evidence identifiers should stay in the agent workspace unless they are valid public references.

Reject a plausible but unsupported reply

I've enabled exports for your workspace. Refresh the page and you'll be able to download the report today.

This response invents an action, promises an outcome, and conceals the permission boundary. Polite language and a quick response time cannot compensate for those errors. Reject the draft even if it includes a valid link to HD-41.

Change one input at a time

Each row changes one fact in the HarborDesk ticket and leaves the rest as above.

The evaluation set below repeats these variations and adds system-level cases.

Design the evidence and access layer

For your technical lead

Retrieval-augmented generation combines retrieval with generation so a model can work from supplied material. Microsoft's overview describes this architecture and its component choices. That pattern is useful here, but it does not prove that a particular answer is correct or that this support team will save money. Microsoft architecture overview

For this proposed design, source permissions are enforced before material reaches the model. Do not give the model a broad collection of customer data and ask it to remember which account it may discuss. A shared public product manual and a private ticket history need different access scopes. Test both retrieval results and generated responses for unauthorized information.

Give each approved article an owner, product applicability, effective date, review date, and retirement rule. Make contradictions visible. If two current articles disagree, the system should surface the conflict to the knowledge owner instead of asking a model to invent a compromise.

Record only the evaluation data the team needs, under an agreed retention and access policy. A case record should identify the source versions, system configuration, proposed draft, agent edits, elapsed human work, and final disposition. Use synthetic cases where they can test behavior without copying real customer content. For actual tickets, establish the authorization and handling rules before constructing the evaluation set.

Keep the initial system read-only. Operational controls should make an unsupported action impossible through this integration, rather than relying only on a sentence in the prompt. Define a manual fallback, an owner who can disable the assistant, and a route for reporting bad drafts without making agents absorb the investigation work invisibly.

Choose models against this task

For your technical lead

Test a complete configuration: model identifier, prompt revision, retrieval method, document snapshot, generation settings, and interface. Changing retrieval or the evidence display can change reviewer effort even if the model stays the same. Preserve those configuration details with each evaluation run.

Compare configurations on supported answers, appropriate clarification, escalation, editing effort, latency, and cost per adequately handled case. Count rejected drafts and retries. A low-cost answer that takes an agent longer to verify may raise total workflow cost.

Use the model-selection guide to structure the comparison, and take candidates' specifications from each provider's current documentation, checked on the date you test. No model has been benchmarked on this fictional HarborDesk task. Provider specifications are starting constraints, not evidence that a candidate meets the acceptance criteria.

Work through the economics

For your finance lead

The following monthly example uses illustrative assumptions. A request means a case with handling and follow-up effort measured consistently. Assisted effort includes review, editing, failures, and rework across the same observation window as the baseline.

This yields 2,400 assisted cases: 5,000 × 60% × 80%. At three minutes released per case, the monthly capacity is 120 hours. Valued at $45/hour, that is $5,400 of capacity value, or $3,400 after the recurring budget, before setup.

An illustrative eight-minute baseline might include one minute of intake, three of research, two of drafting, and two of follow-up and closure. A five-minute assisted process might use one minute of intake, two of evidence review and editing, and two of follow-up and closure. These allocations show where the proposed improvement comes from; they do not establish that review will actually take two minutes.

Make continuing work visible

For planning, split the hypothetical $2,000 monthly budget into $800 for software and usage, $600 for knowledge upkeep, $400 for quality review, and $200 for integration monitoring. The split is invented, not a quote. Upkeep absorbed by existing staff is consumed capacity, not cash, and review already inside the assisted minutes is not deducted again; the economics chapter explains both.

Likewise, do not subtract agent review twice: it is already included in the five-minute assisted handling assumption. Periodic quality sampling and article maintenance are different activities and belong in the continuing-work budget if they are incremental.

Stress-test review time and adoption

These scenarios hold the recurring budget fixed for clarity. Recalculate variable usage costs if lower adoption changes spending materially. At the base volume and hourly value, the recurring budget requires about 1.11 minutes released per assisted case just to break even on capacity value. That calculation excludes setup and does not establish cash break-even.

Explain how capacity becomes money

Suppose, separately, that 70% of the $5,400 capacity value can replace otherwise-paid work at the same assumed rate, and the entire recurring budget is incremental cash spending. The illustrative net cash benefit is $5,400 × 70% − $2,000 = $1,780/month. A $12,000 setup outlay would have simple payback of about 6.7 months, assuming immediate steady-state performance and no additional costs. Ramp-up delays that date.

If no paid expense can actually be avoided, the same workflow can release time while producing no labor cash saving. Under the all-cash recurring assumption, spending still rises by $2,000/month plus setup. The team might use the released time to improve service or absorb demand, but must assess that outcome separately. Capacity value and cash realized from the same hours are one benefit, not two (double counting).

This example is a workflow decision model, not an accounting determination; the economics chapter explains why classification is a separate question.

Define failure severity before running the trial

For whoever runs the pilot

The following categories are proposed pilot rules. Owners should adapt them to their process and agree on them before seeing model results.

A draft caught by an agent remains a model failure but need not become a customer incident. Track both stages. A team with excellent reviewers can otherwise conceal a poor assistant by recording only errors in sent messages. Conversely, do not penalize an appropriate escalation as though it were an invented answer.

Build the evaluation set

For whoever runs the pilot

As an initial scoping exercise, use 120 synthetic or authorized historical cases: 60 straightforward documented questions, 20 ambiguous requests, 15 stale or conflicting-source cases, 15 excluded action requests, and 10 adversarial or account-boundary cases. These counts are an illustrative coverage plan, not a statistically justified sample size or proof of safety. Call this the coverage set, and keep it separate from a traffic sample reflecting the actual traffic mix.

Split by issue family where possible so near-duplicate tickets do not appear on both sides of the development/evaluation boundary. Reserve held-out cases before tuning. For historical replay, use only the source material and context available at that time; do not let a later resolution article supply the answer. Keep a development set for iteration and a fresh held-out set for the decision.

Use a scorecard that exposes rework

For each case, record: case ID and category; correct expected disposition; permitted source IDs; system configuration; retrieved source IDs; factual support; applicability; authority compliance; severity; reviewer edits; human minutes; model and retrieval cost; latency; escalation; and final outcome. Mark unknown or unobserved outcomes explicitly.

Define a material claim as one that could change the customer's action or expectation. Review each such claim against the permitted evidence. A fluent response with one unsupported instruction fails factual support. Two reviewers should independently assess a subset, resolve disagreements, and improve the rubric before relying on aggregate scores.

Report counts with denominators: major errors among evaluated drafts, unauthorized retrievals among boundary tests, correct escalations among cases requiring escalation, and rejected drafts among all assisted cases. Separate ordinary traffic from intentionally difficult challenge cases. Do not combine them into an apparently representative accuracy percentage.

For an initial offline gate, a team might require zero observed critical failures and all excluded-action cases routed correctly in the coverage set, and at least 95% of ordinary answerable cases in the traffic sample free of major errors. Those are illustrative management thresholds. The team must choose sample sizes and tolerances appropriate to the consequences before moving to live use. A clean result on a small set bounds the true failure rate only loosely; pilot design shows how loosely.

Run a supervised pilot and make the decision

First run offline replay with no messages sent. Fix access, source, and interface problems before asking agents to use the system live. Then conduct a supervised trial limited to the agreed queue and trained agents. Keep existing escalation channels and make declining the suggestion easy.

Where practical, assign comparable work prospectively to assisted and baseline conditions. Balance case category, agent experience, time of day, and product version. Avoid comparing only agents who enthusiastically adopt the assistant with everyone else. If random assignment is impractical, document selection differences and treat the outcome as directional rather than a clean causal estimate.

Measure all cases assigned to each condition, including assistant failures and fallback work. Report the subset actually assisted separately. Follow requests through the predeclared observation window; compare total effort, unresolved cases, repeat contacts, escalation, and customer-impacting errors. Customer satisfaction can supplement these measures, but missing responses and unequal response rates need to remain visible.

The support manager owns expansion; the knowledge owner signs off on source maintenance; engineering owns retrieval boundaries and fallback; finance reviews the realization assumptions. Before an expansion decision, complete this checklist:

• Eligibility, excluded actions, observation window, and severity rules were set in advance.
• Permitted-source retrieval and account isolation passed the agreed tests.
• Held-out cases cover missing evidence, conflicts, version changes, and hostile instructions.
• Human effort includes review, failed attempts, escalation, and follow-up.
• Quality results include rejected drafts and errors caught before sending.
• Gains survive a comparison with improved search and templates.
• Continuing work has an owner and a funded capacity or spending allocation.
• Any claimed cash benefit identifies a specific avoidable expense or demand scenario.
• Unresolved incidents, uncertainty, and reasons to stop are visible in the decision memo.

Expand only within the tested scope. Revise when failures point to fixable source or interface problems. Defer when access boundaries cannot be enforced, documentation is unreliable, or eligible volume is too low. Improvements to search and knowledge ownership may still be worthwhile even when generated drafting fails the business case.

People and skills

Agents shift from composing replies to verifying them. Each material claim in a draft needs checking against the cited passage, the product version, and the verified account context. That requires fluency with roles and versions, the habit of opening the source rather than trusting the citation, and the confidence to send a shorter reply or a clarification instead of the draft. Automation bias, excessive deference to automated systems, is a named risk; the authority chapter lists its signs. As a proposed practice, route some tickets through ordinary search and templates so agents keep the skill of answering from documentation unassisted. The published study discussed below reported effects that differed across workers; that finding does not show how this team's agents will respond, so record edits, rejections, and errors by agent experience during the pilot.

Support leads take on reviewing rejected drafts and recurring failure patterns, and deciding which ones point to sources, interface, or training. Knowledge owners take on more visible work: article ownership, version applicability, retirement of archived rules, and resolving conflicts the assistant surfaces. Fund that work in the continuing-work budget rather than absorbing it silently.

Who you need

This pilot needs the roles below, described by what each is responsible for rather than by job title. The proposed source for each is the guide's proposed practice, not a finding: build work can come from outside, but anything the company owns after go-live stays in-house.

For the method behind this table, see Staffing an AI pilot; for bringing in outside help, see Choosing and working with an AI partner.

When to defer

Defer if help articles have no owners or review dates, if product versions are not tracked in articles or account context, or if account boundaries cannot be enforced before retrieval. Defer as well if eligible volume is too low to fund knowledge upkeep and evaluation, or if in-scope tickets turn mostly on troubleshooting that no approved article documents. In those cases, assigning article owners, tagging articles by version, retiring stale ones, improving search, and maintaining response templates are better first investments; they are also the inputs a later drafting pilot would need.

What external research establishes

For your analyst

Brynjolfsson, Li, and Raymond study human support agents receiving AI suggestions at one software company. The paper reports higher issues resolved per hour on average, with differing effects across workers. Its larger rollout used staggered adoption and difference-in-differences analysis, following a small randomized pilot. The authors lacked the pilot's original control-group identities and compared identified treated workers with other untreated colleagues. This matters when interpreting the evidence as a randomized experiment. Generative AI at Work, sections 3 and 4.1.1

That setting supports investigating supervised assistance. It does not establish the results of the fictional workflow above, current model rankings, autonomous support quality, or this company's cash savings. Our reading of the study covered its methods and appendix, not a full review of its robustness checks or how its outcome measures were constructed.

Evidence still needed and related reading

Obtain an attributable operator account of the baseline process, documentation constraints, and review burden. Measure eligible volume, adoption, resolution quality, actual operating cost, and the cash-realization mechanism. No operator has been contacted or customer outcome verified for this chapter.

Continue with support leaders, retrieval and source-grounded answers, pilot design, human review, and the economics worksheet. When comparing products, use build, buy, or configure and vendor diligence.

Related Neuronify resources: AI skills, AI ROI calculator, and project scoping pack.

Companion templates

Prepare your pilot with the pilot kit, evaluation scorecard, and evidence checklist. The companion workbook provides blank economics inputs and case records; worked chapter figures remain illustrative examples.