Roles covered: support director or manager, team leads, quality reviewers, knowledge owners. Functions: service, with handoffs to engineering and customer success. Linked workflows: support replies, customer onboarding, incident triage.
The decisions you own
Your decision is whether assistance improves the service customers receive with the resources available. Reply volume is an intermediate measure. A team that sends more responses but resolves fewer requests has not become more productive; the support-reply chapter explains why the unit of work is a resolved request.
Three decisions belong to you rather than to engineering, finance, or a vendor:
- The service promise and the definition of resolution. What counts as a resolved request, over what observation window, and which errors are unacceptable regardless of speed. Engineering can enforce permissions; it cannot decide whether a customer received a useful answer.
- Which work enters assistance, and at what authority. Which request categories are eligible, which stay with specialists, and whether the system drafts, recommends, or acts. Starting authority for customer-facing replies in this guide is a draft that an agent approves before sending.
- How released capacity is used and who funds the upkeep. Whether saved time goes to backlog, quality, training, or an avoided contractor expense, and who pays for the knowledge maintenance and quality review the assistant depends on.
The measure that looks like progress but is not: average handle time on assisted tickets alone. It excludes rejected drafts, follow-ups, and the unassisted work that remains, and it can fall while customers wait longer for real answers.
Map the work you manage
Support includes understanding the request, finding relevant information, deciding what action is permitted, explaining the answer, and learning from recurring problems. Assistance may help these parts at different rates. A draft assistant can reduce search and writing effort while leaving difficult account decisions unchanged.
Map the queue by issue type and consequence before choosing a starting category. Product how-to questions, payment disputes, account access, and security reports should not enter an experiment under one undifferentiated accuracy target.
The following inventory is a proposed starting point. Adapt the consequences and authority levels to your product and contracts.
Capability verbs in the table follow the capabilities chapter; starting authority uses the arrangements in assistance, approval, and autonomy.
| Task | Consequence of an error | Candidate capability | Proposed starting authority | Canonical chapter or first fix |
|---|---|---|---|---|
| Categorize and route incoming requests | Delay; wrong specialist | Classify | Assistance; agent or rule confirms routing | Support replies |
| Answer documented how-to questions | Wrong instruction; repeat contact | Retrieve, generate | Approval before action; agent approves before sending | Support replies |
| Troubleshoot suspected defects | Misdiagnosis; delayed escalation | Retrieve, summarize | Assistance; agent decides escalation | Support replies, escalation boundary |
| Billing disputes and refunds | Financial and contractual harm | None at first | Unchanged; existing procedure | Finance policy, not this guide |
| Account access and permission changes | Unauthorized access | None at first | Unchanged; verified procedure only | Support replies, permission boundary |
| Security reports | Mishandled vulnerability or breach | None | Unchanged; security process | Incident triage, scope note |
| Customer questions during an outage | Inconsistent or premature statements | Summarize approved status updates | Approval before action; incident commander owns wording | No dedicated chapter yet; incident triage covers the internal responder brief |
| Setup questions from new accounts | Wrong edition or entitlement | Retrieve, generate | Approval before action; agent approves before sending | Customer onboarding |
| Knowledge-article maintenance | Stale or contradictory guidance | Classify gaps, draft revisions | Approval before action; knowledge owner approves | Retrieval |
| Quality review of sent replies | Missed systemic errors | Sample and flag | Assistance; reviewer scores | Human review |
| Reporting recurring problems to product | Defects persist | Cluster and summarize | Assistance; lead confirms themes | No dedicated chapter yet |
Several of these rows start with no assistance. That is deliberate. Where a fixed instruction answers the question, a maintained template may do most of the useful work; where agents cannot find current guidance, better search and article ownership may matter more than generated drafts. The support-reply chapter proposes comparing those three configurations on the same cases; include that comparison in your first evaluation.
Separate the work you perform from the work you supervise. You may not draft replies yourself, but you own the queue design, the escalation paths, the quality standard, and the staffing plan that determine whether an assistant helps.
Who decides what
Responsibilities blur quickly once a tool touches several teams. A proposed allocation:
| Decision or activity | Support leader | Agents | Knowledge owner | Engineering or IT | Security or risk | Finance |
|---|---|---|---|---|---|---|
| Eligible categories and excluded actions | Owns | Consulted | Consulted | Consulted on feasibility | Approves exclusions touching access or data | Informed |
| Definition of resolution and severity rules | Owns | Consulted | Consulted | Informed | Consulted on critical categories | Informed |
| Sending a customer reply | Accountable for the standard | Approves and sends each reply | — | — | — | — |
| Source content and article accuracy | Funds and sets priorities | Report gaps | Owns | Maintains ingestion | — | — |
| Permission boundaries and account isolation | Specifies requirements | Report suspected leaks | — | Owns implementation | Approves design | — |
| Disabling the assistant | Can disable | Can decline any draft | Can withdraw a source | Can disable | Can disable | — |
| Realization of released capacity | Proposes use | Consulted | — | — | — | Reviews any cash claim |
Three distinctions matter. Owning the outcome (you) differs from operating the system (engineering) and from approving an external action (the agent who sends). Declining a draft must always be easy and never penalized. Several people can switch the assistant off, and no one needs permission to do so when a critical failure is suspected.
Plan a first quarter
The following is a fictional planning example. Mossgrange is an invented maker of project-management software for architecture firms; its team, volumes, and queue mix exist only for this illustration.
The support team has 28 agents in two tiers, a knowledge owner who spends part of her week on articles, and two quality reviewers. It handles about 6,000 requests a month. An invented queue mix: 45% product how-to questions, 20% suspected defects, 12% billing questions and disputes, 10% account access and user management, 8% integration questions, 1% security reports, and 4% other.
Weeks 1–3: measure before choosing a tool. The leader samples recent requests from each category and records total human effort to resolution, including follow-ups, over a fixed observation window. The team tags whether each answer came from an article, a template, a colleague, or an agent's memory. That tagging answers a question no vendor can: how much of the effort is finding information, and how much is judgment.
Choose the first category. Product how-to questions are the plausible start: high volume, documented answers, reversible errors, and a clear source of truth. Within that category, the leader narrows eligibility further to the current product version and to articles with a named owner and a review date. Stale areas of the help center are excluded until someone owns them.
The counterexample: account access. Account access has attractive volume, and many of the requests look routine. It is a poor first category. A plausible but wrong reply here can grant or imply access the customer is not entitled to, the harm is not easily reversed, and the correct answer depends on verified identity and contract facts that a text assistant does not establish. Billing disputes fail the same test for a different reason: the answer is a financial decision governed by policy and contract, not a documentation lookup. Both stay on the existing procedure. If volume in those categories is the real problem, the better first investment may be a self-service flow or a clearer policy, not generated text.
The first pilot covers less than half the queue
- First pilot
- Excluded, with stated reason
- Not in the first pilot
Product how-to questions · 45%
First pilot: high volume, documented answers, reversible errors; narrowed to the current version and owned articles
Billing questions and disputes · 12%
Excluded: a financial decision governed by policy and contract
Account access and user management · 10%
Excluded: a wrong reply can grant access; the answer depends on verified identity and contract facts
Everything else · 33%
suspected defects, integration questions, security reports, other: not in the first pilot
Show the data
| Category | Share of requests | First pilot |
|---|---|---|
| Product how-to questions | 45% | First pilot: high volume, documented answers, reversible errors; narrowed to the current version and owned articles |
| Suspected defects | 20% | Not in the first pilot |
| Billing questions and disputes | 12% | Excluded: a financial decision governed by policy and contract |
| Account access and user management | 10% | Excluded: a wrong reply can grant access; the answer depends on verified identity and contract facts |
| Integration questions | 8% | Not in the first pilot |
| Security reports | 1% | Not in the first pilot |
| Other | 4% | Not in the first pilot |
Weeks 4–6: offline replay. Build a case set that includes ordinary how-to questions plus deliberately hard cases: missing product version, conflicting articles, a request that embeds a permission change, and text that tries to instruct the system. The support-reply chapter sets out an illustrative coverage plan and test variations. Fix source and access problems found here before any agent uses the tool live.
Weeks 7–11: supervised pilot. Trained agents in one tier use the assistant on the eligible queue, with existing escalation paths unchanged. Compare against current process and against improved search and templates on comparable work. Measure all assigned cases, including rejected drafts and fallbacks, and follow them through the observation window.
Week 12: decide. The leader writes a decision memo with results by category and agent group, failures found before and after sending, continuing costs, and the proposed use of any released capacity. The decision can be to expand within the tested scope, to fix sources and retest, or to stop and keep the search improvements.
The pilot design playbook and the pilot kit provide the charter and memo structure. The economics, including how review time and adoption change the result, are worked through in the support-reply chapter; this guide does not repeat them.
Measure the right things
| Measure | Why it matters | How it can mislead |
|---|---|---|
| Human effort per resolved request, including follow-ups | Captures end-to-end labor | Excluding follow-ups or rejected drafts hides rework |
| Reopen and repeat-contact rate within a fixed window | Tests whether the answer held up | Case mix can change between periods; shorter windows flatter results |
| Major and critical errors, before and after sending | Separates assistant quality from reviewer catches | Counting only sent errors lets strong reviewers conceal a weak assistant |
| Correct escalation among cases requiring escalation | Tests whether hard cases leave the assisted path | A low escalation rate can mean the assistant is answering what it should not |
| Draft rejection and heavy-edit rate | Shows whether drafts save work | A falling rate can reflect reviewer fatigue rather than better drafts |
| Customer outcome (satisfaction or effort survey) | Connects output to usefulness | Respondents may differ from nonrespondents; check coverage |
| Knowledge-maintenance effort | Captures a recurring cost | May sit outside the support budget and go unfunded |
| Time to competence for new agents | Tests whether assistance supports or replaces learning | Hard to attribute; needs a defined proficiency check |
Segment results by issue category and agent experience. A system that helps new agents can still hinder specialists. Do not require identical usage when the useful tasks differ. Report counts with denominators rather than a single accuracy percentage that mixes easy traffic with challenge cases.
Your team: work, judgment, and skills
The following are proposed changes to plan for and measure. This guide does not report how any support team has actually changed.
Agents. Drafting and searching may shift toward reviewing: checking the customer's circumstances, the applicability of the cited source, the permitted action, and whether the reply actually answers the question. Reading for tone alone is insufficient. Agents need to open the cited passage rather than trust the citation, recognize when an article is for a different version, and notice when a request contains an action the assistant must not take. Those are teachable skills; include them in onboarding and in calibration sessions.
Specialists and senior agents. Their judgment on ambiguous defects, edge-case configurations, and angry or high-stakes customers remains central. A published study of one company's support deployment reports that less experienced and lower-skilled agents improved in speed and quality, while the most experienced and highest-skilled agents saw small gains in speed and small declines in quality. Generative AI at Work, v2 abstract The study examines a specific deployment using a staggered rollout, and some details of how its outcomes were constructed are not public. For your team, treat the pattern as a reason to measure effects separately by experience, and to let specialists decline suggestions without penalty, not as a prediction of what your agents will experience.
Knowledge owners. The assistant depends on articles with owners, applicability, effective dates, and retirement rules. Knowledge ownership may become a defined, funded responsibility rather than a side task. The owner resolves contradictions surfaced by the system instead of letting a model blend them.
Quality reviewers. Reviewers score material claims against permitted evidence, classify severity consistently, and look for patterns that point to fixable source or interface problems. Two reviewers should independently score a subset and reconcile the rubric before aggregate scores are trusted.
Team leads. Leads coach on judgment rather than speed, watch for over-reliance, and route failure reports to the right owner.
Protect practice. When drafts remove the work of searching and composing, new agents can lose the practice through which they learn the product. Protect time for employees to learn product behavior directly: periodic unassisted cases, shadowing specialists, and reviewing why drafts were rejected. Ask reviewers to explain their corrections and build those examples into training. The goal is informed use of assistance, not dependence on plausible drafts.
Staffing, role definitions, and career paths are decisions you will need to make, with HR and finance, based on measured results. Plan them explicitly. Do not announce them on the basis of a vendor projection or an average from another company's study.
Who to bring in. A first support pilot can draw on help from outside your team for three roles: a builder to configure the assistant and permission-filtered retrieval, integration work to connect the help desk and knowledge base without crossing account boundaries, and an independent evaluator to build the offline replay set, hard cases included, and score drafts against the rubric. Security or risk approves the access design whoever builds it. Keep in-house what the "Who decides what" table assigns to support: eligible categories, the definition of resolution, article ownership, and each agent's approval of a sent reply. See Staffing an AI pilot and Choosing and working with an AI partner.
Change management and governance
Communicate the purpose and the limits. Tell the team what the pilot is testing, which categories are excluded, who can switch it off, and how results will be used. If released capacity will go to backlog reduction or training rather than headcount changes, say so; if that has not been decided, say that too.
Involve the people doing the work. Agents can supply the hard cases for the evaluation set, identify articles they no longer trust, and describe where time actually goes. Their participation improves the case set and makes the results more credible to the team.
Change targets that reward unsafe reliance. A handle-time target pushes agents to accept drafts without checking. During the pilot, measure effort to resolution and quality instead, and avoid individual speed rankings based on assisted work.
Make failure reporting cheap. Provide a one-step way to flag a bad draft, and route reports to the knowledge owner or engineering rather than making agents investigate. Treat a caught error as useful information, not an agent failure.
Handle monitoring data carefully. Evaluation records can include agent edits and timing. Agree on who can see individual-level data and for what purpose, and check applicable employment, privacy, and works-council obligations with the appropriate advisers before collecting it. This guide does not state those requirements.
Keep the fallback working. Agents must be able to complete every case without the assistant. If the tool fails, work continues through ordinary search and templates.
Warning signs
These are signals to watch during a pilot, not reported frequencies.
| Signal | What it may indicate | Proposed response |
|---|---|---|
| Draft acceptance rises while repeat contacts also rise | Reviewers accepting drafts they have not checked | Audit a sample of accepted drafts; revisit targets and training |
| Escalations from the eligible queue fall sharply | Assistant answering cases that should escalate | Review escalation-required cases; tighten eligibility |
| Agents stop flagging bad drafts | Reporting is costly or feels futile | Simplify reporting; show what changed after reports |
| Knowledge owner backlog grows | Upkeep is unfunded | Fund or reduce scope before expanding |
| Specialists work around the tool | Drafts slow their work or reduce quality | Measure by group; allow opt-out for complex categories |
| Results look strong only for enthusiastic adopters | Selection, not effect | Compare assigned groups, not self-selected users |
| A draft reveals another customer's information | Access boundary failure | Treat as critical; pause the integration and investigate |
Questions to ask vendors and internal teams
- Where are permissions enforced: before material reaches the model, or only in the final answer?
- Can each draft show the exact passages and article versions used, and what the system could not establish?
- Can the assistant take any action on an account, or send anything, through this integration? How is that prevented beyond the prompt?
- What happens to a request when the assistant is unavailable or slow?
- Can we export case-level records of drafts, edits, and outcomes for our own evaluation?
- How are model, prompt, or retrieval changes announced, and can we rerun our evaluation before they reach agents?
- What does usage cost at our volume, including rejected drafts and retries?
- Who inside our company owns each source collection, and who funds its maintenance?
When to wait
Pause or defer if documentation is unstable, account boundaries are unclear, eligible volume is small, or the assistant creates more follow-up work than it saves. Defer if no one will own the knowledge base, or if the team cannot measure effort to resolution today.
The simpler investment may be the right first step: assign article owners, retire stale content, improve search, and maintain templates for fixed answers. Those improvements help agents immediately and are prerequisites for any generated drafting. If the root cause of a large queue is a product defect or a confusing policy, the next investment may be a product fix, not a support tool.
What external research establishes
For your analyst
Brynjolfsson, Li, and Raymond study a generative AI assistant introduced to 5,172 customer-support agents at one software company. Version 2 of the paper reports a 15% average increase in issues resolved per hour, with substantial differences across workers: less experienced and lower-skilled agents improved in speed and quality, while the most experienced saw small speed gains and small quality declines. Generative AI at Work, v2 The larger rollout was staggered and analyzed with difference-in-differences methods after a small randomized pilot. Methods, sections 3 and 4.1.1 How the outcome measures were built, and how the paper's versions differ in detail, could not be fully verified from the public record. The study supports investigating supervised assistance and measuring effects by experience; it does not establish results for another company, current models, or autonomous support.
Automation bias is a named risk, not a measured reviewer failure rate; the authority chapter lists the signs that approval has become a formality.
We have found no attributable evidence on how support roles, staffing, or career paths change after adoption, and this guide makes no such claim. The team structures, responsibilities, and skills above are proposed practices.
Related reading
Continue with support replies, customer onboarding, incident triage, B2B software, human review, pilot design, and engineering leaders. For the economics framework, see AI economics.
Related Neuronify resources: AI skills, which describes skills named in job postings at New York AI companies, not support-team outcomes.