Capabilities: perceive (speech recognition), extract, generate (speech synthesis), act (voice agents). Related workflows: meeting follow-up, support. Human control: depends on the use, from review of a transcript to confirmation before a spoken request triggers an action.
Speech systems turn audio into text, decide who said what, and, in voice interfaces, answer back. The decision this chapter supports is narrower than "should we use voice AI": it is whether a particular speech component performs well enough, on your speakers, audio, and vocabulary, for the job it would do, and whether the people relying on it can catch its mistakes.
The meeting follow-up workflow is this guide's canonical worked example: it shows a transcript excerpt, a misheard number, and how a reviewer catches it. This chapter covers the technology underneath and the questions that apply to any speech use. Descriptions of how components work draw on the sources cited; the measurement and control practices are this guide's proposed design, not observed results.
The decision
A speech project starts with a task, not with a speech model. Three common shapes need different things from the technology:
- Batch transcription. Recorded audio becomes text after the fact: a meeting, an interview, a call recording. Accuracy matters more than speed, and a person can review before anything depends on the text.
- Live assistance. Text appears while people talk, for captions, agent prompts, or note-taking. Delay matters, and the text may be corrected only after someone has already read it.
- Voice interaction. A system listens, responds in speech, and may take an action: a phone menu that understands requests, or a voice agent that books, cancels, or changes something. Here recognition errors can become actions, and speed shapes whether the conversation feels usable at all.
The further a use sits toward voice interaction, the more the design has to plan for misrecognition, interruption, and confirmation, and the less a later reviewer can repair. A team that has only evaluated batch transcription has not evaluated a voice agent.
The parts of a speech system
For your technical lead
Each part produces something that can be checked separately. Evaluating only the final output hides which part failed.
| Component | Output | How to check it | Failure that matters |
|---|---|---|---|
| Audio capture | Recording or stream | Listen to samples from each channel; record sample rate, channel, and noise conditions | Clipped, distant, or overlapping speech that no model can recover |
| Speech recognition (speech-to-text) | Words, with timestamps and sometimes confidence | Compare with a reference transcript; inspect critical tokens | A misheard name, number, or date; dropped or invented words |
| Speaker attribution (diarization) | Which speaker said which segment | Compare segment labels with a reference | An action or commitment attributed to the wrong person |
| Understanding the transcript | Extracted items, intents, or summaries | Check each item against its audio timestamp | A hedge recorded as a commitment; an intent misread |
| Speech synthesis (text-to-speech) | Spoken response | Listen for pronunciation of names, numbers, and terms | A correct answer that the listener mishears |
| Dialogue and action | A reply, a question, or a call to another system | Trace each action to the confirmed request | An action taken on a misrecognized request |
The first row can be the cheapest to improve, and it is easy to leave out of planning. A recognition model receives whatever the microphone captured. A room microphone at the end of a conference table, a phone line, and a headset produce different inputs from the same speaker.
Batch and live are different problems
For your technical lead
In batch transcription the whole recording is available, so the system can use later context to resolve earlier words, and a reviewer can replay any segment. The meeting follow-up workflow relies on that: every extracted item links to a timestamp a person can open.
Live use removes both advantages. Text must appear before the speaker has finished, so the system commits to words with less context and may revise them a moment later. In a voice interface, the system must also decide when the person has stopped speaking, handle interruptions, and keep a delay short enough that people do not talk over it or give up. Those requirements belong in the requirements sheet described in model selection, stated as a latency limit at a stated percentile rather than an average, because the slow responses are the ones people notice.
A practical consequence: a configuration that wins a batch comparison may not be usable live, and the reverse. Evaluate the mode you will run.
Measure what the task needs
For your analyst
Word error rate, and its blind spot
Word error rate counts substitutions, deletions, and insertions relative to the number of words in a reference transcript. Koenecke et al., 2020, PNAS It is the standard starting measure and it is useful for comparing configurations on the same audio. It weights every word equally, so a dropped "the" and "fifteen" heard as "fifty" cost the same.
For most business uses a small number of words carry the consequence: names, numbers, dates, amounts, product and account identifiers, and negations ("do not renew"). As a proposed practice, measure a critical-token error rate alongside word error rate: of the critical tokens in the reference, how many were transcribed wrongly or dropped. The meeting follow-up chapter applies this to recaps, where a misheard quantity changes what the team commits to.
Who said it
Speaker attribution errors are separate from word errors. A transcript can be word-perfect and still assign an action to the wrong person. When owners or commitments matter, score attribution on the items that depend on it: for each extracted action, was the speaker correct? Overlapping speech and similar voices on one channel are the conditions to include.
Delay, for live use
For live captions and voice interfaces, record the time from the end of speech to text or response, as a distribution. Include the slow tail, network conditions, and peak load. A single average hides the experience that drives abandonment.
By group and by condition
Speech recognition performance can differ systematically between groups of speakers. Koenecke and colleagues tested five commercial systems on structured interviews and reported an average word error rate of 0.35 for Black speakers and 0.19 for white speakers; the study concerns systems tested before 2020 on interview audio, and the two groups' recordings came from different regions and collection efforts. Koenecke et al., 2020, PNAS The narrower lesson for any project: report results separately for the speakers, accents, languages, channels, and noise conditions you actually have, rather than one overall rate. An acceptable average can conceal a group for whom the system does not work.
The authors of one widely used open recognition model report a similar dependence on data: across the languages of one multilingual benchmark, error rates fell as the amount of training audio for a language rose, with a strong correlation on a log scale. Radford et al., 2022 That finding concerns one model family and its training data in 2022; it supports testing in each language and dialect you serve, not any particular expected rate.
Failure modes and controls
For your technical lead
Speech systems share the failures of the models that process their output, and add some of their own.
| Failure | Consequence | Detection | Proposed control |
|---|---|---|---|
| Critical token misheard | Wrong quantity, name, or date reaches a record or reply | Critical-token checks in evaluation; reviewer flags at low confidence | Confirm critical values back to the speaker in live use; link each extracted value to its audio in batch use |
| Words invented or repeated | Text that was never said appears in the record | Compare with reference on long recordings; check silence and noise segments | Segment long audio; review items that rest on a single unclear passage |
| Start or end of a segment dropped | A condition or negation is lost ("...unless the client objects") | Reference comparison at segment boundaries | Overlap segments; treat items near boundaries as needing review |
| Wrong speaker attributed | Action assigned to someone who did not accept it | Attribution scoring on owned items | Leave owners unresolved unless attribution is reliable; ask participants |
| Out-of-vocabulary terms | Product names and jargon turned into common words | Term list checks in evaluation | Supply vocabulary hints if the service supports them; maintain a glossary for review |
| Spoken instructions treated as commands | Text in the audio steers a downstream model | Adversarial test recordings | Treat transcripts as data, not instructions (see below) |
| Synthetic or impersonated voice | A request appears to come from an authorized person | Out-of-band verification tests | Do not authenticate by voice alone for consequential actions |
The second and third rows are worth stating plainly because they can look like ordinary transcripts. The authors of one open recognition model describe failure modes in long-form transcription that include getting stuck in repeat loops, not transcribing the first or last few words of an audio segment, and complete hallucination of a transcript unrelated to the audio. Radford et al., 2022, section 6 Those observations concern that model family in 2022; they are a reason to include long recordings, silence, and segment boundaries in any evaluation, not an estimate of how current systems behave.
Voice interfaces that take action
When a spoken request can change something, recognition error becomes an authority question. The authority chapter describes the arrangements; for voice, three proposed controls follow.
Confirm consequential actions in a form the person can check. Read back the specific action and its key values ("cancel the order for 12 cases, delivery Thursday?") and require an explicit confirmation before acting. A confirmation that only repeats the system's interpretation is useful only if the interpretation is shown or spoken in full.
Make the human route easy and always available. A caller who is not understood after a small, fixed number of attempts should reach a person without starting over. Record those handoffs; they are evaluation data.
Do not treat a voice as an identity. The US Federal Trade Commission has stated that voice cloning technology could make it easier for scammers to impersonate family, friends, or business executives. FTC, November 2023 Whatever the system's recognition quality, an action that matters should rely on the organization's existing authentication, not on how a caller sounds.
Transcripts are also untrusted input. Words spoken in a call, including words designed to manipulate a downstream model, arrive as text that a language model may read. Prompt injection, covered in the retrieval chapter, is the broader category of manipulating a model through input it processes. Keep the system's permissions narrow, keep instructions separate from transcript content, and test with recordings that contain instruction-like speech.
Consent, recording, and retention
Recording and transcribing conversations can be subject to consent requirements that vary by jurisdiction, by the parties involved, and by contract. This guide does not state those requirements. As a proposed process gate, confirm consent and the applicable policy before recording, record that confirmation, and decide in advance how long audio, transcripts, and derived summaries are kept and who can access them. Audio may be more sensitive than the transcript because it carries the voice itself. The meeting follow-up chapter treats consent as the first step in its workflow map.
Design choices
For your technical lead
Improve capture before changing models. Microphone placement, a headset for agents, separate channels for each party on a call, and reduced background noise can change recognition more than a model change, and they are cheaper to test. Compare configurations that differ only in capture.
Decide where the audio goes. A hosted speech service and a model run on the organization's own infrastructure differ in data handling, operating effort, and how updates arrive. The open-weight and hosted deployment chapter covers that decision; data-handling requirements are pass-or-fail gates, as model selection describes.
Separate recognition from understanding. Store the transcript with timestamps before any summarization or extraction. Then a reviewer can trace each item to audio, and a change to the extraction step does not require re-recording.
Keep confidence and uncertainty visible. If the service returns word-level confidence, show low-confidence critical tokens to the reviewer rather than hiding them in fluent text, and check locally whether low confidence actually predicts errors before relying on it.
Plan for the vocabulary. Expect product names, client names, and internal terms to be among the likeliest and costliest misrecognitions, and test them explicitly. Maintain the term list as a curated asset with an owner.
Build the evaluation
For whoever runs the pilot
Construct a test set from audio you are permitted to use, with reference transcripts created or checked by people who know the vocabulary. As an illustrative coverage plan, not a statistically justified sample, include:
- ordinary recordings from each channel you will use (phone, room, headset);
- every speaker group, accent, and language you serve, reported separately;
- noisy, overlapping, and quiet segments, and long recordings with pauses;
- passages dense with critical tokens: numbers, dates, names, negations;
- for voice interfaces, interruptions, corrections ("no, I said Thursday"), requests outside scope, and instruction-like speech.
Score word error rate, critical-token error rate, attribution on owned items, and, for live use, delay at a stated percentile. For voice interfaces, score task completion with confirmation, wrong actions prevented by confirmation, and handoffs to people. Report counts with denominators for each group and condition. The pilot design playbook covers stages and acceptance criteria; the human review playbook covers how reviewers check transcripts and when items escalate.
Re-run the evaluation when the service, model version, capture setup, or vocabulary changes. Speech services can be updated by the provider; recording the exact identifier and re-evaluation triggers is part of model selection.
Cost and latency
Costs come from audio processing time, storage, any downstream language-model processing, and the people who review transcripts and handle handoffs, which may be the largest line. Voice interfaces add telephony and the cost of calls that fail and reach a person anyway. This guide gives no prices; compare configurations on the cost of an acceptably completed task including review and handoff, as the model selection chapter shows for text.
For live use, delay is a requirement, not a cost line. A cheaper configuration that is slower may fail the requirement outright.
When a simpler approach works better
Speech is a strong interface when hands and eyes are busy, when the information is already being spoken, or when typing is a barrier. It is a weak one for entering precise values. Alternatives to compare against:
- Structured forms and fields for numbers, dates, and identifiers, where a typed or selected value avoids recognition entirely.
- Keypad and menu options for a short list of choices in a phone flow.
- Human notes with a template for meetings where recording is not permitted or the stakes of a misheard value are high. The meeting follow-up chapter compares this baseline directly.
- Better search and articles in support, where the problem is finding the answer, not capturing the question.
If a speech system mainly adds review work to catch its own errors on precise values, the simpler alternative is likely to win on total effort.
People and skills
Reviewers need to check text against audio efficiently: jump to the timestamp, recognize when a fluent sentence rests on one unclear word, and know which tokens matter. The automation-bias risk applies: people may defer to a clean-looking transcript. NIST AI 600-1, section 2.7 Someone must own the vocabulary list and the evaluation set. For voice interfaces, the people who take handoffs need the full context of what the caller already said, or the handoff costs more than it saves.
What external research establishes
For your analyst
- Definition of word error rate, and evidence that commercial recognition systems tested before 2020 performed worse for one group of speakers than another on interview audio. Koenecke et al., 2020
- One open model family's own account of its training scale (680,000 hours of weakly supervised audio), of error rates falling with more training audio per language, and of long-form failure modes including repeat loops, dropped segment boundaries, and hallucinated transcripts. Radford et al., 2022
- A regulator's statement of concern that voice cloning could make it easier to impersonate family, friends, or business executives. FTC, 2023
None of these sources measures the performance of any current product, any business workflow's error rate, or the effect of speech systems on effort or outcomes. The controls in this chapter are proposed practice.
Evidence still needed and related reading
Still needed: measured critical-token and attribution error on permitted business audio across channels and speaker groups; reviewer effort for checking transcripts; handoff and completion rates for a voice interface; and the effect of capture improvements compared with model changes. Consent and retention practice needs review by counsel for the relevant jurisdictions. No operator has been interviewed and no system tested for this chapter.
Related reading: meeting follow-up, support replies, model selection, open-weight and hosted deployment, human review, and vendor diligence.