A retrieval-based assistant finds relevant information and supplies it as context for a generated answer. The practical goal is to help a reader use the right evidence with less search and interpretation effort.
Microsoft describes retrieval-augmented generation as a pattern connecting information retrieval with generation. That architecture gives a model access to selected source material; it does not guarantee that a generated claim is correct or that the selected source applies to the user's circumstances. Microsoft RAG overview
The term comes from a 2020 research paper by Lewis and colleagues, which combined a pre-trained language model with a dense vector index of Wikipedia accessed through a neural retriever, and reported that the combination produced more specific and factual language than a comparable model without retrieval on the tasks tested. Lewis et al., arXiv 2005.11401 Systems built today under the same name vary widely; the paper establishes the idea, not the behavior of any particular product or collection.
This chapter explains the parts of such a system that an operator or implementer has to decide. It is used by the support, onboarding, knowledge retrieval, and incident triage workflows. Design recommendations below are proposed practice; sourced statements are cited where they appear.
Understand the chain of work
The proposed system selects an authorized collection, prepares searchable records, retrieves candidate passages, constructs context, generates an answer, and presents evidence. Each stage can introduce a different failure.
| Stage | Question to ask | Example failure |
|---|---|---|
| Source selection | Is the information approved and accessible to this user? | Restricted client material enters the collection |
| Preparation | Are structure and metadata preserved? | A table loses its units or column headings |
| Permission filtering | Is the candidate set limited to what this user may see before ranking? | A restricted document shapes the ranking even if it is not shown |
| Retrieval | Did the system find the evidence needed? | Search misses a relevant product version |
| Ranking | Do the most useful passages reach the top? | A superseded article outranks the current one |
| Context assembly | Is enough surrounding information available? | An exception is separated from its rule |
| Generation | Does the answer follow the evidence? | The model invents a missing step |
| Presentation | Can the user inspect the support? | A citation points to a long unrelated document |
| Checking | Is each material claim supported by its cited passage? | A real citation accompanies a claim the source does not make |
Evaluate these stages separately before changing the model. An answer cannot reliably reflect evidence that retrieval never supplied.
Build a collection that can be maintained
For your knowledge owner
Begin with authoritative sources, owners, review dates, and permitted audiences. Preserve version and applicability metadata. If documents conflict, decide which source governs or make the conflict an explicit reason to escalate.
Indexing every available file can create more ambiguity than value. Old drafts, restricted client work, and superseded instructions need explicit treatment. A permission change or deletion must propagate to the searchable representation and relevant caches.
Metadata does as much work as text. A proposed minimum for each document:
| Field | Why it matters |
|---|---|
| Owner | Someone can fix it when it is wrong |
| Version and effective date | Retrieval can prefer the current rule and show history as history |
| Applicability | Product version, edition, region, or client the document covers |
| Access class | The permission filter needs it before ranking |
| Reuse status | Whether material may be quoted beyond its original audience |
| Review date | Stale material can be flagged rather than silently served |
| Supersedes / superseded by | Conflicts become visible instead of being blended |
Documents with no owner or no access label belong outside the index until someone classifies them. Treat a missing label as a curation task, not as permission.
Prepare documents so the pieces still make sense
For your technical lead
Long documents are split into smaller pieces, known as chunks, so that retrieval can return the relevant part rather than a whole manual. How the split is made changes what the system can find and what the model sees.
Microsoft's vendor guidance for its search service describes fixed-size chunks with some overlap between neighbors, variable-size chunks that follow sentences or document structure, and suggests a starting point of 512 tokens with 25% overlap for fixed-size chunking. Microsoft, chunking documents That is a vendor's suggested starting point for its product, to be tested on your documents, not a rule.
Proposed practices that matter regardless of size:
- Split along structure where it exists. A section, a procedure, or a table row with its headings is more interpretable than an arbitrary window of characters.
- Keep tables whole or repeat their headers. A row of numbers without its column names and units is evidence for nothing.
- Carry the document's metadata onto every piece. A passage that does not know its version, owner, or access class cannot be filtered or ranked correctly.
- Keep rules with their exceptions. If a procedure says “except for enterprise accounts”, the exception must travel with the rule.
Check preparation directly: sample pieces from the index and ask whether a reader could act on each one alone. If not, the model cannot either.
Choose retrieval for the actual questions
For your technical lead
Exact identifiers, product codes, and specific phrases can make keyword matching valuable. Questions phrased differently from the documents may benefit from semantic retrieval. Microsoft describes these complementary uses of keyword and vector search. A combined approach is a candidate to evaluate, not an automatic requirement or a verified improvement on this collection. Microsoft hybrid search overview In that product, a hybrid query runs full-text and vector search in parallel and merges the two result lists using Reciprocal Rank Fusion, a method that combines rankings rather than raw scores.
Test whether users need a generated answer at all. For a policy lookup, the correct passage and a clear link may be sufficient. For a support reply, a draft that synthesizes several approved sources may save work. The extra generation step should serve a defined purpose.
Preserve enough context for tables, exceptions, dates, and qualifications. Splitting documents into smaller searchable pieces is useful only if the resulting pieces remain interpretable.
Collect real questions before choosing. If users search by part number, error code, or contract clause, keyword retrieval may carry most of the load. If they describe a problem in their own words, semantic retrieval may find documents keyword search misses. Measure both on the same questions rather than assuming.
Enforce permissions before ranking
For your technical lead
Access control belongs before model context is assembled. Hiding a citation after a restricted passage has already been used does not restore the original information boundary.
Filtering must also come before ranking. If permissions are applied after a top-k cut, restricted documents can displace permitted ones from the results, change how many results are reported, or feed an aggregate such as a suggested colleague, even if their text is never shown. A reranker that scores candidates together can also be influenced by them. The knowledge retrieval chapter works through a case where any of those signals would reveal a confidential client relationship.
Implementation matters. In one vendor's filter-based pattern, which it offers for solutions that cannot use the product's built-in access-control support, the security identifiers on each document are plain strings used in a query filter, and the application must supply the caller's identities and apply the filter to every query. Azure AI Search security filters OWASP's community guidance lists misaligned access controls on embeddings and leakage across user groups among retrieval risks, and recommends permission-aware stores and partitioning. OWASP LLM08:2025 Neither source shows that any particular deployment is correct; both explain why the boundary needs testing with realistic accounts, including access granted and withdrawn.
Assemble context deliberately
For your technical lead
Retrieving the right passages is not the end of the job. The model receives them in some order and amount, and both affect the answer.
Research by Liu and colleagues found that the language models they tested on multi-document question answering and key-value retrieval performed best when relevant information appeared at the beginning or end of the input and degraded when it sat in the middle of long contexts, including for models built for long inputs. Liu et al., Lost in the Middle, arXiv 2307.03172 The study tested specific 2023 models and tasks; current systems may differ. It supports a narrower practice: test whether answer quality changes with the amount and order of context, rather than assuming more retrieved text helps.
Proposed practices:
- Cap the number of passages and send those that pass a relevance cutoff validated on your own questions, rather than a fixed large number. Check what happens when no passage passes the cutoff.
- Label each passage with its source, version, and date so the model and the reader can tell current from superseded.
- When two current sources conflict, present the conflict to the reviewer instead of asking the model to reconcile it.
- Keep instructions and retrieved material visibly separate, so a reviewer can tell which is which and the application can limit what retrieved text is allowed to change. Separation reduces the prompt-injection risk described below; it does not eliminate it.
Evaluate support and completeness separately
For your analyst
Ask whether the retrieved material contains the needed answer. Then ask whether each material generated claim is supported by that material. Finally, ask whether the answer covers the user's actual question.
An answer can be fully supported but incomplete. It can be complete-looking but unsupported. A correct citation can accompany a claim the source does not make. Review citations for entailment, not simply their existence.
Two separate checks, four kinds of answer
Complete
Incomplete
Supported
“Settings → Audit → Export CSV. Exports cover the preceding 30 days.”
The goal: every claim in the source, and the question fully answered.
“Use Settings → Audit → Export CSV.”
Every claim checks out, but it leaves out the 30-day window the administrator needs.
Unsupported
“Go to Account → History and export. Exports cover the preceding 30 days.”
Looks complete, but the path comes from the archived version-3 guide.
“Audit export isn’t available in version 4.”
Unsupported and leaves the question unanswered.
Include questions the collection cannot answer. The expected behavior may be a clarification, a statement of missing evidence, or escalation. Rewarding a response to every question encourages guessing.
Design the evaluation around three measures
Score retrieval and generation separately, so a failure can be traced to the stage that caused it.
| Measure | Question | How to score |
|---|---|---|
| Required-source retrieval | Did the passages needed for a correct answer reach the candidate set? | For each answerable case, list the required sources in advance; count cases where all were retrieved |
| Claim support | Is each material claim entailed by its cited passage? | A reviewer checks each claim against its citation; one unsupported material claim fails the answer |
| Completeness | Does the answer address what the user actually asked? | Rubric per case, written before results are seen |
| Correct abstention | For cases with no permitted answer, did the system say so? | Count among unanswerable cases only |
| Boundary compliance | Did any restricted material reach ranking, context, or output? | Test accounts with realistic entitlements; any leak is critical |
If required-source retrieval is low, changing the model will not help; fix preparation, metadata, or retrieval first. If retrieval is high but support is low, the generation step or its instructions need work. Report each measure with its denominator, and keep the ordinary sample separate from deliberately difficult cases.
Treat retrieved content as information
For your technical lead
Documents can contain instructions, misleading text, or content from people outside the organization. Those passages should not expand the system's tools or permissions. Keep the allowed behavior in the application design and test whether retrieved material can redirect it.
NIST defines prompt injection as an attack exploiting untrusted input combined with a higher-trust prompt. That supports testing this boundary; the application controls above are our proposed design, not proof that injection has been eliminated. NIST glossary OWASP describes indirect prompt injection through external content, and mitigations including least privilege, segregating untrusted content, and human approval for high-risk operations. OWASP LLM01:2025
For an initial knowledge assistant, read-only operation can be a useful boundary. If later versions can act, evaluate those action permissions separately from answer quality. Include documents with embedded instructions in the evaluation set, and record whether behavior changed.
Account for cost and latency
Each stage adds cost and time: embedding new and changed documents, running searches, sending retrieved text to the model, and generating the answer. Larger contexts cost more and take longer, and a reviewer reading long cited passages spends more time than one reading a short, precise one.
Measure the whole path on real questions: time to first useful result, time for the reviewer to verify, and cost per answered question including failed and abandoned attempts. A configuration that returns faster but sends reviewers to read whole documents may cost more in total. Re-indexing is a recurring cost when collections change quickly; include it in the budget along with curation.
When search alone is enough
Generation is not always the useful step. Prefer a well-maintained search with good metadata and clear snippets when:
- the answer is a single passage the user can read directly;
- users need to see the source text verbatim, as with policy or contract language;
- the questions use the same identifiers the documents do;
- reviewers would read the source anyway before acting.
In these cases, the work that pays off is curation: owners, versions, labels, and retiring stale material. That work is also a prerequisite for any generated layer added later.
A practical pilot
Collect real questions from the intended users and define approved answers or supporting passages. Compare search alone with generated answers. Measure time to a verified useful result, missing evidence, unsupported claims, citation quality, and human correction effort.
Include stale documents, conflicting versions, restricted sources, and questions outside the collection. Observe what reviewers actually inspect. If verification takes as long as the original research, the economics should reflect that result.
Assign ownership for source updates and problem reports before expansion. Defer when no one can decide which documents are authoritative or when ordinary search already serves the workflow adequately.
Related chapters: project knowledge, support replies, onboarding guidance, incident triage, model selection, evidence, human review.