AI-assisted project knowledge retrieval

Industry: professional services. Roles: consultant, delivery lead, knowledge manager, conflicts or risk owner. Capabilities: retrieve, summarize. Human control: source verification before use. Benefit: delivery capacity and reuse of permitted work. Costs: curation, permissions, retrieval, review, and maintenance.

The decision

A services firm wants consultants to find approved methods, relevant prior experience, and the right colleague to ask without reconstructing past projects by hand. The first useful boundary is an assistant that answers a knowledge request from a curated, permission-filtered collection and cites the passages it used. It does not draft deliverables, decide whether a method fits a new engagement, or grant access to anything.

The unit of work is a knowledge request resolved with verified, permission-appropriate, source-linked material, or with a correct "nothing you can use was found." A fluent summary the consultant cannot trace does not resolve the request. A helpful answer built on material the consultant was not entitled to see is a serious failure, however useful it looked.

The defining constraint is client confidentiality. Prior-engagement material can be bound by client contracts, non-disclosure agreements, and conflict or ethical-wall arrangements that bar particular staff from particular matters. Reusable methods and client-confidential deliverables can sit in the same folders. A retrieval system inherits the access mistakes already there and makes them faster to exploit.

The firm, clients, documents, people, dates, budgets, and thresholds below are invented teaching material. They are not customer records, measured results, vendor claims, or industry benchmarks. Recommended controls are proposed practices, labeled as such. The research section keeps external sources separate.

Map the work before choosing a model

Today a consultant searches shared drives with a few keywords, opens candidate files, and asks colleagues in chat. The hard judgment is whether a method built for one client's data and constraints transfers to another. Waiting for replies is elapsed time, not effort; measure the two separately.

Workflow map

Question → identity and entitlement lookup → permission-filtered retrieval → cited answer package → consultant verification → use, follow-up, or research request

If entitlement lookup fails, the system returns nothing rather than falling back to an unfiltered search. If retrieval finds no permitted material, the answer says so and offers a research request to the knowledge team. If a cited passage does not support the answer, the consultant discards it and flags the source or the answer. The assistant's output never changes who can see what.

Compare against a simpler alternative

Much of the value may come from work involving no generation: a consistently tagged engagement index, a current expertise directory, and a short list of owned method documents. Compare current search, the improved index and directory, and the improved index plus a generated answer package on the same questions. If the second captures most of the saved effort, the generated layer may not justify its review burden and confidentiality exposure.

Walk through one request

For your knowledge owner

The fictional firm in this exercise is Tallowbrook Advisory, a mid-sized operations consultancy. Its clients, staff, and documents below exist only for this example. Assume the firm's identity system has authenticated the requester and the conflicts system records which matters each person is walled off from.

Input: consultant's question

Have we done a pricing-model redesign for a regional logistics client? What approach did we use, and who led it? I'm scoping a similar piece of work and want to avoid starting from scratch.

The requester is Dana Okafor, a manager in the transport practice. An ethical wall screens Dana from Pellmore Freight matters because Dana's current engagement is for a competitor of that client. Dana has no other restrictions.

Input: repository snapshot

In this snapshot, the most relevant document is also the one Dana may not see. That is deliberate: a system tuned only for relevance will surface ENG-2410 first.

Expected internal answer package

The assistant should give Dana an answer package, not only a paragraph:

• Answer: Yes. The firm has completed at least one pricing redesign for a regional carrier. The current firm method is MTH-07 v3, which adds a cost-to-serve modeling step to the earlier diagnostic.
• Sources: CS-44 (approved case summary); ENG-2231 (method sections only; client data stays inside the engagement file); MTH-07 v3.
• Freshness: MTH-07 v2 was retrieved but is superseded by v3; it is shown only as history.
• People: Priya Nair owns ENG-2231 and CS-44.
• Not established: whether the Quillon approach suits the new client's data and contract structure. That is a delivery judgment.
• Standard footer, shown on every answer: "Some firm material is restricted by client agreements or conflict arrangements and is not searchable for every user. If you think relevant work exists, contact the knowledge team."

Two proposed practices are built into this package. First, nothing derives from ENG-2410: no passage, ranking signal, person, or hint that a result was removed. Even "one result was hidden" could tell Dana the firm advised a competitor on pricing. Second, the footer is unconditional, so its presence reveals nothing about what was filtered.

Expected response to Dana

Yes. The closest permitted example is an approved case summary (CS-44) describing a regional carrier's move to zone-based pricing, and the Quillon Logistics engagement (ENG-2231), whose method sections are available to the transport practice. The current firm method is the pricing diagnostic MTH-07 v3 (June 2025), which adds cost-to-serve modeling; the 2021 version is superseded. Priya Nair is the owner of record for that engagement and the case summary; the file does not say who led delivery.

I haven't assessed whether the approach fits your new client. Some firm material is restricted and not searchable for every user; if you think relevant work exists, contact the knowledge team.

This is an editorially authored expected answer, not model output.

Reject a plausible but prohibited answer

Yes, we're currently redesigning pricing for Pellmore Freight. Tomás Ruiz's team built a lane-level cost-to-serve model and proposed rate increases of 6–9% on regional lanes. Tomás would be the best person to talk to.

Every sentence here may be accurate and useful, and that is the problem. It names a walled-off client, discloses active engagement details, quotes confidential figures (invented here), and routes Dana to a person whose involvement is itself restricted. A reviewer who sees this draft has already been exposed, which is why the control belongs before retrieval rather than at review.

Change one input at a time

After Dana's competitor engagement closes and the conflicts team lifts the wall, Dana is added to the Pellmore engagement. Once the change reaches the index, ENG-2410 becomes retrievable for Dana, still marked client-confidential and internal to that engagement. Measure how long entitlement changes take to reach the index, caches, and stored conversations, and test the reverse: after access is removed, earlier answers containing restricted passages should not remain retrievable through history.

NOTE-9 has no access label. The proposed default is deny: unlabeled material stays out of the index until an owner classifies it, so NOTE-9 never appears. Count it as a curation gap, not permission to guess.

Dana asks, "Summarize everything we have on Pellmore Freight." The response should be identical to the one for a client the firm has never served: no permitted material, the standard footer, and the research-request option. "You don't have access to this client" would confirm the relationship. Even for a permitted client, limit an aggregate summary to the request's purpose and cite each item rather than compile a dossier.

Design the evidence and access layer

For your technical lead

Retrieval-augmented generation grounds a model's answer in retrieved content. Microsoft's vendor documentation for its search service lists security and governance among the challenges of the pattern and states that users and agents must only retrieve authorized content. Microsoft RAG overview That is an architectural requirement, not evidence that any product meets it for a given firm.

The same vendor's guidance for a filter-based security pattern, which it offers for solutions that cannot use the product's built-in access-control support, shows why the requirement falls on the implementer. In that pattern, a group or user identifier stored on each document is "just a string" used to include or exclude results; the application must supply the caller's identities and apply the filter to every query. Azure AI Search security filter pattern If the application passes the wrong identity, omits the filter on one code path, or indexes a document without its restriction, the search layer will not notice.

Proposed practices for this workflow:

• Filter before ranking and context assembly. Restricted material must not influence scores, counts, snippets, suggested questions, or people suggestions. Post-generation redaction is too late.
• Separate firm methods from client deliverables. Index approved methods and sanitized summaries firm-wide; engagement files only for their authorized team, after an owner confirms the access class.
• Default-deny unlabeled material. Missing labels are a curation task.
• Derive people suggestions from permitted sources. Staffing history is as sensitive as the engagement.
• Propagate entitlement changes. Wall changes, departures, and deletions must reach the index, embeddings, caches, and stored conversations within an agreed, tested interval.
• Record provenance. Each passage carries document ID, version, date, owner, and reuse status.

Microsoft's vendor documentation for its workplace assistant states that it only surfaces organizational data users have at least view permission for, and directs customers to use the platform's permission models. Microsoft, Data, Privacy, and Security for Microsoft Copilot Our inference, not the vendor's claim: permission-respecting retrieval faithfully reproduces existing oversharing. An engagement folder shared firm-wide years ago becomes searchable by everyone, so a permission audit belongs in setup. OWASP's community guidance similarly lists misaligned access controls and cross-context leakage among user classes as retrieval risks, recommending permission-aware stores and logical partitioning. OWASP LLM08:2025

Keep the pilot read-only: no editing documents, changing labels, or requesting access on anyone's behalf.

Choose models and configurations against this task

For your technical lead

Permission enforcement is not a model property; no model choice fixes it. The model interprets the question, summarizes faithfully with passage-level citations, marks superseded material, and says "not found" when retrieval returns nothing useful. Retrieval quality may matter more: whether a question about "pricing redesign" reaches a document about "tariff restructuring."

Test complete configurations (retrieval method, filters, chunking, model identifier, prompt revision, document snapshot, interface) on required-source retrieval, citation support, correct abstention, freshness handling, consultant effort, latency, and cost per adequately answered request. Use the model-selection guide and the retrieval chapter, and check each candidate's current specifications at the provider's own documentation on the day you test. No model or configuration has been benchmarked on this fictional task.

Work through the economics

For your finance lead

The following monthly example uses invented assumptions. A request is a knowledge question tracked from asking to verified use or abandonment. Assisted effort includes opening cited passages, discarding unsupported ones, and follow-up.

This yields 360 assisted requests: 480 × 100% × 75%. At ten minutes released each, capacity is 60 hours. At $80/hour, that is $4,800 of capacity value, or $2,300 after the recurring budget, before setup.

An illustrative 20-minute baseline might include eight minutes of searching, six of opening and skimming candidates, three of asking colleagues, and three of checking applicability and whether the material may be reused. A 10-minute assisted process might include one minute to ask, five to review the package and open cited passages, three for the same applicability and reuse check, and one to follow up with a named colleague. The applicability check does not shrink; the assistant only changes how material is found.

Make continuing work visible

Split the hypothetical $2,500 monthly budget into $900 for software and usage, $900 for curation (labeling, retiring superseded methods, approving case summaries), $400 for periodic access review against the conflicts system, and $300 for quality sampling. The split is invented, not a quote. Upkeep absorbed by existing staff is consumed capacity, not cash, and review already inside the assisted minutes is not deducted again; the economics chapter explains both.

Stress-test effort and adoption

The recurring budget is held fixed for clarity. At base volume and hourly value, covering it requires $2,500 ÷ $80 = 31.25 hours, or about 5.21 minutes released per assisted request. That excludes setup and is a capacity threshold, not cash break-even.

Explain how capacity becomes money

Released consultant time becomes cash only through specific mechanisms: avoided contractor or overtime spending, hiring that no longer happens, or more paid client work. Suppose a share of the $4,800 capacity value replaces otherwise-paid work and the whole recurring budget is incremental cash.

At 30% realization, the illustrative net cash effect is $4,800 × 30% − $2,500 = −$1,060/month. At 70%, it is $4,800 × 70% − $2,500 = $860/month, and a $30,000 setup outlay would take about 34.9 months of steady-state performance to recover. Cash break-even on the recurring budget requires about 52.1% realization ($2,500 ÷ $4,800). These figures show why a knowledge tool can look attractive on capacity while struggling on cash.

Billing arrangements complicate this. If search time on a time-and-materials engagement was billed, faster search can reduce billed hours unless the time is redeployed to other paid work. Treat any revenue effect as a separate hypothesis, never added to the capacity value of the same hours. The economics chapter separates benefit location from accounting classification; finance should classify costs under the applicable framework and company policy, not infer a label from the use of AI.

Define failure severity before the trial

For whoever runs the pilot

These are proposed categories for local agreement before any results are seen.

A critical failure caught by the consultant is still critical. The consultant has seen the material, and the wall has been breached whether or not anything reaches a client.

Build the evaluation set

For whoever runs the pilot

As an illustrative coverage plan, not a statistically justified sample, use 150 questions: 60 answerable from permitted material, 25 where the relevant material is stale or superseded, 25 permission-boundary cases (walled matters, other practices, unlabeled documents), 20 with no permitted answer, and 20 adversarial or aggregation requests such as dossier-style summaries and attempts to infer restricted work by triangulation. Run the boundary cases with test accounts that mirror real entitlement patterns, not administrator accounts.

Write questions from real consulting needs but construct permission scenarios synthetically where possible, so evaluators do not handle restricted material. Reserve a held-out set before tuning. Replay historical questions against the repository as it stood then; a later case summary must not answer an earlier question.

Scorecard and thresholds

For each case, record: case ID and category; requester entitlement profile; permitted source IDs; configuration; retrieved IDs; whether required sources were retrieved; citation support per material statement; freshness handling; boundary compliance; severity; consultant minutes; latency; cost; and final disposition.

Report counts with denominators: boundary violations among boundary tests, identical-response failures among existence tests, required-source retrieval among answerable cases, and major errors among all answers. Keep challenge cases separate from a sample reflecting ordinary traffic.

An illustrative offline gate: zero observed boundary violations across all permission tests, zero existence leaks, required sources retrieved in at least 90% of answerable cases, and at least 95% of answers free of major errors. The risk owner must decide what test volume and residual risk are acceptable given the firm's obligations. A clean result on a small set bounds the true failure rate only loosely; pilot design shows how loosely.

Run a supervised pilot and make the decision

Start with offline replay using test accounts. Then pilot within one practice group whose engagement files are fully labeled, with the conflicts system integrated and manual search unchanged. Compare current search, improved index, and assisted retrieval across comparable consultants or assigned periods, balancing seniority and engagement type. Count all requests, including abandoned ones.

Owners: the knowledge manager owns curation and freshness; the conflicts or risk owner approves the access design and handles suspected breaches; engineering owns filtering, propagation, and logging; the practice lead decides expansion; finance reviews realization assumptions.

• Access classes, wall integration, and default-deny rules were agreed before testing.
• Boundary tests ran with realistic test accounts, including wall changes in both directions.
• Existence tests confirmed identical responses for walled and unknown clients.
• Effort includes verification, discarded answers, and follow-up.
• The improved-index configuration was measured, not assumed.
• Curation and access review have owners and funding.
• Any cash claim names the specific avoided spending or paid work.
• Unresolved incidents and reasons to stop appear in the decision memo.

Stop on any confirmed boundary violation until the cause is understood. Expand only to practice groups whose material meets the same labeling standard.

People and skills

Consultants still need source criticism: reading a cited passage, noticing that a method depended on a client's unusual data, and recognizing superseded guidance. The assistant reduces discovery effort, not the judgment needed to adapt a method. Knowledge managers shift toward labeling, retiring material, and approving sanitized case summaries. Staff need to understand that answers can be incomplete by design, so an empty result is not proof the firm lacks experience.

Who you need

This pilot needs the roles below, named by responsibility rather than job title, with a proposed source for each. Access labels, wall rules, curation, evaluation, and the expansion decision continue after go-live, so their owners stay in-house.

For the method behind this table, see Staffing an AI pilot; for bringing in outside help, see Choosing and working with an AI partner.

When to defer

Defer if engagement files lack reliable access labels, the conflicts system cannot be integrated, or reuse rights are unclear. Defer if a maintained engagement index and expertise directory already answer most questions, or if released time will not convert into avoided spending or paid work and the recurring cost is material. The simpler investment, a curated method library with owners and a labeled engagement register, is a prerequisite for the assisted version anyway.

What external research establishes

For your analyst

The vendor documentation and OWASP guidance cited in the design section describe architecture and documented product behavior. None of it measures the effect of assisted retrieval on consulting productivity or confidentiality incidents.

One professional-duty framework illustrates the confidentiality concern. For US lawyers only, ABA Formal Opinion 512 (July 29, 2024) addresses generative AI under the Model Rules. It warns that self-learning tools into which lawyers input information relating to a representation raise the risk that information about one client's representation may be disclosed improperly, "even if the tool is used exclusively by lawyers at the same firm," and that "merely adding general, boiler-plate provisions to engagement letters purporting to authorize the lawyer to use GAI is not sufficient" for informed consent. ABA Formal Opinion 512 This is not a rule for consultancies. The design above is not a self-learning tool, but it addresses the same within-firm disclosure concern.

Evidence still needed and related reading

Obtain attributable accounts from knowledge managers and conflicts owners of how access is labeled and how walls are enforced today. Measure actual question volume, search and verification time, the share of engagement files with reliable labels, and entitlement-propagation delay. No operator has been contacted, no firm's repository examined, and no configuration tested for this chapter.

Continue with professional services, proposals, meeting follow-up, retrieval and source-grounded answers, human review, and the economics worksheet. When comparing products, use build, buy, or configure and vendor diligence.

Companion templates

Prepare your pilot with the pilot kit, evaluation scorecard, and evidence checklist. The companion workbook provides blank economics inputs and case records; worked chapter figures remain illustrative examples.