SurfingBearHome › Blog › Internal document AI (RAG)
A practical guide to rolling out internal document search AI (RAG)
RAG means "read the company documents and answer from them". Because the state of the documents matters more than the model, getting the order wrong means even a strong model returns nothing useful.
5 steps Permissions Failure points
The short answer
Retrieval-augmented generation in one line: a question comes in, the relevant passages are found in your documents, and the answer is built only from those passages. Because the model is not answering from what it "knows", you can show a source, and you can cover things that exist nowhere outside your company.
That same structure fixes its weaknesses. If retrieval misses, there is no answer; if a document is stale, the stale answer arrives with full confidence. Which is why most of a RAG rollout is document work, not model work.
Five steps
Common failures and what actually causes them
| Symptom | Real cause | Where to look first |
|---|---|---|
| Answers look right but are wrong | A stale document is still indexed | Versioning and retirement |
| It only ever says "I do not know" | Question wording differs from document vocabulary | Coverage against the question list |
| Only one team gets good answers | Only that team’s documents are in order | Scope of the document clean-up |
| Content people should not see appears | Permissions handled at answer time | Retrieval-stage permission filter |
| The same question gets different answers | Duplicate documents disagree with each other | De-duplication |
Build it, or buy a tool?
The deciding factors are where the documents live and how complex the permissions are. If they sit in one place — a company wiki, say — and permissions are simple, starting with the search AI already attached to that tool is faster. A separate build usually just adds cost.
If instead the documents are spread across file servers, mail and groupware, or different teams can see different subsets, then integration and permissions become most of the work. Design comes before tool choice in that case.
Either way the first step is identical: test small against the step-1 question list, record the accuracy, then widen. Make company-wide rollout the first goal and scale grows without verification.
Frequently asked questions
How is RAG different from fine-tuning a model?
RAG looks the documents up each time and uses them as the basis for the answer. Edit a document and the answer changes immediately, and you can cite it. Fine-tuning adjusts the model itself, so every content change means retraining, and citation is hard. For things that change often, like internal policy, RAG is the right fit.
Our documents are a mess. Can we still start?
You do not need to fix everything. Organising the documents that answer 30–50 questions is enough to start. Making a full clean-up a precondition is how these projects never begin.
Who is accountable when an answer is wrong?
Citations are the mechanism. Showing the source passage lets the user check it, and separates a document problem from a retrieval problem. Where consequences are heavy — HR, legal, safety — it is safer to treat answers as reference only and require a human to confirm.
Does our data leave the building?
It depends on the architecture. Passages from your documents are included in the model request, so you need the contract to state where that request goes, whether it is stored, and whether it is used for training. Until that is settled, keep sensitive documents out of the index.
Start with document readiness
See how far along you already are
The AX readiness assessment scores your current state across work, data, technology and organisation. The data axis is a direct read on how hard a RAG rollout will be.