Search and Question Your Own Documents With a Private AI (RAG Explained)

Part 9 of the AI series. You will need the setup from part 3. The jargon is explained in part 1.

The most useful thing most people can do with a small AI server is not chat. It is asking questions of their own material: notes, manuals, contracts, saved articles, years of paperwork. The technique is called RAG (retrieval-augmented generation), and the name is much scarier than the idea. This guide explains it in plain words, shows the quick way to get it, then gives a short script you can read, run and change, so you understand what is happening inside the tools.

What is in this guide

• The idea: an open-book exam
• The six steps inside every RAG system
• The easy way: let a tool do it
• The do-it-yourself way: a short script
• Making answers better
• When it goes wrong
• Privacy and safety
• When your library grows
• Which plan do you need?

The idea: an open-book exam

A language model knows a lot in general, and nothing about your files. You could paste a whole document into the chat, but models can only read so much at once, and long inputs are slow on a CPU. RAG does what a sensible student does in an open-book exam: look up the right pages first, then answer using only those. The model never needs to hold your whole library in its head. It reads a few relevant paragraphs and writes the answer.

The six steps inside every RAG system

1. Load. Read your files (text, Markdown, PDFs with OCR where needed).
2. Chunk. Cut them into passages of a few paragraphs, with a little overlap so ideas are not cut in half.
3. Embed. Turn each passage into a list of numbers with an embedding model. Passages with similar meaning get similar numbers.
4. Store. Save the passages and their numbers in an index. For a few thousand passages a file is enough, and for more you use a vector database.
5. Retrieve. When you ask a question, embed it the same way and find the passages with the closest numbers.
6. Answer. Give those passages and your question to a language model, with an instruction to answer only from them and to say so when the answer is not there.

Steps 1 to 4 happen once for each document (indexing). Steps 5 and 6 happen on every question.

The easy way: let a tool do it

Open WebUI and AnythingLLM both do all six steps for you. You upload files in the interface, pick an embedding model and a chat model, and start asking. This is the right place to begin, because you learn how your documents behave before you build anything. Tips for a good start:

• Pick a dedicated embedding model rather than a chat model for the search part. Small embedding models run quickly on a CPU. In Ollama, nomic-embed-text is a popular one.
• Upload ten documents, not ten thousand, and ask questions whose answers you already know.
• Look at the sources the tool shows. If the right passage is not among them, the problem is in the search, not the model.
• Scanned PDFs have no text until you run OCR on them (see the 1-2 GB guide).

The do-it-yourself way: a short script

Seeing it work end to end makes everything else clearer. This script uses only Python's standard library and your Ollama server. It indexes a folder of .txt and .md files, then answers questions with sources. It is meant for a small library of a few thousand passages, such as personal notes.

Prepare Ollama. The script runs on the server, outside Docker, so Ollama needs to listen on the server itself, and only on the server itself. In the ollama service of your compose.yaml from part 3, add:

    ports:
      - "127.0.0.1:11434:11434"

Then restart it and download an embedding model:

docker compose up -d
docker compose exec ollama ollama pull nomic-embed-text

Save this as rag.py:

#!/usr/bin/env python3
"""Tiny private document search. Usage:
  python3 rag.py index ~/notes
  python3 rag.py ask "what did I decide about the garden?"
"""
import json, math, os, sys, urllib.request

OLLAMA = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434")
EMBED_MODEL = os.environ.get("EMBED_MODEL", "nomic-embed-text")
CHAT_MODEL = os.environ.get("CHAT_MODEL", "llama3.2:3b")
INDEX = os.environ.get("RAG_INDEX", "index.json")
CHUNK, OVERLAP, TOP_K = 800, 150, 4


def call(path, payload):
    req = urllib.request.Request(OLLAMA + path, json.dumps(payload).encode(),
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=600) as r:
        return json.load(r)


def embed(texts):
    return call("/api/embed", {"model": EMBED_MODEL, "input": texts})["embeddings"]


def chunks(text):
    step = CHUNK - OVERLAP
    for start in range(0, max(len(text), 1), step):
        piece = text[start:start + CHUNK].strip()
        if piece:
            yield piece


def build(folder):
    items = []
    for root, _, files in os.walk(folder):
        for name in sorted(files):
            if name.lower().endswith((".txt", ".md")):
                path = os.path.join(root, name)
                with open(path, encoding="utf-8", errors="ignore") as f:
                    for piece in chunks(f.read()):
                        items.append({"file": path, "text": piece})
    for i in range(0, len(items), 16):
        batch = items[i:i + 16]
        for item, vec in zip(batch, embed([b["text"] for b in batch])):
            item["vec"] = vec
        print("embedded", min(i + 16, len(items)), "of", len(items))
    with open(INDEX, "w") as f:
        json.dump(items, f)


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb) if na and nb else 0.0


def ask(question):
    with open(INDEX) as f:
        items = json.load(f)
    q = embed([question])[0]
    best = sorted(items, key=lambda it: cosine(q, it["vec"]), reverse=True)[:TOP_K]
    context = "\n\n".join("[%d] (%s)\n%s" % (n + 1, b["file"], b["text"])
                          for n, b in enumerate(best))
    prompt = ("Answer the question using only the passages below. "
              "Cite passage numbers like [1]. If the passages do not contain "
              "the answer, say you do not know.\n\n"
              + context + "\n\nQuestion: " + question)
    out = call("/api/generate", {"model": CHAT_MODEL, "prompt": prompt, "stream": False})
    print(out["response"].strip())
    print("\nSources:")
    for n, b in enumerate(best):
        print(" [%d] %s" % (n + 1, b["file"]))


if __name__ == "__main__":
    if len(sys.argv) == 3 and sys.argv[1] == "index":
        build(sys.argv[2])
    elif len(sys.argv) >= 3 and sys.argv[1] == "ask":
        ask(" ".join(sys.argv[2:]))
    else:
        print(__doc__)

Use it:

python3 rag.py index ~/notes
python3 rag.py ask "what did I decide about the garden?"

Indexing prints its progress and saves index.json in the current folder. Asking finds the four closest passages, builds a prompt around them and prints the model's answer followed by the list of sources. Change the chat model with the CHAT_MODEL environment variable, and the embedding model with EMBED_MODEL. If you change the embedding model, rebuild the index, because numbers from different models cannot be compared.

What to try changing. The three numbers near the top set the passage size, the overlap and how many passages are retrieved. Try larger and smaller values and see how the answers change. That is the best way to learn what matters.

Making answers better

• Chunk by meaning. Splitting at headings or paragraph breaks beats cutting every 800 characters. Keep a passage's title or filename in its text, so the search can see it.
• Retrieve a little more than you need, then show the model only the best few. More passages means more memory and slower answers, and can confuse a small model.
• Tell the model what to do when it does not know. "If the passages do not contain the answer, say you do not know" removes a lot of invented answers.
• Always show sources, and check them for anything that matters.
• Clean your documents. Duplicates, old versions and scanning noise all pollute the search.
• Test with real questions. Write twenty questions you know the answers to, run them after every change, and keep score.
• Use a better embedding model if the search keeps missing the right passage. Of everything you can change, it often helps most.

When it goes wrong

Symptom	Likely cause	Fix
The answer ignores your documents	The right passage was not retrieved	Print the retrieved passages. Adjust chunking or the embedding model
Confident but wrong answers	The model filled a gap from its general knowledge	Strengthen the instruction. Use a bigger model. Check sources
Answers mix up two documents	Passages from different files look alike	Put filenames and dates in the passage text
Questions about the whole document fail ("summarise the report")	Retrieval finds pieces, not the whole	Summarise documents when you index them, or send the whole text to a model with a larger context
Very slow answers	Too many or too long passages, or a model too big for the CPU	Retrieve fewer passages and use a smaller model
Numbers and tables come out garbled	The PDF text extraction lost the layout	Convert tables to plain text or CSV first

Privacy and safety

• With the embedding model and chat model both on your server, nothing leaves it. If you point the answering step at an outside service, the retrieved passages go with the question.
• The index contains your text. Treat index.json like the documents themselves: permissions, backups, and no public access.
• Documents can contain instructions. A file or web page may include text like "ignore previous instructions". Do not index material you do not trust, and never connect a RAG system to tools that can send mail, delete files or spend money without a human check.
• If several people use the same system, remember that anyone can ask about anything in the index. Keep separate indexes for separate audiences.

When your library grows

The script compares your question with every passage, which is fine for a few thousand and slow for hundreds of thousands. At that point move the storage to a vector database such as Qdrant, or to PostgreSQL with the pgvector extension, which find the closest passages quickly and let you filter by file, date or tag. Keep the same six steps. Only the "store" and "retrieve" parts change. Very large libraries also take a long time to embed on a CPU, so plan to index in batches overnight, as described in the dedicated server AI guide.

Which plan do you need?

• 2 GB: search only. Embed documents and show the matching passages, with no answering model.
• 4 GB: search plus a 1-3B answering model. Fine for personal notes.
• 8 GB: a 7-8B answering model, which follows the passages more closely and makes fewer mistakes.
• Dedicated server: big libraries, bigger embedding models and answers from larger models.

Compare more in the cheapest plans guide, and see the sales page for current deals.