RAG: AI over your own documents

How tools answer from your files instead of from memory. About 40 min, advanced level.

You'll be able to

• Explain how retrieval-augmented generation works: chunking, embeddings, retrieval and grounded generation.
• Diagnose whether a bad answer came from retrieval or from generation.
• Stress-test a grounded tool with answerable and unanswerable questions, and check its citations.

Answering from your documents

A language model on its own answers from patterns learned in training. That knowledge stops at a cutoff date and never included your lecture notes, your lab's protocols or your office's policies.

Retrieval-augmented generation (RAG) fixes this in two steps. First, a search step finds the passages in your documents most relevant to the question. Then those passages are placed in the prompt, with an instruction like “answer using only these sources, and cite them”. It's the difference between a closed-book and an open-book exam.

NotebookLM and most “chat with your PDF” tools work this way, as do many campus and company help bots.

Why not paste every document into the prompt? For a few short files, that can work. But models have a limit on how much text they take at once, long inputs cost more and are slower, and relevant details get diluted in a sea of irrelevant ones. Retrieval sends only what matters.

Chunks and embeddings

Before anything can be retrieved, documents are prepared once:

• Chunking. Each document is split into pieces: paragraphs, sections, or fixed-size blocks, often overlapping slightly so a sentence isn't cut off from its context. Adding the document title or section heading to each chunk helps it make sense on its own.
• Embedding. Each chunk is turned into an embedding: a long list of numbers produced by a model so that texts with similar meaning get similar numbers. “Car” and “automobile” end up close together even though they share no letters.
• Indexing. The embeddings are stored in a vector index or database, which can quickly find the chunks closest to a new query.

This enables semantic search: matching by meaning rather than exact words. Many systems combine it with ordinary keyword search, called hybrid search, because exact terms like course codes, names and acronyms are sometimes better matched literally.

Retrieval decides the answer

When you ask a question, it's embedded the same way, and the system fetches the few chunks whose embeddings are closest, often measured by cosine similarity. Some systems then re-rank those results with a second model before passing the best ones to the language model.

The model can only work with what retrieval hands it. Bad retrieval means bad answers, however good the model is. Typical retrieval failures:

• The right passage is split across two chunks, and neither makes sense alone.
• Vocabulary mismatch: your question says “leave policy”, the handbook says “time off”, and the right passage ranks low.
• Near-miss chunks: similar-sounding but wrong passages, such as last year's version of a policy.
• Nothing relevant exists, but the system still returns the closest chunks, because nearest isn't the same as relevant.

When an answer is wrong, look at what was retrieved first. That tells you whether to fix the search or the prompt.

Grounding and citations

Grounding means the answer is built from the retrieved passages rather than the model's general knowledge. A good grounded prompt says three things: answer only from these sources, cite the passage for each claim, and say so if the sources don't contain the answer.

Citations are what make grounded answers checkable. In NotebookLM you can click a citation to jump to the exact passage. Use that every time it matters.

Grounding reduces made-up facts; it doesn't eliminate errors. A grounded answer can still:

• misread a passage or drop its caveats
• blend two chunks into a claim neither makes
• drift into general knowledge when the sources run thin
• faithfully repeat a source that is itself wrong or out of date

So the habit is the same as in research: open the cited passage and confirm it says what the answer claims.

Stress-test before you trust

Whether you're using NotebookLM or building your own assistant, test it with questions you already know the answers to.

• Answerable: the sources clearly answer it. Is the answer right, and does the citation support it?
• Unanswerable: the sources don't cover it. Does it say so, or does it fill the gap from general knowledge?
• Partly answerable: does it separate what the sources say from what they don't?
• Conflicting: two sources disagree. Does it notice, or pick one silently?
• Wording changes: ask the same question with different vocabulary. Does retrieval still find the passage?

For builders, write these down as a small test set with expected answers, and record which retrieved chunks each answer used. Checking retrieval and generation separately shows you what to fix.

And remember the documents themselves: only upload material you're allowed to share with the service.

Stress-testing a notebook of three readings

Setup: a student loads three course readings on urban heat into NotebookLM.

Test 1, unanswerable: “What does the city's 2030 climate plan commit to?” None of the readings mention the plan. A well-grounded answer says the sources don't cover it. If the tool answers anyway, with no citations or with citations that don't mention the plan, it's filling the gap from general knowledge, and you note that.

Test 2, answerable: “Why are some neighborhoods hotter than others?” The answer lists tree cover, paved surfaces and building density, with citations. You click each one. Two passages say exactly that. The third mentions building density only as a possible factor the authors didn't measure, while the answer states it as a finding.

Verdict: retrieval found the right passages; generation overstated one of them. You record it as partly supported.

Test 3, rephrased: you ask about the “urban heat island effect” instead of “hotter neighborhoods”, and check that the same passages come back.

Try it

A short test log for a notebook of three sources: an unanswerable question and how the tool handled it, an answerable question with each citation checked against its passage, and your verdict on how far you'd trust it.