What is RAG? (how your AI looks things up before answering)

What it actually is

RAG stands for Retrieval-Augmented Generation. It means the AI searches for relevant information first, then writes its answer using what it found, instead of answering purely from what it memorized during training.

Think of the difference between a closed-book exam and an open-book one. A model answering from memory alone is doing the closed-book version: working off patterns learned before a cutoff date, with no idea what’s in your product docs or the PDF you uploaded yesterday. RAG turns it into an open-book exam. Before the model writes a word, it pulls the specific pages that relate to the question, then answers using those pages open in front of it.

Under the hood: your documents get split into chunks, each chunk gets converted into a vector embedding (numbers capturing its meaning), and those embeddings get stored somewhere searchable, often Postgres with the pgvector extension, since Supabase supports that directly. When a user asks a question, that question gets embedded too, the system finds the stored chunks closest in meaning, and only those few chunks get inserted into the prompt the model actually sees.

Why your AI just did this

Your agent reaches for RAG when you’ve asked for something that answers using your own material, not the model’s general knowledge. A support bot that knows your actual FAQ, a chatbot over your product docs, search over your uploaded files: none of that lives in the model’s training data, so it has to be fetched at answer-time.

Pasting your whole knowledge base into every prompt is the naive alternative, and it breaks down fast: you’ll hit the context window limit once your content grows past a few pages, and pay in tokens for content unrelated to the question. Retrieving just the relevant chunks keeps the prompt small, keeps costs down, and gives the model less room to wander into a plausible-sounding guess (see AI hallucination) instead of your real content.

When you’ll run into it

• Your agent says something like “I’ll store your docs as embeddings in a vector column and search that on each question.”
• Building a chatbot, support widget, or internal search tool meant to answer from your own files, not general knowledge.
• The bot answers “I don’t have that information” for something that is in your docs, usually meaning the document was never actually ingested and embedded, or the vector table is empty.
• Answers feel randomly wrong or off-topic even though the source material is correct, often meaning the embedding model used to store your docs doesn’t match the one used to embed the live question.
• Costs or latency creep up because the app is re-embedding your entire knowledge base on every request instead of once, at ingestion time.

What to check

• Confirm your documents were actually chunked and embedded, not just wired up in code. Query the vector table and check the row count is what you expect
• Confirm the same embedding model embeds your stored documents and the live user question. A mismatch quietly returns irrelevant chunks with no error
• Ask what happens when nothing relevant is found: does it say so honestly, or does the model fill the gap with a guess anyway
• Check the chunk size. Chunks too small lose surrounding context, chunks too large drag in irrelevant text and cost more tokens per query
• If a document changes, check whether re-ingestion is automatic or something you have to trigger by hand

Say it like a dev

Instead of: “make the AI know my docs” Say: “set up RAG: chunk and embed my docs into a vector store, then retrieve the relevant ones per question”

Instead of: “why does it still make things up when the answer is right there in my files” Say: “is retrieval actually finding the relevant chunks, or is it falling back to guessing when nothing matches?”

Instead of: “just paste all my files into the prompt so it knows everything” Say: “that’ll blow past the context window, let’s retrieve just the relevant chunks instead”