How RAG came to be

Larkspur Supply Co. sells office and warehouse hardware to about 4,000 businesses. Its 260 staff lose hours hunting through policies, contracts, price lists and meeting notes. The company wants an AI assistant that answers their questions from those documents, and you are on the team building it.

This chapter: Why can’t an AI model simply answer questions about your company? And how did the industry arrive at RAG between 2020 and 2026?

You will learn

Why a language model invents confident answers about things it never read
What Retrieval-Augmented Generation (RAG) is, in one sentence
Why RAG was essential in 2023, and why today it is one tool an AI agent can choose to use

Words you’ll meet (5)

LLM: Large language model. It writes text by predicting what comes next, using what it learned in training and what is in its prompt.
Training data: The text a model learned from. Your company’s private documents were never part of it.
Context window: How much text a model can read in one request, counted in tokens (a token is about ¾ of a word).
RAG: Retrieval-Augmented Generation: look up the relevant passages first, then have the model answer from them.
Tool calling: A model asking the software around it to run something, such as a search, and reading the result.

The picture on the left is live. Each step below asks you to do one thing in it; the story continues here as you go.

Step 1 of 6

It’s 2023. An employee asks the company’s new AI assistant a question about a customer contract.