How to scope a RAG build without burning the budget

Retrieval-augmented generation looks cheap in a demo and expensive in production. A small team can lose weeks tuning chunk sizes and swapping vector stores before anyone has agreed on what “good” means. The way to avoid that is to scope the build backwards: decide what the system must answer, set the bar for answering it, and only then choose the machinery.

This is an engineering approach we use on fixed-scope AI work, not a promise that retrieval solves every problem.

Start with the questions, not the documents

The first scoping mistake is to index everything and hope the model sorts it out. It rarely does, and a large index makes every later decision slower and more expensive.

Instead, write down the ten to twenty real questions the feature has to handle, in the words a user would actually type. Those questions tell you which documents matter, how fresh they need to be, and whether the answer is a fact to quote or a judgment to synthesize. A support assistant that answers “what is your refund window” needs exact retrieval from one policy page. A research helper that answers “how have our margins trended” needs something closer to summarization over structured data. Those are different builds, and only the questions reveal which one you are signing up for.

Set the eval bar before writing the pipeline

An LLM system fails quietly: it can state a wrong answer with the same confidence as a right one. So the scope has to include how you will know it works, and that belongs in the plan before the retrieval code, not after.

For most small builds this is a labelled set of those same real questions, each paired with the correct answer and the source it should come from. Three checks cover a lot of ground: did the answer come from the right document, did it leave out anything the question required, and did it stay accurate when the question was phrased loosely. Every change to prompts, chunking, or the retriever runs against that set before it reaches staging. Without this, “better” is a feeling, and feelings are what burn the budget.

Choose the smallest machinery that clears the bar

Once the questions and the eval set exist, most of the architecture decisions get smaller and less religious.

Chunking: start with whole sections or documents. Only split further if the eval set shows retrieval pulling in the wrong passage. Aggressive chunking is a common early cost with little payoff.
Vector store: for a few thousand documents, a library that runs in your existing database is usually enough. A dedicated vector service earns its keep at a scale most early features never reach.
Reranking and query rewriting: add them only when the eval numbers say retrieval quality, not generation, is the limiting factor. They are easy to add later and hard to justify up front.

The point is not to under-build. It is to let the evaluation set, rather than a blog post or a vendor, decide what the project needs.

Decide what happens when retrieval fails

A production RAG feature spends real time retrieving nothing useful. Scope has to cover that case explicitly, because the default behaviour of a model handed weak context is to invent a confident answer.

The honest options are to say “I do not have that information,” to route the question to a human, or to fall back to a narrower non-retrieval response. Choosing one is a product decision with a cost, and putting it in scope early keeps it from becoming an incident later.

What this buys a small team

Scoping a retrieval build this way trades some flexibility for predictability. You commit to a set of questions and a bar up front, which means saying no to features the eval set does not cover. In return you get a project that can be estimated, a definition of done that a client can verify, and a system whose failures are visible instead of silent.

It does not guarantee the model will be right every time. It makes the promise concrete enough to check, which is the part small teams can actually deliver.

FAQ

What is retrieval-augmented generation, in one line?

RAG is when a language model answers using documents you supply at query time — retrieved from a vector store, database, or search index — instead of relying only on what the model already learned during training.

Do I need a dedicated vector database for a RAG feature?

Usually not at first. For a few thousand documents, a vector extension inside your existing database (for example pgvector in Postgres) is enough. A dedicated service like Pinecone or Weaviate earns its keep at a scale most early features never reach.

How do I know if my RAG system is actually working?

Build a labelled evaluation set of real user questions paired with the correct answer and source, and score three things on every change: whether the answer came from the right document, whether it omitted anything required, and whether it stayed accurate when the question was phrased loosely.

What should a RAG system do when retrieval finds nothing useful?

Decide this in scope, not in an incident. The honest options are to say “I do not have that information,” route the question to a human, or fall back to a narrower non-retrieval answer — anything but letting the model invent a confident response from weak context.

If you are weighing a retrieval feature and want a second opinion on scope before you commit budget, that conversation is worth having early.