The short answer (August 2026): A RAG pipeline retrieves relevant chunks from your own documents and injects them into an LLM prompt before generation, so the model answers from your data instead of training memory. You need four components: a document loader, a text splitter, an embedding model, and a vector store. The fastest starting point is pip install langchain langchain-openai langchain-chroma chromadb, an OpenAI API key, and about an afternoon. OpenAI's text-embedding-3-small costs $0.02 per million tokens on their standard API, so a 50,000-chunk corpus typically costs under $0.50 to index from scratch.
Last verified: August 17, 2026, against official docs and pricing pages.
How a RAG Pipeline Works
RAG stands for retrieval-augmented generation. The mechanics: you convert your documents into vector embeddings ahead of time, store them in a vector database, and at query time you embed the user's question with the same model and pull back the most semantically similar chunks. Those chunks go into the LLM prompt alongside the question.
Four discrete stages run every time:
Indexing (run once): load documents, split them into chunks, embed each chunk, write vectors to the database.
Retrieval: embed the incoming query, run a nearest-neighbor search, return the top-k chunks.
Augmentation: drop those chunks into a prompt template next to the user's question.
Generation: send the filled prompt to the LLM; it answers using the retrieved context as grounding.
The key property RAG gives you: the LLM can only cite what retrieval surfaced. Hallucinations are not eliminated, but they are substantially reduced because the model has concrete text to reference. If your retrieval step misses the relevant chunk, the model has nothing to work with, so retrieval quality is the main lever you optimize.
Choose Your Stack
Two Python frameworks handle the majority of production RAG workloads: LangChain (v1.3.15 as of August 2026) and LlamaIndex (llama-index-core v0.14.23 as of June 2026). They support the same vector stores and embedding models, so switching is not catastrophic, but the programming models differ.
LangChain's composition model, LCEL (LangChain Expression Language), connects steps with the | pipe operator. It gives you explicit control over each stage, which matters when you want to add reranking, routing, or agentic loops. If you plan to build AI agents alongside your RAG pipeline, LangChain integrates that work more naturally.
LlamaIndex is purpose-built for data-intensive document Q&A. Its VectorStoreIndex.from_documents() call handles loading, chunking, embedding, and indexing in one line. Fewer moving parts to manage, but also fewer hooks to customize each step. It defaults to OpenAI's text-embedding-ada-002 for embeddings; you need to override that explicitly to use a newer model.
For the vector store, your choice determines both your operational burden and your cost at scale:
| Vector Store | Free Tier | Paid Starting Price | Self-Host Option | Best Fit |
|---|---|---|---|---|
| Chroma | Free (open source, runs in-process) | Chroma Cloud: $5 credits, then usage-based | Yes (Apache 2.0) | Local dev; in-process prototypes |
| Qdrant | Free forever: 0.5 vCPU, 1 GB RAM, 4 GB disk | Standard: hourly usage-based billing | Yes (Apache 2.0) | Production self-hosting; filtered search |
| Pinecone | Starter: 2 GB storage, 5 serverless indexes | Builder: $20/month flat rate | No (managed cloud only) | Zero-ops managed deployments |
| pgvector | Free (PostgreSQL extension) | Your existing Postgres hosting costs | Yes | Teams already running Postgres |
Chroma is the right default for local development. Qdrant's free cloud tier is generously provisioned for a prototype that needs to persist across restarts. Move to Pinecone or a production Qdrant cluster when you need uptime SLAs.
Install and Configure Your Environment
LangChain with Chroma:
pip install langchain langchain-openai langchain-text-splitters langchain-chroma chromadb
Set your API key:
export OPENAI_API_KEY="sk-..."
LlamaIndex with the default in-memory store:
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai
LlamaIndex 0.10 and later uses a modular package structure. Install only the integrations you need. For Qdrant as your vector store, add llama-index-vector-stores-qdrant. For Pinecone, add llama-index-vector-stores-pinecone. Both frameworks read OPENAI_API_KEY from your environment automatically.
Put your documents in a folder called data/ before moving to the next step. Both frameworks handle PDFs, plain text, Markdown, and HTML out of the box.
Chunk Your Documents
Text splitters divide documents into retrievable pieces. Chunk size has as much influence on retrieval quality as embedding model choice, so set it deliberately.
The current consensus starting point is 512 tokens per chunk with 10 percent overlap (roughly 50 tokens for a 512-token chunk). Smaller chunks give more precise retrieval for factoid questions. Larger chunks give the model more surrounding context per retrieved piece, which helps when answers require multi-paragraph reasoning.
Recursive character splitting is the most reliable default. It attempts to break on paragraph boundaries first, then sentences, then words, preserving semantic units where possible.
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader
loader = DirectoryLoader("data/")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
)
chunks = splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")
One warning: overlap beyond a certain threshold adds storage and embedding cost without measurable retrieval improvement in most benchmarks. If your recall is low, try smaller chunks or a better embedding model before raising overlap.
Embed and Index
Embedding converts each chunk into a vector of floating-point numbers that encode semantic meaning. You run this step once at index time; the identical model runs again at query time on the user's question.
OpenAI's current embedding models, verified against their official pricing page:
- text-embedding-3-small: $0.02 per million tokens, 1,536 dimensions, 8,192-token input limit
- text-embedding-3-large: $0.13 per million tokens, 3,072 dimensions, 8,192-token input limit
text-embedding-3-small is the correct default for most RAG workloads. The large model scores 64.6% on MTEB benchmarks versus 62.3% for small, a modest quality gain at a 6.5x price premium. Start with small; benchmark both against your actual retrieval queries before paying for large.
Indexing with LangChain and Chroma:
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="./chroma_db",
)
print(f"Indexed {vectorstore._collection.count()} vectors")
With LlamaIndex, override the default embedding model before building the index:
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.openai import OpenAIEmbedding
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
Without the Settings.embed_model override, LlamaIndex defaults to text-embedding-ada-002 ($0.10/M tokens), which is older and outperformed by text-embedding-3-small at a fifth of the cost.
Query: Retrieve and Generate
At query time the pipeline embeds the question, retrieves the top-k chunks, and passes them to the LLM. The k parameter is the single most impactful tuning knob at this stage.
LangChain LCEL pattern:
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
prompt = ChatPromptTemplate.from_template(
"Answer using only the context below. "
"If the context does not contain the answer, say so.\n\n"
"Context: {context}\n\nQuestion: {question}"
)
llm = ChatOpenAI(model="gpt-4o-mini")
chain = (
{"context": retriever, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
answer = chain.invoke("What does the refund policy cover?")
print(answer)
LlamaIndex pattern:
query_engine = index.as_query_engine(similarity_top_k=4)
response = query_engine.query("What does the refund policy cover?")
print(response)
Start with k=4. Increase to 6 or 8 if the model frequently says it lacks context. Each additional retrieved chunk consumes context-window tokens and raises inference cost, so there is a real tradeoff to manage. The prompt instruction "if the context does not contain the answer, say so" materially reduces confident hallucinations in production.
FAQ
Does the embedding model matter more than chunking strategy?
A peer-reviewed NAACL 2025 study (Vectara) found that chunking decisions had as much influence on retrieval quality as embedding model choice. Set your chunk size deliberately before upgrading to a more expensive model.
Can I run this entirely locally with no external API calls?
Yes. Replace OpenAI with a local model via Ollama for LLM inference and install llama-index-embeddings-huggingface or a similar package for local embeddings. Chroma runs fully in-process with no network calls. Expect lower throughput and typically lower answer quality compared to hosted models.
What happens when a document exceeds the embedding model's 8,192-token limit?
The document must be chunked before embedding. Both LangChain's text splitters and LlamaIndex's node parsers handle this automatically. Keep your chunk size comfortably below 8,000 tokens; the model tokenizer uses a fixed vocabulary and your character-count estimates may be slightly off.
Is pgvector good enough to replace a dedicated vector store?
For smaller corpora with an HNSW index, pgvector performs comparably to dedicated vector stores and eliminates one service to operate. At larger scales or when you need advanced filtered search performance, Qdrant and Pinecone offer more tuning options.
How do I update the index when my documents change?
Most vector stores support upsert operations keyed by a document ID. Assign stable IDs (a hash of the file path plus chunk position works), then delete and re-upsert only the chunks from the changed document. Pinecone and Qdrant both support this pattern natively. Avoid full re-indexing on every update; at 50,000 chunks that costs $0.50 every time.
More on agentic patterns that extend RAG systems in AI Weekly's agent setup guide. Get AI Weekly free, 3 issues a week, read by 40,000+ practitioners.