RAG — Retrieval Augmented Generation

RAG grounds LLM responses in your actual data rather than the model's training knowledge. Instead of fine-tuning, you retrieve relevant documents at query time and inject them into the prompt.

Why RAG?

Problem	RAG Solution
LLM doesn't know your private docs	Retrieve and inject them
Knowledge cutoff (stale data)	Retrieve from live database
Hallucinations	Ground answers in retrieved text
Can't fit all docs in context	Retrieve only the relevant ones

RAG Architecture

Offline (index your documents):
  Documents → Chunker → Embeddings → Vector Store (Pinecone/Chroma)

Online (answer a question):
  Query → Embed query → Vector search → Top-K chunks
  → Inject into prompt → LLM → Grounded answer

Implementation with Spring AI

// Step 1: Ingest documents
@Service
@RequiredArgsConstructor
public class DocumentIngestionService {

    private final VectorStore vectorStore;
    private final TokenTextSplitter splitter = new TokenTextSplitter();

    public void ingest(String text, Map<String, Object> metadata) {
        Document doc = new Document(text, metadata);
        List<Document> chunks = splitter.apply(List.of(doc)); // ~512 token chunks
        vectorStore.add(chunks); // embeds + stores in Chroma/Pinecone
    }
}

// Step 2: Query with retrieval
@Service
@RequiredArgsConstructor
public class RagService {

    private final VectorStore vectorStore;
    private final ChatClient chatClient;

    public String answer(String question) {
        // Retrieve top-3 relevant chunks
        List<Document> relevant = vectorStore.similaritySearch(
            SearchRequest.query(question).withTopK(3)
        );

        String context = relevant.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n\n---\n\n"));

        String prompt = """
            Answer the question based ONLY on the context below.
            If the answer isn't in the context, say "I don't know."

            Context:
            %s

            Question: %s
            """.formatted(context, question);

        return chatClient.prompt()
            .user(prompt)
            .call()
            .content();
    }
}

Chunking Strategy

// Fixed-size chunks (simple)
TokenTextSplitter splitter = new TokenTextSplitter(512, 50); // 512 tokens, 50 overlap

// Semantic chunking (better):
// Split at paragraph/section boundaries; preserve semantic units
// Spring AI RecursiveCharacterTextSplitter respects natural boundaries

Embedding Models

OpenAI text-embedding-3-small:  1536 dimensions, ~$0.02/1M tokens
OpenAI text-embedding-3-large:  3072 dimensions, better quality
BGE-M3 (open source):           1024 dimensions, multilingual

Vector Stores

Store	Scale	Best For
Chroma	Dev/small	Local development
Pinecone	Large	Production, managed
Weaviate	Large	Hybrid search
pgvector	Medium	Already using PostgreSQL
Qdrant	Large	High-performance open source

Advanced: Hybrid Search

// Combine semantic search (embeddings) + keyword search (BM25)
// Semantic: understands meaning
// BM25: catches exact terms, technical names

List<Document> semantic = vectorStore.similaritySearch(question);
List<Document> keyword = bm25Index.search(question);
List<Document> combined = rerank(semantic, keyword, question); // RRF or cross-encoder

Interview Tips

RAG is the standard approach for company knowledge bases, documentation Q&A, and product support.
Chunking strategy significantly impacts quality — overlapping chunks prevent boundary cutoffs.
Re-ranking: after initial retrieval, use a cross-encoder model to re-rank results by relevance before injecting into prompt.

RAG — Retrieval Augmented Generation

RAG grounds LLM responses in your actual data rather than the model's training knowledge. Instead of fine-tuning, you retrieve relevant documents at query time and inject them into the prompt.

Why RAG?

Problem	RAG Solution
LLM doesn't know your private docs	Retrieve and inject them
Knowledge cutoff (stale data)	Retrieve from live database
Hallucinations	Ground answers in retrieved text
Can't fit all docs in context	Retrieve only the relevant ones

RAG Architecture

Offline (index your documents):
  Documents → Chunker → Embeddings → Vector Store (Pinecone/Chroma)

Online (answer a question):
  Query → Embed query → Vector search → Top-K chunks
  → Inject into prompt → LLM → Grounded answer

Implementation with Spring AI

// Step 1: Ingest documents
@Service
@RequiredArgsConstructor
public class DocumentIngestionService {

    private final VectorStore vectorStore;
    private final TokenTextSplitter splitter = new TokenTextSplitter();

    public void ingest(String text, Map<String, Object> metadata) {
        Document doc = new Document(text, metadata);
        List<Document> chunks = splitter.apply(List.of(doc)); // ~512 token chunks
        vectorStore.add(chunks); // embeds + stores in Chroma/Pinecone
    }
}

// Step 2: Query with retrieval
@Service
@RequiredArgsConstructor
public class RagService {

    private final VectorStore vectorStore;
    private final ChatClient chatClient;

    public String answer(String question) {
        // Retrieve top-3 relevant chunks
        List<Document> relevant = vectorStore.similaritySearch(
            SearchRequest.query(question).withTopK(3)
        );

        String context = relevant.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n\n---\n\n"));

        String prompt = """
            Answer the question based ONLY on the context below.
            If the answer isn't in the context, say "I don't know."

            Context:
            %s

            Question: %s
            """.formatted(context, question);

        return chatClient.prompt()
            .user(prompt)
            .call()
            .content();
    }
}

Chunking Strategy

// Fixed-size chunks (simple)
TokenTextSplitter splitter = new TokenTextSplitter(512, 50); // 512 tokens, 50 overlap

// Semantic chunking (better):
// Split at paragraph/section boundaries; preserve semantic units
// Spring AI RecursiveCharacterTextSplitter respects natural boundaries

Embedding Models

OpenAI text-embedding-3-small:  1536 dimensions, ~$0.02/1M tokens
OpenAI text-embedding-3-large:  3072 dimensions, better quality
BGE-M3 (open source):           1024 dimensions, multilingual

Vector Stores

Store	Scale	Best For
Chroma	Dev/small	Local development
Pinecone	Large	Production, managed
Weaviate	Large	Hybrid search
pgvector	Medium	Already using PostgreSQL
Qdrant	Large	High-performance open source

Advanced: Hybrid Search

// Combine semantic search (embeddings) + keyword search (BM25)
// Semantic: understands meaning
// BM25: catches exact terms, technical names

List<Document> semantic = vectorStore.similaritySearch(question);
List<Document> keyword = bm25Index.search(question);
List<Document> combined = rerank(semantic, keyword, question); // RRF or cross-encoder

Interview Tips

RAG is the standard approach for company knowledge bases, documentation Q&A, and product support.
Chunking strategy significantly impacts quality — overlapping chunks prevent boundary cutoffs.
Re-ranking: after initial retrieval, use a cross-encoder model to re-rank results by relevance before injecting into prompt.