Building a Production-Ready RAG Application

01 // The Problem

Standard RAG setups run simple vector lookups and feed the top results to the LLM. In production, this approach fails because vector similarity does not guarantee semantic relevance. The database may return text fragments that contain similar words but do not answer the question, leading to generic or incorrect model responses.

02 // The Context

To improve accuracy, we must implement a multi-stage search pipeline. When a query is sent, the system should rewrite it for search optimization, query the database using a hybrid vector-keyword algorithm, rerank the retrieved chunks, and filter context before sending it to the LLM.

03 // The Solution

We construct an optimized RAG pipeline. We implement BM25 keyword matching alongside vector similarity search to capture both exact matches and semantic meaning. We pass the retrieved segments to a cross-encoder model to rerank them by relevance, ensuring only the best context is sent to the LLM.

04 // System Architecture

05 // The Implementation

We write database search functions in Python. We use pgvector for similarity queries and BM25 for keyword search. We merge the results using Reciprocal Rank Fusion (RRF), run a Cohere reranking model, and inject the top 3 segments into the LLM system prompt.

06 // Key Engineering Lessons

• Implementing a reranking step increases final response accuracy by up to 40% while reducing model input tokens.
• Always implement an embeddings cache. Storing vectors for common questions avoids redundant API calls and reduces latency.
• Set up strict token limit parameters. Too much context degrades LLM output quality.

07 // Technical Code Implementation

def reciprocal_rank_fusion(vector_results, keyword_results, k=60):
    scores = {}
    # Run Reciprocal Rank Fusion on merged search pools
    for rank, item in enumerate(vector_results):
        scores[item.id] = scores.get(item.id, 0) + 1.0 / (rank + k)
    for rank, item in enumerate(keyword_results):
        scores[item.id] = scores.get(item.id, 0) + 1.0 / (rank + k)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)