Building a Production-Ready RAG Application

01 // The Technical Problem

Standard RAG setups run simple vector lookups and feed the top results to the LLM. In production, this approach fails because vector similarity does not guarantee semantic relevance. The database may return text fragments that contain similar words but do not answer the question, leading to generic or incorrect model responses.

02 // Background & Engineering Constraints

To improve accuracy, we must implement a multi-stage search pipeline. When a query is sent, the system should rewrite it for search optimization, query the database using a hybrid vector-keyword algorithm, rerank the retrieved chunks, and filter context before sending it to the LLM.

03 // Architecture Solution & Implementation

We construct an optimized RAG pipeline. We implement BM25 keyword matching alongside vector similarity search to capture both exact matches and semantic meaning. We pass the retrieved segments to a cross-encoder model to rerank them by relevance, ensuring only the best context is sent to the LLM.

We write database search functions in Python. We use pgvector for similarity queries and BM25 for keyword search. We merge the results using Reciprocal Rank Fusion (RRF), run a Cohere reranking model, and inject the top 3 segments into the LLM system prompt.

Implementation Snippet (python)

def reciprocal_rank_fusion(vector_results, keyword_results, k=60):
    scores = {}
    # Run Reciprocal Rank Fusion on merged search pools
    for rank, item in enumerate(vector_results):
        scores[item.id] = scores.get(item.id, 0) + 1.0 / (rank + k)
    for rank, item in enumerate(keyword_results):
        scores[item.id] = scores.get(item.id, 0) + 1.0 / (rank + k)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Frequently Asked Technical Questions