Engineering8 min
Why we built our own RAG pipeline instead of buying one
Off-the-shelf retrieval kept getting our answers wrong. Here's the hybrid approach that fixed it.
GLGuideLite EngineeringMay 14, 2026
The retrieval problem
Vanilla dense retrieval, meaning embed the query and find nearest neighbours in vector space, handles semantic similarity beautifully. Ask "how do I change my subscription" and it finds the billing FAQ even if the exact phrase doesn't appear. But it falls apart on exact lookups: version numbers, SKU codes, proper nouns your embedding model has never seen. Cosine similarity doesn't know that GL-2.1.4 matters more than GL-2.1.3.
Sparse retrieval (BM25 and friends) has the opposite failure profile. It's exact-match-first, which is great for lookup queries. But ask anything conceptual, such as what's the best way to handle rate limits, and it returns documents with high keyword overlap but low semantic relevance. Users get articles that mention rate limits in passing instead of the one that actually explains the backoff strategy.
The failure mode nobody talks about enough is the compounding effect: when retrieval misses, the LLM doesn't say "I don't know." It reads the top-k irrelevant chunks and tries to synthesise an answer anyway. The result is confident-sounding hallucination, and hallucinations in a customer-facing support agent are worse than a graceful escalation.
Why we didn't buy
We evaluated three off-the-shelf solutions seriously. The first gave us clean infrastructure and fast iteration, but the hybrid search configuration required us to maintain two separate indexes and hand-tune the interpolation weight between dense and sparse scores, a number that shifted every time we updated our document corpus. Operational burden was too high.
The second option's hybrid retriever was easier to configure, but the reranking step used a cross-encoder that added 400 to 600ms of latency. For a real-time chat widget with a 1500ms total budget, that left almost nothing for the LLM call. We'd have needed to run the reranker asynchronously with speculative generation, which introduced a complexity cliff we weren't ready to climb.
The third was closest to what we needed, but its managed tier didn't support the custom tokenizer we needed for handling our domain-specific terminology: version strings, product codes, and abbreviations that generic tokenizers split incorrectly.
Build when the seam between retrieval and your prompt is a source of bugs and latency you can't control, or when domain-specific tokenization is critical to answer quality. Buy when generic retrieval is good enough and you want to ship in weeks, not months.
Our hybrid approach
Our pipeline runs two stages. Stage one is parallel retrieval: we fire a BM25 query and a dense embedding query simultaneously, collecting the top 20 results from each. Stage two is a lightweight cross-encoder reranker that scores all unique candidates from the combined pool and returns the top 5. The key optimisation is that we run the reranker on a fine-tuned 33M-parameter model rather than a general-purpose one, which keeps latency under 80ms.
The tokenizer was the hardest part. We trained a custom BPE tokenizer on our entire help-centre corpus so that product identifiers and version strings are kept as single tokens. This alone moved our exact-match recall from 71% to 91% on version-specific queries.
# Two-stage hybrid retrieval pipeline
from gl_retrieval import BM25Index, DenseIndex, CrossEncoderReranker
def retrieve(query: str, top_k: int = 5) -> list[dict]:
# Stage 1: parallel sparse + dense retrieval
bm25_hits = bm25_index.search(query, top_n=20)
dense_hits = dense_index.search(query, top_n=20)
# Merge and deduplicate by chunk_id
candidates = {h["chunk_id"]: h for h in bm25_hits + dense_hits}.values()
# Stage 2: cross-encoder reranker
scored = reranker.score(query, list(candidates))
scored.sort(key=lambda x: x["score"], reverse=True)
return scored[:top_k]
Numbers and lessons
Recall@5 moved from 61% to 89% on our production query set after six weeks of running the hybrid pipeline. Hallucination rate, measured by our eval harness flagging factually incorrect claims against our source documents, dropped by 73%. End-to-end latency for the retrieval step is 110ms at p95, well within budget.
The main lesson: retrieval quality is the highest-leverage place to invest in RAG. A mediocre model with excellent context beats an excellent model with poor retrieval on accuracy benchmarks every time. When answer quality is your bottleneck, look at what's in the prompt before you reach for a bigger model.
89%recall@5 on production queries
Want this running on your own agents?
Start free, or book 30 minutes and we will walk through it against your stack.
Keep reading
EngineeringBuilding an evaluation harness for production agentsHow to test your agents the way you test code, with regression suites and golden conversations.ProductIntroducing Agent Versioning: ship faster, rollback fasterEvery change you make to your agent is now tracked, comparable, and reversible. Here's how it works.GuidesPrompts vs. fine-tuning: when to reach for whichA practical decision tree based on years of building agents, with cost and latency math.