RAG architecture · Hybrid search · Multi-agent systems · SEC RAG Engine
From naive vector search to production RAG: Intent routing, hybrid retrieval, and multi-agent execution
Why basic vector search breaks down on financial filings and complex documents, and how we architected a production-level RAG engine with intent classification, cross-encoders, and parallel agent execution.
By the CognifySystems engineering team · · 8 min read
The problem with standard vector search in production
Most off-the-shelf RAG tutorials follow a simplistic formula: split documents into static character chunks, generate dense vector embeddings, store them in a vector database, and retrieve top-k results by cosine similarity. In high-stakes enterprise domains—such as SEC financial filings or commercial legal contracts—this naive pattern fails repeatedly.
First, dense embeddings suffer from exact-match blindness. They capture broad semantic themes but struggle to isolate specific company tickers (AVGO vs. NVDA), exact dollar figures ($2.4B vs. $2.4M), or specific legal clauses. Asking for 'debt covenant restrictions' often returns generic paragraphs about corporate debt from unrelated sections.
Second, naive retrieval fails at multi-document synthesis. When an analyst asks 'Compare supply chain bottlenecks between Nvidia, Broadcom, and TSMC', basic vector search retrieves an unorganized slurry of paragraphs from all three companies. The downstream LLM loses track of entity attribution, frequently attributing one company's risks to another.
To solve this, a production system cannot be a single vector query. It must be an intelligent, multi-stage decision pipeline.
Stage 1: Intent classification, confidence scoring, and routing
Every query entering the system is first evaluated by a fast Intent Classifier. Rather than immediately querying a vector database, the classifier determines the nature of the request and assesses its confidence score:
1. Context Follow-up: If the user asks a conversational follow-up ('What about their operating margins in Asia?'), the system loads the active conversation summary and synthesizes a self-contained standalone question before initiating retrieval.
2. Simple Policy RAG: Focused, single-entity queries are routed directly through a streamlined hybrid retrieval and cross-encoder pipeline.
3. Complex Multi-Entity Tasks: Questions requiring cross-document comparison or multi-step analysis are routed to an Agent Planner.
4. Unclassified Intent: If a prompt is ambiguous or out of domain, the system triggers a clarification loop rather than wasting compute on hallucinated retrieval.
Stage 2: Hybrid retrieval — BM25 + dense embeddings with RRF
For queries routed to retrieval, we run two complementary search engines in parallel:
Dense Retrieval (via Nvidia NV-EmbedQA) maps conceptual queries that do not match exact wording (for example, mapping 'geopolitical trade barriers' to regulatory disclosures about export restrictions).
Sparse Retrieval (via BM25) pinpoints exact tickers, part numbers, executive names, and financial line items.
The results are merged using Reciprocal Rank Fusion (RRF): RRF_Score(d) = Σ [1 / (k + rank(d))], with k=60. RRF mathematically combines rank positions without requiring fragile score normalization between dense and sparse models.
Stage 3: Cross-encoder reranking and the evidence verification gate
Bi-encoder embedding models are fast because they encode queries and documents independently, but that speed comes at the cost of fine-grained cross-token attention.
We pass the top 25 RRF candidates through a neural Cross-Encoder (Llama-Nemotron-Rerank hosted on Nvidia NIM). The cross-encoder evaluates the query and candidate chunk together with full multi-head self-attention, generating a precise relevance score between 0.0 and 1.0.
Crucially, the pipeline features an Evidence Verification decision gate: if candidate chunks fail to surpass the relevance threshold, the system halts generation and returns an explicit 'No Supporting Evidence' status rather than hallucinating.
Stage 4: Complex tasks — Task decomposition & parallel agent execution
When a user asks a complex comparative or multi-part question, a single retrieval pass cannot gather sufficient context without hitting context window pollution.
In our architecture, the Agent Planner analyzes the task, breaks it down, and formulates a multi-part Query Strategy. Queries are deposited into a central Query Pool, while an Agent Pool dispatches parallel sub-agents (Sub-Agent A, Sub-Agent B, Sub-Agent C) each dedicated to a specific entity or sub-problem.
Each sub-agent retrieves its own verified evidence pool. An Answer Aggregation layer then consolidates all sub-agent reports and query results into a structured, cohesive synthesis with XML-grounded citations.
Stage 5: Production resilience and cost engineering
In production, uptime and latency are as critical as retrieval accuracy. Our implementation incorporates three reliability layers:
- Stateless Cloud Architecture: Heavy vector lookups (Qdrant Cloud), embeddings and reranking (Nvidia NIM), and generation (Groq LPUs) run via stateless APIs, preventing local out-of-memory (OOM) crashes under concurrent user load.
- Automated Provider Failover: Dynamic multi-key rotation across Groq Llama 3.3 70B, with automatic fallback to 8B models (separate rate limit buckets) or Gemini Flash Lite if provider quotas are approached.
- Verbatim Grounding Verification: Before final delivery, citations are verified against source document chunks; any non-grounded claim is flagged or regenerated.
Takeaways for engineering enterprise RAG
Treat information retrieval as a deterministic software engineering problem rather than a single vector similarity call:
- Classify intent before retrieval to route between simple lookup, conversational rewrite, and multi-agent decomposition.
- Combine BM25 and dense embeddings with Reciprocal Rank Fusion to cover both exact keywords and conceptual semantics.
- Always gate generation with a cross-encoder and an explicit 'No Supporting Evidence' fallback.
- Use parallel sub-agents for multi-document comparisons instead of cramming everything into one prompt.