SEC RAG Engine

A query starts in the React client and hits our FastAPI backend. Instead of naively converting the prompt into an embedding and running cosine similarity against millions of vectors, the query passes through three distinct phases: Pre-Retrieval Intent Analysis, 2-Pronged Hybrid Retrieval with Reciprocal Rank Fusion, and Cross-Encoder Neural Reranking.

Only the top reranked chunks—verified for semantic relevance and exact keyword presence—are formatted into an XML-delimited prompt passed to Llama 3.3 70B on Groq LPUs. The response is streamed back with paragraph-level source citations and latency telemetry.

The primary failure mode in multi-document financial RAG is context cross-contamination: asking 'Compare supply chain risks between NVDA and AVGO' causes standard vector search to retrieve a tangled mix of paragraphs where company identity gets lost.

We solve this before search begins using an LLM Query Analyzer. A fast Llama model extracts the exact stock tickers, fiscal years, and target document sections (Item 1A Risk Factors, Item 7 MD&A, or Item 8 Financial Statements). These become deterministic boolean filters applied at the database payload level in Qdrant, restricting the search space before any vector computation happens.

Dense embeddings (Nvidia NV-EmbedQA) excel at conceptual similarity ('how does export control impact revenue?') but struggle with specific alphanumeric strings, ticker symbols, financial metrics, and legal clause IDs. BM25 sparse keyword search has the exact opposite strength.

We execute both in parallel: Qdrant runs dense vector search while an in-memory BM25 index scores keyword frequency over the pre-filtered corpus. The candidate lists are merged using Reciprocal Rank Fusion: RRF_Score = Σ (1 / (k + rank)), with k=60. This mathematically balances conceptual relevance with exact keyword hits without requiring arbitrary weighting calibrations.

Bi-encoders (embedding models) evaluate query and document separately to allow fast vector indexing, losing fine-grained cross-token attention. To restore accuracy, our top 25 RRF candidates pass through a Cross-Encoder Reranker (Llama-Nemotron-Rerank hosted on Nvidia NIM).

The cross-encoder feeds the query and document chunk together into full multi-head self-attention layers, scoring exact relevance from 0 to 1. Chunks below threshold are discarded, reducing context window bloat and eliminating hallucination from weakly related paragraphs.

The LLM generation layer runs Llama 3.3 70B on Groq LPUs for fast time-to-first-token. The context is injected using strict XML delimiters: <document id='NVDA_2025_10K_p42'>...

The system prompt enforces strict grounding: every claim must cite its document ID, no speculative extrapolations are permitted, and if the retrieved filings do not contain the answer, the model explicitly declares insufficient context. ProviderManager provides automatic key rotation and fallback to 8B models if primary rate limits are approached.

To move from an architecture demo to enterprise production, we designed four critical extensions: (1) Multi-Query Decomposition: splitting complex comparative questions into parallel sub-queries per entity, retrieving per-company candidate pools, and merging them. (2) Conversational Query Rewriting: resolving conversational pronouns and follow-up questions ('What about their margins?') into standalone search queries using chat history.

(3) Hierarchical Parent-Child Chunking: indexing 200-word child chunks for high-precision retrieval, but returning the enclosing 1,000-word parent section to the LLM to preserve complete context. (4) Self-Reflective Verification: running a lightweight evaluator pass that validates whether citations exist verbatim in source text before displaying them to users.