VeriRAG — Self-Evaluating Agentic RAG Platform

Verifying Every Answer Before It's Trusted

The Challenge

Standard RAG systems hallucinate when internal documentation is incomplete, ambiguous, or irrelevant to a query. The goal was an autonomous agent that scores retrieved context quality mid-flight and dynamically seeks external validation via web search when needed — all while keeping sensitive user data out of persistent model telemetry.

Complexity and Innovation

An evaluation node classifies retrieved context as Correct, Ambiguous, or Incorrect relative to user intent using structured outputs. The graph then branches programmatically: accurate context proceeds to refinement, while ambiguous or incorrect matches instantly trigger a live web search via Tavily. Custom span processors hook into the MLflow tracking cycle to automatically detect and redact personal email addresses from traces in real time.

The Process

We combined LangGraph state graphs with FastAPI endpoints for a fast, modular backend, then integrated MLflow's GenAI evaluation engine to run prompt variant tests, scoring iterations on safety, answer accuracy, and style before promoting them to production.

Self-Verifying Answers, Zero Blind Trust

VeriRAG treats retrieval quality as an active decision point rather than a passive assumption, catching weak context before it reaches the user.

Feature Inventory

Engineering Trustworthy, Self-Correcting Retrieval VeriRAG closes the loop between retrieval, evaluation, and continuous prompt improvement.

• Intent query router separating conversational queries from document-dependent lookups
• Self-evaluating retrieval node with Pydantic validation on vector outputs
• Web search failover for out-of-knowledge-base topics
• Automated telemetry redaction to scrub PII from traces
• LLM-as-a-judge evaluation scoring safety, accuracy, and relevance
• Prompt registry comparison against real validation datasets

“Moving from linear RAG pipelines to state-graph agent networks let us turn retrieval evaluation into an active decision, not an afterthought.” — Manu Sharma, CEO

Transforming Agentic RAG Reliability

Blending Structured Evaluation with Adaptive Retrieval VeriRAG's architecture makes context quality a first-class decision in every response cycle.

• Multi-Tier Structured Routing: Classifies retrieved context as Correct, Ambiguous, or Incorrect before generation.
• Dynamic Fallback Switching: Automatically triggers web search when retrieved documents fall short.
• In-Flight Data Sanitization: Redacts personal information from traces before they're saved.
• State-Graph Architecture: LangGraph orchestrates multi-step reasoning across agent states.
• Judge-Based Quality Scoring: Evaluates responses on safety, accuracy, and relevance automatically.
• Prompt Version Benchmarking: Compares prompt iterations against real validation datasets.

Conclusion

Delivering Verified, Privacy-Safe AI Responses VeriRAG proved that self-verification and strict privacy compliance can scale together in production.

• Reduced Hallucinations: Dynamic context validation and web fallback minimized inaccurate responses.
• Strict Privacy Compliance: Automated PII redaction secured sensitive data in telemetry.
• Accelerated Optimization: Automated LLM evaluation replaced slow manual testing cycles.
• Improved Response Quality: Judge-based scoring maintained consistently high safety and accuracy standards.
• Full Observability: MLflow tracing gave complete visibility into agent decision paths.