Agentic RAG System: Resilient Local-Only Enterprise Architecture

An end-to-end, local-first Agentic Retrieval-Augmented Generation (RAG) system built for enterprise reliability, absolute data privacy, and deterministic grounding. Designed specifically for deep financial filings, corporate annual reports, 10-Ks, and sensitive internal documentation, this system executes 100% locally with Zero Data Leakage.

---

Architectural Shift: The Zero Data Leakage Imperative

Enterprise documents—such as earnings reports before public disclosure, board minutes, M&A filings, and healthcare records—cannot be routed through third-party cloud inference APIs (e.g., OpenAI, Anthropic, or cloud providers) without violating strict data privacy regulations (GDPR, HIPAA, SOC 2 Type II) and non-disclosure agreements.

This repository implements a deliberate, uncompromising architectural shift to a 100% local, air-gapped pipeline:

Component	Technology	Enterprise Role
Local Orchestration	CrewAI	Multi-agent deliberation, goal decomposition, and dynamic tool selection on local threads.
Persistent Retrieval	ChromaDB	On-disk local vector index (./company_db) with automatic SQLite FTS5 fallback.
Local Inference	Ollama	Local execution of open-weight foundation models (ollama/llama3.2, llama3.1, mistral).
Local Embeddings	Sentence-Transformers	all-MiniLM-L6-v2 dense vector embeddings (384 dimensions) generated in-process on CPU.
Zero Cloud Keys	Sanitized Environment	All external API keys and cloud dependencies stripped to eliminate outbound data leakage.

                               ┌──────────────────────────────────────────────────────────┐
                               │       100% AIR-GAPPED LOCAL PROCESS BOUNDARY             │
                               │                                                          │
┌──────────────────────┐       │   ┌────────────────────┐      ┌──────────────────────┐   │
│  Enterprise Report   │ ────> │   │   DocumentLoader   │ ───> │   DocumentSplitter   │   │
│  (PDF / TXT / 10-K)  │       │   │ (pypdf / metadata) │      │ (800 char / 150 ovlp)│   │
└──────────────────────┘       │   └────────────────────┘      └──────────────────────┘   │
                               │                                           │              │
                               │                                           ▼              │
┌──────────────────────┐       │   ┌────────────────────┐      ┌──────────────────────┐   │
│ Glass Box Live Audit │ <──── │   │    CrewAI Agent    │ <─── │   ChromaDB (Local)   │   │
│   (Streamlit / UI)   │       │   │ (Ollama Llama 3.2) │      │ (all-MiniLM-L6-v2)   │   │
└──────────────────────┘       │   └────────────────────┘      └──────────────────────┘   │
                               │             ▲                            │               │
                               │             │ (Circuit Breaker)          │ (Fallback)    │
                               │             └────── SQLite FTS5 ◄────────┘               │
                               │                                                          │
                               └──────────────────────────────────────────────────────────┘

---

The Production Hook: Reliability & Three-Tier Guardrails

Production RAG systems fail when they encounter missing context, vector database latency spikes, or hallucination vulnerabilities. The system enforces a Three-Tier Reliability Guardrail architecture (src/agentic_rag/guardrails.py):

User Query
    │
    ▼
[ Tier 1: Input Guardrail ] ─────────> [ BLOCKED ] (Adversarial Injection / Empty)
    │ Passed
    ▼
[ Tier 2: Retrieval Guardrail ]
    ├── Vector DB Timeout? ──────────> [ CIRCUIT BREAKER ] ──> SQLite FTS5 Fallback
    │
    └── Similarity < Threshold (0.65)?
            │
            ├── Yes ─────────────────> [ DISCIPLINED REFUSAL ] (Missing Context Detected)
            └── No  ─────────────────> High-Confidence Evidence Chunks
                                              │
                                              ▼
[ Tier 3: Output Guardrail ] ────────> [ HALLUCINATION AUDITOR ] ──> Verified Answer

1. Tier 1: Input Guardrails & Injection Defense

• Adversarial Pattern Interception: Detects and neutralizes prompt injections, system override signatures (ignore previous instructions, DAN mode, developer mode), and malicious inputs before token transmission.
• Query Normalization: Validates length, strips control tokens, and deconstructs question intent.

2. Tier 2: Retrieval Guardrails & Missing-Context Fallback

• Dynamic Confidence Gating: Calculates cosine similarity for all retrieved chunks. If the top similarity score falls below the configurable confidence gate (default: 0.65), the system refuses to speculate.
• Missing-Context Refusal Protocol: When an out-of-domain query is detected (e.g., asking about Apple or Tesla in a TCS annual report), the pipeline immediately triggers an explicit refusal stating that document coverage is absent, preventing factual hallucination.
• Vector DB Timeout Circuit Breaker: ChromaDB vector operations are wrapped with a timeout threshold (default: 3.0s). If the vector store hangs or fails, the pipeline automatically falls back to local SQLite FTS5 full-text keyword search without crashing.

3. Tier 3: Output Guardrails & Hallucination Auditor

• Numerical Metric Extraction: Automatically extracts all numbers, currency figures ($, ₹), percentages (%), and financial metrics from the agent's output.
• Grounding Cross-Check: Verifies each extracted figure against the verbatim text in the retrieved chunks.
• Hallucination Risk Score: Computes a real-time risk metric (0.0% for fully grounded outputs) and enforces page-level citations ([Source X • Page Y]).

---

The Live Audit: Glass Box Workflow

Rather than hiding agent decisions inside a black box, the system provides a Glass Box Live Audit Console (built with Streamlit) that exposes the agent's internal state in real time:

1. Step-by-Step Reasoning Traces (Chain of Thought): Live stage-by-stage inspection showing intent deconstruction, tool selection rationale, search parameter formulation, and metric verification.
2. ChromaDB Similarity Scores & Distance Inspector: Every retrieved chunk displays its exact cosine distance, normalized similarity percentage, chunk ID, page number, and comparison to the confidence threshold.
3. Real-Time Fallback Trigger Monitor: Prominent visual status alerts show whether the query executed nominally, triggered a low-confidence refusal, engaged the SQLite FTS5 circuit breaker, or was quarantined by input guardrails.
4. Interactive Reliability Test Presets:
   • 🟢 Grounded In-Domain Audit: Runs a standard financial inquiry (Operating Margin & RoE) with high similarity (> 0.85), passing all guardrails.
   • 🟡 Missing Context Fallback: Runs an out-of-domain query (e.g., Apple iPhone revenue in a TCS filing), demonstrating automatic similarity drop (~0.42 < 0.65) and disciplined refusal.
   • 🟠 Vector Timeout Circuit Breaker: Simulates a ChromaDB timeout to verify zero-downtime fallback to SQLite FTS5.
   • 🔴 Adversarial Injection Block: Demonstrates instant prompt injection quarantine.

---

Getting Started: Local Setup Guide

Prerequisites

• Operating System: macOS, Linux, or Windows (WSL2 recommended)
• Python: 3.10, 3.11, 3.12, or 3.13
• Ollama: Installed locally (ollama.com)

Step 1: Install Ollama & Pull the Model

Download and install Ollama from ollama.com/download.

# Verify Ollama installation
ollama --version

# Start the Ollama daemon (runs on http://localhost:11434)
ollama serve

# In another terminal, pull the recommended lightweight model
ollama pull llama3.2

(Optional: You can also use other local models such as ollama pull llama3.1 or ollama pull mistral).

Step 2: Clone the Repository & Set Up Environment

git clone https://github.com/JohnnyWilson16/agentic-rag-system.git
cd agentic-rag-system

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Step 3: Configure Environment Variables

Copy the template to .env. Note that no cloud API keys are required or supported:

cp .env.example .env

Your .env file should contain the local-only configuration:

LLM_PROVIDER=ollama
LLM_MODEL=ollama/llama3.2
OLLAMA_BASE_URL=http://localhost:11434
CONFIDENCE_THRESHOLD=0.65
VECTOR_DB_TIMEOUT_SECONDS=3.0
ENABLE_GUARDRAILS=true
ZERO_DATA_LEAKAGE_ENFORCED=true
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
VECTOR_STORE_DIR=./company_db
RETRIEVER_K=3

Step 4: Index a Document into Local ChromaDB

Index the included sample corporate report or your own PDF:

# Ingest sample report
python -m agentic_rag.cli ingest data/sample_annual_report.txt --reset

# Ingest a PDF document
python -m agentic_rag.cli ingest data/annual_report_2025_2026.pdf --reset

Output:

Document Ingestion Summary
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Property            ┃ Value                               ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Document Name       │ annual_report_2025_2026.pdf         │
│ Raw Pages / Sections│ 360                                 │
│ Chunks Created      │ 1,818                               │
│ Total Vectors in DB │ 1,818                               │
│ Embedding Model     │ sentence-transformers/all-MiniLM-L6 │
│ Persist Directory   │ ./company_db                        │
└─────────────────────┴─────────────────────────────────────┘
✓ Ingestion completed successfully.

---

Launching the User Interfaces

Option A: Streamlit Glass Box Live Audit Console (Recommended for Demos)

Launch the interactive audit interface with live telemetry, fallback triggers, and confidence controls:

./run_demo.sh
# Or directly:
streamlit run app.py

Open http://localhost:8501 in your browser.

Option B: Enterprise Web Console & REST API

Launch the FastAPI backend serving the single-page application:

./run_app.sh
# Or directly:
python -m uvicorn agentic_rag.api:app --host 127.0.0.1 --port 8000

Open http://127.0.0.1:8000 in your browser.

Option C: Command-Line Interface (CLI)

Run queries directly from your terminal:

# Targeted query
python -m agentic_rag.cli query "What was the operating margin and Return on Equity for FY 2026?"

# Full multi-section audit
python -m agentic_rag.cli analyze --company "Tata Consultancy Services"

# Check system status
python -m agentic_rag.cli status

---

Project Structure

agentic-rag-system/
├── app.py                           # Streamlit Glass Box Live Audit Console
├── main.py                          # ASGI server root entrypoint
├── run_demo.sh                      # One-click launcher for Streamlit Audit Console
├── run_app.sh                       # One-click launcher for Web Console & FastAPI
├── data/
│   ├── sample_annual_report.txt     # Sample corporate financial text document
│   └── annual_report_2025_2026.pdf  # Comprehensive 360-page financial report PDF
├── scripts/
│   ├── create_sample_pdf.py         # Utility to generate structured benchmark PDFs
│   └── run_analysis.py              # Quickstart offline analysis script
├── src/
│   └── agentic_rag/
│       ├── __init__.py              # Package initialization and exports
│       ├── agents.py                # CrewAI agent definitions & grounding prompts
│       ├── api.py                   # FastAPI REST API & session manager
│       ├── cli.py                   # Command-line interface with Rich formatting
│       ├── config.py                # Pydantic settings with Zero Data Leakage enforcer
│       ├── demo_engine.py           # Glass-box execution engine with guardrails telemetry
│       ├── document_loader.py       # PDF/TXT loader with page metadata extraction
│       ├── embeddings.py            # Low-memory ONNX / CPU sentence embeddings
│       ├── guardrails.py            # Three-Tier Reliability Guardrails & Enforcer
│       ├── llm_factory.py           # CrewAI local Ollama LLM constructor
│       ├── pipeline.py              # End-to-End Agentic RAG pipeline coordinator
│       ├── tasks.py                 # Structured analytical task definitions
│       ├── text_splitter.py         # Recursive semantic chunking (800 chars / 150 ovlp)
│       ├── tools.py                 # CrewAI tool wrapper for vector retrieval
│       ├── vector_store.py          # ChromaDB persistence & SQLite FTS5 fallback
│       └── static/                  # Web Console SPA (HTML, CSS, JS)
├── tests/
│   ├── conftest.py                  # Pytest fixtures and environment configuration
│   ├── test_api.py                  # API endpoint and health tests
│   ├── test_config.py               # Configuration & Zero Data Leakage tests
│   ├── test_document_processing.py  # PDF loading and chunking tests
│   ├── test_guardrails.py           # Comprehensive tests for three-tier guardrails
│   ├── test_pipeline.py             # Pipeline orchestration & CrewAI mock tests
│   ├── test_tools_and_agent.py      # Search tool formatting and task tests
│   └── test_vector_store.py         # ChromaDB indexing & similarity search tests
├── .env.example                     # Environment template for local Ollama
├── LICENSE                          # MIT License
├── pyproject.toml                   # Build metadata and package dependencies
└── requirements.txt                 # Frozen dependency specifications

---

Automated Verification & Testing

Execute the comprehensive test suite with pytest:

pytest tests/ -v

The test suite covers:

• Zero Data Leakage: Verifies that cloud API keys are stripped and external URLs are rejected.
• Input Guardrail: Tests valid queries, length bounds, and prompt injection signatures.
• Retrieval Guardrail: Tests confidence gating, missing-context fallback, and vector timeout circuit breaker.
• Output Guardrail: Validates numerical metric cross-referencing and hallucination risk calculation.
• Vector Store & Ingestion: Tests chunking, embedding generation, ChromaDB persistence, and SQLite FTS5 fallback.
• Pipeline & API: Verifies CrewAI tool binding, task coordination, and REST endpoint contracts.

---

License

MIT License. See LICENSE for details.