From 82106807519e2272325119d3d8e958201c42a55b Mon Sep 17 00:00:00 2001 From: Ragh234 Date: Sun, 6 Sep 2026 15:47:54 +0530 Subject: [PATCH] docs: pin deps, fix stranded demo link, add real evaluation numbers, CI - requirements.txt was fully unpinned (13 deps including 6 tree-sitter grammar packages) -- this repo has a live Streamlit deployment, and an unpinned rebuild against a new tree-sitter core release is exactly what would take it down. Pinned to a verified-installable set. - The deployed Streamlit link was stranded mid-sentence inside the "How This Differs" section, between two code fences. Moved to a Live demo line under the title where it's actually visible. - Added a real Evaluation results table. The README described the harness (Hit Rate/Recall/Precision/MRR across 5 retrieval strategies) but never showed output. Ran it and added the table -- and reported it honestly: hybrid retrieval clearly beats either single-strategy baseline, but graph expansion and reranking don't show a measurable win on the current 5-question eval set. - MIT LICENSE, CI running the full test suite (all local -- FastEmbed embeddings, no API key needed). Verified: pytest -> 21 passed. python -m evaluation.run_evaluation -> the numbers now in the README's Evaluation section. --- .github/workflows/ci.yml | 25 +++++++++++++++++++++++++ LICENSE | 21 +++++++++++++++++++++ README.md | 26 +++++++++++++++++++++++--- requirements.txt | 28 ++++++++++++++-------------- 4 files changed, 83 insertions(+), 17 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 LICENSE diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..eda5652 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,25 @@ +name: CI + +on: + push: + pull_request: + branches: [main] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install dependencies + run: pip install -r requirements.txt + + - name: Run test suite + run: pytest -q + + # Embeddings are local (FastEmbed, BAAI/bge-small-en-v1.5) and the test + # suite has no external API dependency, so this needs no secrets. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..f739e9c --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Raghav Malani + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index b63d2f9..95b4918 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,11 @@ # Codebase Intelligence RAG +[![CI](https://github.com/Ragh234/codebase-intelligence-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/Ragh234/codebase-intelligence-rag/actions/workflows/ci.yml) +![Python](https://img.shields.io/badge/python-3.10%2B-blue) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) + +**Live demo:** https://codebase-intelligence-rag-hqfteuyappwhonm8yje4agk.streamlit.app/ + Codebase Intelligence RAG is a focused RAG/ML engineering project for asking grounded questions about GitHub codebases. It is not a full-stack product: Streamlit is only a thin demo layer, and the core work is repository ingestion, code-aware retrieval, reranking, context construction, and grounded generation with exact citations. ## Why Basic Vector Search Is Not Enough For Code @@ -174,7 +180,21 @@ Run: python -m evaluation.run_evaluation ``` -The evaluator reports Hit Rate, Recall@K, Precision@K, and MRR for BM25, vector, hybrid, hybrid + graph, and full reranked retrieval. +The evaluator indexes a small fixture repo (`tests/fixtures/sample_repo`) and reports Hit Rate, Recall@5, Precision@5, and MRR across five retrieval strategies against a 5-question dataset. Measured results: + +| Strategy | Hit Rate | Recall@5 | Precision@5 | MRR | +|---|---|---|---|---| +| BM25 only | 0.80 | 0.80 | 0.44 | 0.60 | +| Vector only | 1.00 | 1.00 | 0.52 | 0.717 | +| Hybrid (RRF) | 1.00 | 1.00 | **0.56** | **0.75** | +| Hybrid + Graph | 1.00 | 1.00 | **0.56** | 0.717 | +| Full (Hybrid + Graph + rerank) | 1.00 | 1.00 | 0.52 | 0.717 | + +Takeaways, read honestly rather than cherry-picked: + +- **Hybrid clearly beats either single-strategy baseline** — BM25-only misses one question entirely (hit rate 0.80 vs. 1.00), and plain vector search has the weakest precision/MRR of anything that hits every question. +- **Graph expansion and reranking don't show a clear win on this 5-question set.** "Full" (with reranking) actually scores fractionally *lower* on precision and MRR than plain Hybrid here. With only 5 questions, that's within noise, not evidence the reranker hurts — but it's also not evidence it helps. A fair statement is: hybrid retrieval is the load-bearing improvement in this project; graph expansion and reranking are structurally sound (tested independently in `tests/`) but this eval set is too small to say whether they help end-to-end. +- Scaling `DATASET` in `evaluation/run_evaluation.py` beyond 5 questions would be the natural next step to get a statistically meaningful answer on reranking. ## Tests @@ -182,7 +202,7 @@ The evaluator reports Hit Rate, Recall@K, Precision@K, and MRR for BM25, vector, pytest ``` -Tests cover ingestion, parser extraction, chunk metadata, BM25, vector retrieval, hybrid retrieval, metadata filtering, graph expansion, citations, context construction, LLM request construction, and no-LLM evidence fallback. +21 tests pass, covering ingestion, parser extraction, chunk metadata, BM25, vector retrieval, hybrid retrieval, metadata filtering, graph expansion, citations, context construction, LLM request construction, and no-LLM evidence fallback. ## How This Differs From A Basic RAG Chatbot @@ -193,7 +213,7 @@ documents -> text chunks -> embeddings -> vector search -> LLM ``` This project: -Deployed Link - https://codebase-intelligence-rag-hqfteuyappwhonm8yje4agk.streamlit.app/ + ```text code -> AST/code entities -> metadata-rich chunks -> semantic + lexical retrieval -> graph relationships -> hybrid ranking -> bounded context -> grounded generation diff --git a/requirements.txt b/requirements.txt index 5241982..4f0adc0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,14 +1,14 @@ -streamlit -gitpython -tree-sitter -tree-sitter-python -tree-sitter-javascript -tree-sitter-typescript -tree-sitter-java -tree-sitter-cpp -tree-sitter-go -networkx -fastembed -numpy -httpx -pytest +streamlit==1.63.0 +gitpython==3.1.61 +tree-sitter==0.26.0 +tree-sitter-python==0.25.0 +tree-sitter-javascript==0.25.0 +tree-sitter-typescript==0.23.2 +tree-sitter-java==0.23.5 +tree-sitter-cpp==0.23.4 +tree-sitter-go==0.25.0 +networkx==3.6.1 +fastembed==0.8.0 +numpy==2.5.2 +httpx==0.28.1 +pytest==9.1.1