Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: CI

on:
push:
pull_request:
branches: [main]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install dependencies
run: pip install -r requirements.txt

- name: Run test suite
run: pytest -q

# Embeddings are local (FastEmbed, BAAI/bge-small-en-v1.5) and the test
# suite has no external API dependency, so this needs no secrets.
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Raghav Malani

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
26 changes: 23 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Codebase Intelligence RAG

[![CI](https://github.com/Ragh234/codebase-intelligence-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/Ragh234/codebase-intelligence-rag/actions/workflows/ci.yml)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**Live demo:** https://codebase-intelligence-rag-hqfteuyappwhonm8yje4agk.streamlit.app/

Codebase Intelligence RAG is a focused RAG/ML engineering project for asking grounded questions about GitHub codebases. It is not a full-stack product: Streamlit is only a thin demo layer, and the core work is repository ingestion, code-aware retrieval, reranking, context construction, and grounded generation with exact citations.

## Why Basic Vector Search Is Not Enough For Code
Expand Down Expand Up @@ -174,15 +180,29 @@ Run:
python -m evaluation.run_evaluation
```

The evaluator reports Hit Rate, Recall@K, Precision@K, and MRR for BM25, vector, hybrid, hybrid + graph, and full reranked retrieval.
The evaluator indexes a small fixture repo (`tests/fixtures/sample_repo`) and reports Hit Rate, Recall@5, Precision@5, and MRR across five retrieval strategies against a 5-question dataset. Measured results:

| Strategy | Hit Rate | Recall@5 | Precision@5 | MRR |
|---|---|---|---|---|
| BM25 only | 0.80 | 0.80 | 0.44 | 0.60 |
| Vector only | 1.00 | 1.00 | 0.52 | 0.717 |
| Hybrid (RRF) | 1.00 | 1.00 | **0.56** | **0.75** |
| Hybrid + Graph | 1.00 | 1.00 | **0.56** | 0.717 |
| Full (Hybrid + Graph + rerank) | 1.00 | 1.00 | 0.52 | 0.717 |

Takeaways, read honestly rather than cherry-picked:

- **Hybrid clearly beats either single-strategy baseline** — BM25-only misses one question entirely (hit rate 0.80 vs. 1.00), and plain vector search has the weakest precision/MRR of anything that hits every question.
- **Graph expansion and reranking don't show a clear win on this 5-question set.** "Full" (with reranking) actually scores fractionally *lower* on precision and MRR than plain Hybrid here. With only 5 questions, that's within noise, not evidence the reranker hurts — but it's also not evidence it helps. A fair statement is: hybrid retrieval is the load-bearing improvement in this project; graph expansion and reranking are structurally sound (tested independently in `tests/`) but this eval set is too small to say whether they help end-to-end.
- Scaling `DATASET` in `evaluation/run_evaluation.py` beyond 5 questions would be the natural next step to get a statistically meaningful answer on reranking.

## Tests

```bash
pytest
```

Tests cover ingestion, parser extraction, chunk metadata, BM25, vector retrieval, hybrid retrieval, metadata filtering, graph expansion, citations, context construction, LLM request construction, and no-LLM evidence fallback.
21 tests pass, covering ingestion, parser extraction, chunk metadata, BM25, vector retrieval, hybrid retrieval, metadata filtering, graph expansion, citations, context construction, LLM request construction, and no-LLM evidence fallback.

## How This Differs From A Basic RAG Chatbot

Expand All @@ -193,7 +213,7 @@ documents -> text chunks -> embeddings -> vector search -> LLM
```

This project:
Deployed Link - https://codebase-intelligence-rag-hqfteuyappwhonm8yje4agk.streamlit.app/

```text
code -> AST/code entities -> metadata-rich chunks -> semantic + lexical retrieval
-> graph relationships -> hybrid ranking -> bounded context -> grounded generation
Expand Down
28 changes: 14 additions & 14 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
streamlit
gitpython
tree-sitter
tree-sitter-python
tree-sitter-javascript
tree-sitter-typescript
tree-sitter-java
tree-sitter-cpp
tree-sitter-go
networkx
fastembed
numpy
httpx
pytest
streamlit==1.63.0
gitpython==3.1.61
tree-sitter==0.26.0
tree-sitter-python==0.25.0
tree-sitter-javascript==0.25.0
tree-sitter-typescript==0.23.2
tree-sitter-java==0.23.5
tree-sitter-cpp==0.23.4
tree-sitter-go==0.25.0
networkx==3.6.1
fastembed==0.8.0
numpy==2.5.2
httpx==0.28.1
pytest==9.1.1
Loading