title	Civil Prep
emoji	📖
colorFrom	red
colorTo	yellow
sdk	docker
app_port	7860
pinned	false

Civil Prep

A retrieval-augmented generation (RAG) system that answers UPSC current-affairs questions only from a curated set of source articles, with inline citations and a retrieval-based confidence score. Answer quality is continuously measured by an LLM-judge evaluation harness, and a GitHub Actions gate blocks any change that drops quality below a fixed threshold.

The Problem

Most RAG demos stop at "it answers questions." That's not enough for a study tool where a wrong or fabricated answer is worse than no answer. This project treats answer quality as a metric to protect, not a one-time check:

• The system refuses to answer when retrieval confidence is low, rather than guessing.
• Every answer is grounded with citations back to the source passage.
• An automated eval harness scores every answer for faithfulness (is every claim actually supported by the retrieved context?) and retrieval relevance (did the retriever surface the right passage at all?), and a CI gate fails the build if a prompt or pipeline change regresses either score below threshold. A demo chatbot can silently get worse after a code change; this pipeline can't.

Architecture

flowchart LR
    A["data/raw/*.txt"] -->|"ingest.py"| B["Chunk + tag<br/>source/date/category"]
    B -->|"embed_finetune.py"| C["Fine-tuned<br/>sentence-transformer"]
    B -->|"embed_store.py"| D[("Chroma<br/>vector store")]
    C --> D
    Q["User question"] --> E["Retrieve top-k<br/>chunks"]
    D --> E
    E -->|"confidence < 0.5"| F["Refuse:<br/>not enough information"]
    E -->|"confidence >= 0.5"| G["Gemini: answer<br/>from context only"]
    G --> H["Cited answer"]
    F --> H
    H --> I["eval.py: LLM judge<br/>faithfulness + relevance"]
    I -->|"avg F1 < 0.70"| J["CI fails, blocks merge"]
    I -->|"avg F1 >= 0.70"| K["CI passes"]

Loading

How to run locally

# 1. Clone and set up the environment
git clone <this-repo>
cd upsc-rag-assistant
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# 2. Configure your API key
cp .env.example .env
# edit .env and set GEMINI_API_KEY (free, no card: https://aistudio.google.com/app/apikey)
# gemini-flash-lite-latest is the default model. It has a much larger free
# daily quota than gemini-flash-latest, which currently aliases to a
# preview model capped at ~20 free requests/day.

# 3. Add source articles
# Drop .txt files into data/raw/, named like:
#   gs3-economy_rbi-monetary-policy-inflation_2026-08-01.txt
# (a small sample set is already included for testing)

# 4. Build the index
python src/ingest.py          # chunk + tag -> data/chunks/chunks.json
python src/embed_finetune.py  # fine-tune embeddings -> models/upsc-embeddings/
python src/embed_store.py     # embed + index -> chroma_db/

# 5. Ask a question
python src/ask.py "What is the RBI's CPI inflation target band?"

# 6. Run the evaluation gate
python src/eval.py            # writes eval_report.json, exits non-zero if avg F1 < 0.70

# 7. Run tests
pytest tests/

# 8. Run the API + frontend
uvicorn app:app --reload          # FastAPI on http://localhost:8000 (see /health, /ask)
streamlit run frontend.py         # Streamlit UI on http://localhost:8501

Eval results

Latest run against all 25 questions in eval_questions.json (threshold: F1 ≥ 0.70):

Metric	Score
Avg. faithfulness	1.00
Avg. retrieval relevance	1.00
Avg. F1	1.00
Result	✓ Passed

4 of the 25 questions were correctly refused (confidence below the 0.5 threshold) rather than answered with a guess. Refusals are scored as faithful (an honest "I don't have enough information" is never a fabrication) and were confirmed by the relevance judge to correspond to genuinely weak retrieval matches, not false negatives.

Full per-question detail (retrieved sources, confidence, and both judge scores for every question) is in eval_report.json after running python src/eval.py, and is also uploaded as a CI build artifact on every GitHub Actions run.

Why RAG matters

The app (/ask and the Streamlit UI) shows the RAG answer side by side with a raw, no-retrieval Gemini answer to the same question (answer_without_rag() in src/ask.py). This comparison exists specifically to prove retrieval adds real value on questions about content Gemini has no way of knowing (recent, specific current-affairs facts), not just to assert it.

Daily Mains Practice

Below the main Q&A section, the app offers UPSC Mains-style practice questions (src/practice.py, GET /daily-topic). Each request picks a random article from the existing indexed articles in data/raw/ and generates a fresh question from it, so tapping "Give me today's topic" again gives you a new topic rather than repeating the same one. Question phrasing is guided by real, current UPSC Mains question patterns via few-shot examples in the prompt (FEW_SHOT_EXAMPLES). This is prompt engineering, not a separately trained model.

Submitted answers are graded (POST /grade-answer) strictly against that question's source article, not general knowledge: the response includes a score, what the student got right, what they missed, anything that contradicts the article, and specific improvement tips. This is AI-generated practice feedback grounded in one article: a self-check tool, not a substitute for examiner evaluation.

Why the CI gate matters

.github/workflows/eval.yml runs src/eval.py on every push and pull request to main. eval.py exits with a non-zero status if the average F1 score (the harmonic mean of faithfulness and retrieval relevance across all 25 eval questions) drops below 0.70, which fails the workflow and blocks the merge. This means a change that looks fine in a quick manual test, a tweaked prompt, a different chunk size, a new embedding model, can't silently make answers worse or more prone to hallucination without the pipeline catching it first. The eval-report artifact uploaded on every run also gives a per-question breakdown for debugging regressions.

Free deployment guide

Backend (FastAPI) on Render's free tier:

1. Push this repo to GitHub.
2. In Render, choose "New +" -> "Blueprint" and point it at the repo, it will pick up render.yaml automatically (Docker-based web service, free plan, health check at /health).
3. Set the GEMINI_API_KEY secret in the Render dashboard (it's marked sync: false in render.yaml so it's never committed).
4. Deploy. The Dockerfile builds the chunk/embed/index pipeline at image build time, so the deployed container ships with a ready-to-query Chroma store, no separate data step needed.

Frontend (Streamlit) on Streamlit Community Cloud or Hugging Face Spaces:

1. Point either platform at this repo with frontend.py as the entry point.
2. Set GEMINI_API_KEY as a secret in the platform's dashboard.
3. Both platforms install from requirements.txt automatically.

Both tiers are free with no credit card required.

Tech stack

• LLM: Google Gemini (google-generativeai), free tier, all calls go through a single call_llm() function in src/utils.py so the provider can be swapped without touching the rest of the codebase.
• Vector store: Chroma, embedded/local, no separate hosting cost.
• Embeddings: sentence-transformers (all-MiniLM-L6-v2), fine-tuned on question/chunk pairs from eval_questions.json, with an automatic fallback to the base model if there's too little training data.
• Orchestration: LangChain for chunking and the Chroma retriever.
• API: FastAPI (app.py).
• Frontend: Streamlit (frontend.py).
• Eval: an LLM-as-judge harness (src/eval.py) scoring faithfulness and retrieval relevance, combined into an F1-style score.
• CI/CD: GitHub Actions (.github/workflows/eval.yml), free for public repos.
• Deployment: Render free web service (backend) + Streamlit Community Cloud or Hugging Face Spaces (frontend), both free.