CSE 476 Final Project

JSON Submission

See submission_json directory

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Create a .env file in the repo root with:

OPENAI_API_KEY=your_key_here

You must be on the ASU network or VPN to reach the API.

Verify setup

cd src
python verify_connection.py

Should print '45' as the response.

Running

All commands run from src/.

# Run a strategy on the first 5 questions
python run.py --strategy baseline --limit 5

# Run on all questions
python run.py --strategy cot

Available strategies

Strategy	Description
baseline	Direct single-shot answer
cot	Chain of Thought prompting
self_consistency	Majority vote across 3 samples
self_refinement	Draft then review/refine
react	Thought/Action/Observation loop with tools (calculator, python, search, wikipedia)
tool_augmented_reasoning	Calculator-augmented reasoning for math
decomposition	Break problem into sub-questions
tree_of_thought	Propose N approaches, expand each, judge for best
debate	N debaters answer independently, then critique each other and revise; judge picks best

Flags

# Shuffle question order (default seed 42)
python run.py --strategy react --shuffle

# Custom seed
python run.py --strategy react --shuffle --seed 123

# Filter to a single domain
python run.py --strategy react --domain math
python run.py --strategy cot --domain common_sense

# Choose between the dev set questions and the test set
python run.py --strategy cot --questions dev
python run.py --strategy tree-of-thought --questions text

# Choose a starting question index
python run.py --strategy cot --questions dev --start 200

# Disable LLM-as-a-judge fallback grading
python run.py --strategy cot --limit 10 --no-llm-judge
python run.py --strategy cot --questions dev --start 200 --limit 20

Valid domains: math, common_sense, coding, planning, future_prediction

Scoring (already done by default, this is for rescoring/testing)

# Score an existing predictions file
python score.py predictions.json

# Regrade with LLM judge
python score.py predictions.json --regrade --llm-judge