Competitive-MARL-Racing

A Multi-Agent Reinforcement Learning (MARL) system where autonomous racing agents learn competitive driving strategies — drafting, blocking, and overtaking — entirely through self-play. No hand-coded tactics. No scripted behavior. Just emergent intelligence forged through competition.

---

Overview

Standard autonomous driving AI is trained to be safe, predictable, and rule-abiding. This project takes a different approach.

We built a racing system where agents must win in a zero-sum, high-speed environment against opponents that are constantly improving. The core challenge is not just driving fast, but learning when to draft, when to block, and how to adapt strategy based on the opponent. The result is a framework that tackles three hard problems in modern AI simultaneously: continuous control, non-stationarity, and reward shaping.

Features

• Custom Racing Environment: Built on highway-env's racetrack-v0 with a multi-agent wrapper providing clean per-agent observations, rewards, and actions via Gymnasium.
• PPO From Scratch: Proximal Policy Optimization implemented without high-level RL libraries. Full Actor-Critic architecture with GAE, clipped surrogate objective, and minibatch updates.
• Self-Play League Training: Agents train by competing against a pool of their own previous snapshots, continuously raising the skill ceiling.
• Agent Archetypes: Distinct behavioral profiles (Balanced, Speedster, Blocker, Drifter) created through modular reward shaping, forcing the learning agent to adapt its tactics per opponent.
• Modular Reward System: Reward components (speed, lane centering, collision penalty, drafting bonus, blocking, overtaking, progress) are individually weighted and toggleable for ablation studies.
• Elo Evaluation: Skill progression is quantified via Elo ratings across training generations and a final round-robin Grand Prix tournament.
• Checkpoint-Based Curriculum: Agents learn track navigation progressively — starting with short track segments and extending as they improve.

Tech Stack

• Environment: highway-env + Gymnasium
• RL Algorithm: PPO (from scratch, PyTorch)
• Multi-Agent Training: Custom self-play league with opponent pool sampling
• Logging: TensorBoard
• Evaluation: Custom Elo rating system + round-robin tournament
• Language: Python 3.11+

Getting Started

# Clone the repo
git clone https://github.com/sebcheung/Competitive-MARL-Racing.git
cd Competitive-MARL-Racing

# Install dependencies
pip install -r requirements.txt

# Verify environment setup
python scripts/verify_env.py

# Run tests
python -m pytest tests/ -v

Usage

# Train base agent with curriculum checkpoint rewards
python src/rewards/checkpoint_reward.py --timesteps 200000 --n-envs 4

# Train archetype agent via self-play
python scripts/train.py --archetype speedster \
    --warmup-checkpoint checkpoints/best_racer.pt \
    --timesteps 100000 --lr 1e-5 --n-epochs 1

# Watch a trained agent race
python scripts/watch_racer.py --checkpoint checkpoints/best_racer.pt --episodes 5

# Record demo videos
python scripts/record_demo.py --checkpoint checkpoints/speedster.pt --episodes 3

# Run round-robin Grand Prix tournament
python scripts/grand_prix.py checkpoints/*.pt --episodes 10 --save-elo results/elo.json

# Measure lap times across agents
python scripts/lap_timer.py checkpoints/*.pt --episodes 10

# Visualize tournament results
python scripts/visualize_tournament.py --elo-file results/elo.json

# Visual sandbox with random-action agents
python scripts/sandbox.py --agents 2 --steps 300

Project Structure

├── src/
│   ├── agents/       # PPO agent, Actor-Critic network, rollout buffer
│   ├── env/          # Racing environment wrapper, configuration
│   ├── eval/         # Elo rating system, match runner, tournament
│   ├── rewards/      # Modular reward components, archetypes, checkpoint curriculum
│   └── training/     # Self-play loop, opponent pool, sampling strategies, logger
├── scripts/          # Training, evaluation, demo, and visualization entry points
├── tests/            # Unit and integration tests
├── checkpoints/      # Saved model weights and opponent pool
├── demos/            # Recorded demo videos
├── results/          # Tournament results, lap times
└── plots/            # Visualization outputs

Team

Sebastian Cheung, Filip Ilic, Max Mayer, and Raahil Gadhoke

License

This project is licensed under the MIT License. See LICENSE for details.