rl

Single-file RL algorithm implementations, practiced against Gymnasium benchmarks.

Setup

Conda env rl (Python 3.12) already provisioned. From scratch on a new machine:

bash scripts/setup_env.sh

Activate:

conda activate rl

Verify all benchmark envs load:

conda run -n rl python scripts/check_envs.py

Benchmarks

Env ids live in benchmarks.py, grouped by category. Import from there rather than hardcoding ids in algo files.

Category	Envs	Action space
Tabular	CliffWalking-v1, FrozenLake-v1, Taxi-v3	discrete, discrete obs
Classic control	CartPole-v1, Acrobot-v1	discrete
Classic control	Pendulum-v1, MountainCarContinuous-v0	continuous
Box2D	LunarLander-v3	discrete
Box2D	LunarLanderContinuous-v3, BipedalWalker-v3	continuous
MuJoCo	HalfCheetah-v5, Hopper-v5, Walker2d-v5, Ant-v5	continuous
Atari	ALE/Pong-v5, ALE/Breakout-v5	discrete, pixel obs

Suggested starter env per algo family is in benchmarks.STARTER.

Algo convention

Each algorithm is one self-contained file under algos/, e.g. algos/dqn.py, algos/ppo.py, algos/sac.py. A file should run standalone:

conda run -n rl python algos/dqn.py

Comparing algos

Each algo's train() saves per-episode returns via results.save_returns(algo, env_id, seed, returns) to results/<env_id>/<algo>__seed<seed>.npy. To compare algos that share a benchmark:

conda run -n rl python algos/sarsa.py --seed 0
conda run -n rl python algos/q_learning.py --seed 0
conda run -n rl python scripts/plot_benchmark.py --env CliffWalking-v1

Writes a smoothed multi-seed learning-curve overlay to results/plots/<env_id>.png. Run more seeds (--seed 1, --seed 2, ...) before plotting to average out noise. Colors are assigned per algo in results.ALGO_COLORS — add new algos there to keep their color stable across charts.

Notes

• gymnasium[mujoco] bundles MuJoCo itself — no separate license/binary needed.
• Atari ROMs: ale-py 0.12 expects .bin files inside its own package dir (ale_py/roms/), not wherever AutoROM downloads them. scripts/setup_env.sh handles the copy; see that file if ALE/* envs throw FileNotFoundError.