Active Learning, Behavioral-cloning and Evolution-based Reinforcement for Thrifty Agents
Sparse expert supervision for league-based reinforcement learning, with response-aware bi-objective acquisition.
ALBERTA is a research project studying how a very small amount of human expert supervision can be allocated during reinforcement learning.
The experimental environment is Atomic Chess. A neural agent is first pretrained by behavioral cloning, then trained through league-based self-play with PPO. Starting from a shared RL checkpoint, different branches receive the same expert-intervention budget but differ in how annotated positions are selected.
The main experiment studies a response-aware acquisition strategy: instead of selecting positions only from uncertainty statistics, ALBERTA measures how expert annotations locally change the learner and models two response signals:
- policy displacement, measured by KL divergence;
- absolute value displacement,
|ΔV|.
Predicted responses are then used as a bi-objective Pareto acquisition criterion.
ALBERTA does not acquire expert labels directly from uncertainty. Instead, it uses pre-annotation signals to predict how strongly the current learner would respond to supervision, separately for policy and value, then allocates a sparse annotation budget on the resulting Pareto front.
The full RL10 candidate population is projected into predicted learner-response space. The red boundary is the first non-dominated front, and the 121 selected states are retained by crowding-distance truncation to preserve coverage across policy- and value-responsive regions.
Expert feedback is potentially valuable in reinforcement learning, but it is also expensive. When only a tiny number of interventions is available, the problem is no longer simply how to learn from expert supervision, but where that supervision should be spent.
Atomic Chess provides a compact testbed for this question: self-play can generate large amounts of experience cheaply, while tactical positions can produce sharp disagreements between policies, value estimates, and expert judgment. It is also a practical setting for human-in-the-loop experimentation: the author has substantial prior experience playing Atomic Chess, allowing expert annotations to be provided directly without relying on an external annotation pipeline.
ALBERTA therefore treats expert supervision as a scarce resource. Rather than asking only where the learner is uncertain, the project asks a complementary question: where is an expert intervention expected to change the learner the most?
Atomic Chess is a chess variant in which every capture triggers an explosion centered on the destination square. The capturing piece, the captured piece, and all pieces on the eight adjacent squares are removed; adjacent pawns are immune unless they are directly involved in the capture. A king cannot make a capture that would cause it to explode, and the game is won either by checkmate or by exploding the opposing king. One unusual consequence of these rules is that the two kings may legally occupy adjacent squares. More generally, explosions take precedence over several standard chess constraints, including checks and pins.
These rules fundamentally change the tactical structure of the game: local captures can have non-local consequences, material value becomes highly context-dependent (see Inferring Piece Value in Chess and Chess Variants by Steven Pav), and many positions contain forced tactical sequences that differ sharply from those of standard chess. Despite these changes, Atomic Chess remains a deterministic, fully observable, two-player Markov game, making it a convenient controlled environment for reinforcement-learning experiments.
The experimental pipeline is:
Lichess Atomic Chess games
│
▼
Behavioral Cloning
│
▼
League-based PPO self-play
│
▼
RL10
│
├───────────────┬──────────────────┐
│ │ │
▼ ▼ ▼
PPO only Random acquisition Pareto acquisition
│ │
│ 200 calibration probes
│ │
│ local learner response
│ (ΔKL, |ΔV|)
│ │
│ response prediction
│ │
│ 2-D Pareto selection
│ │
└────────┬─────────┘
│
121 expert interventions
│
▼
RL20
The intervention budget is approximately 0.01% of the ~1.2M self-play observations collected before the branch point: 121 expert annotations per supervised treatment.
The Pareto treatment additionally uses 200 expert-annotated calibration probes to learn the local response model. These calibration annotations are separate from the 121-position intervention budget.
Three final policies are compared:
| Policy | Training after RL10 |
|---|---|
| RL | PPO self-play only |
| Random | PPO + 121 randomly selected expert interventions |
| Pareto | PPO + 121 response-aware Pareto-selected expert interventions |
Each pair was evaluated over 500 games at four action-sampling temperatures.
| Temperature | Pareto score |
|---|---|
| 0.25 | 63.8% |
| 0.50 | 58.3% |
| 1.00 | 53.7% |
| 2.00 | 55.1% |
Pareto finishes ahead of the PPO-only control at all four tested temperatures in this controlled training run.
| Temperature | Pareto score |
|---|---|
| 0.25 | 77.4% |
| 0.50 | 62.3% |
| 1.00 | 56.8% |
| 2.00 | 50.3% |
The difference is strongly temperature-dependent and becomes small at high sampling temperature.
These results are from one controlled training seed (42). The 500-game evaluations reduce match-level sampling noise, but they are not independent training replications. The experiment should therefore be interpreted as a proof of concept rather than evidence of multi-seed or cross-domain robustness.
For each calibration annotation, ALBERTA starts from the same RL10 learner and performs one controlled Oracle-only optimization step.
The learner is reset before every intervention.
Two local response magnitudes are measured:
ΔKL policy displacement after the expert update
|ΔV| absolute value-function displacement
These quantities measure how much the learner changes, not whether the local update is beneficial.
An ExtraTrees multi-output regressor predicts both responses from contemporary learner-dependent features:
H policy entropy
U historical-league value disagreement
HU interaction between H and U
The historical uncertainty estimator uses trained historical RL critics and the current critic; behavioral-cloning models with untrained value heads are excluded.
Candidate positions are ranked using exact two-dimensional Pareto non-dominance and crowding over predicted (ΔKL, |ΔV|). The final intervention queue contains 121 positions from the first Pareto front.
The downstream tournament, rather than the local response magnitude itself, determines whether the resulting training intervention was useful.
All final branches share:
- the same RL10 branch point;
- the same neural architecture;
- the same PPO/self-play setup;
- the same master seed (
42); - the same 121-position intervention budget for supervised treatments.
Internal trajectory checks at epochs 10, 15 and 20 show substantial learning in all three branches. The final comparison therefore does not rely on a regressing PPO control or a collapsed Random branch.
No completed training comparison against the earlier uncertainty-score acquisition condition is reported. Likewise, the project does not currently include multi-seed training replication, policy-only/value-only response ablations, scalarization-vs-Pareto ablations, or annotation-budget sweeps.
AL/
├── HMI/ Human Oracle annotation interface
├── pareto/
│ ├── sample_pareto_probe.py Calibration-probe sampling
│ ├── mesure_local_response.py
│ └── select_pareto.py Response model + Pareto acquisition
├── estimate_active_learning_weights.py
└── seed_oracle_queue.py
analysis/ Optional scientific diagnostics
evaluation/ BC/RL tournaments and evaluations
making_dataset/ Lichess → BC dataset pipeline
src/ Models, agents, self-play and RL utilities
training/
├── train_bc.py
├── train_rl.py
└── train_al.py
data/
├── queue/ Versioned expert annotation queues
└── pareto/ Versioned calibration responses/results
lichess_bot/ Lichess deployment integration
Large datasets and model checkpoints are intentionally not stored in Git.
The behavioral-cloning policy uses a residual convolutional network over an 8 × 8 Atomic Chess board representation.
The final BC architecture uses:
Input channels: 19
Residual channels: 32
Residual blocks: 4
Action space: 20,160
Parameters: ~11.5M
RL wraps the pretrained policy backbone in an Actor-Critic model and trains it through league-based PPO self-play.
The action vocabulary is fixed across BC, RL, datasets and checkpoints for compatibility.
Behavioral cloning is trained from rated Atomic Chess games from the Lichess database.
The local pipeline produces:
data/processed/positions_2300.jsonl
data/processed/positions_2300_bc.jsonl
The final BC dataset is approximately 340 MB and is therefore not committed to Git.
It can be constructed using:
python making_dataset/extract_positions.py
python making_dataset/build_bc_dataset.pyHistorical self-play statistics used by the active-learning analyses are likewise kept outside Git because of their size.
The small expert-annotation queues and Pareto calibration artifacts used by the final experiments are versioned under data/queue/ and data/pareto/.
The repository includes the exact small artifacts defining the supervision experiments:
data/queue/oracle_queue_1-10_random.jsonl
data/queue/oracle_queue_1-10_AL.jsonl
data/queue/oracle_queue_1-10_pareto_probe.jsonl
data/queue/oracle_queue_dynamic_pareto_10.jsonl
data/pareto/pareto_local_responses.jsonl
data/pareto/pareto_selection_10.json
oracle_queue_1-10_pareto_probe.jsonl contains the 200 manually annotated calibration interventions.
oracle_queue_dynamic_pareto_10.jsonl contains the final 121-position Pareto intervention queue.
The final response-aware acquisition pipeline consists of exactly three stages:
python AL/pareto/sample_pareto_probe.py
python AL/pareto/mesure_local_response.py
python AL/pareto/select_pareto.pyConceptually:
historical RL states
│
▼
200 calibration positions
│
▼
human expert annotations
│
▼
controlled one-step learner responses
│
├── ΔKL
└── |ΔV|
│
▼
ExtraTrees response model
│
▼
refresh H/U/HU at RL10
│
▼
predict responses over candidate pool
│
▼
2-D Pareto selection
│
▼
121 expert interventions
Behavioral cloning:
python training/train_bc.py --helpLeague PPO:
python training/train_rl.py --helpSparse expert-supervised training:
python training/train_al.py --helpThe training scripts expose their experimental configuration through their command-line interfaces. Checkpoints are written locally under checkpoints/ and are ignored by Git.
RL/AL checkpoints can be evaluated with:
python evaluation/tournament_rl.py --helpBC checkpoints can be evaluated with:
python evaluation/tournament_bc.py --helpThe reported final experiment uses action temperatures:
0.25, 0.5, 1.0, 2.0
with 500 games for each pairwise matchup at each temperature.
analysis/ contains diagnostics developed during the project, including:
- uncertainty progression and reward-signal analysis;
- annotation-response analysis;
- queue-composition analysis;
- calibration-probe coverage;
- trained-policy shift;
- PPO/Oracle gradient-alignment diagnostics;
- reference-gradient stability.
These scripts are supplementary diagnostics and are not all required to reproduce the main training experiment.
ALBERTA is a single-domain research prototype built around Atomic Chess.
The project investigates whether learner response to supervision can provide a useful acquisition signal under an extremely sparse expert-intervention budget.
The current evidence supports this idea in one controlled training run. It does not establish that the method is universally superior to uncertainty sampling, nor that the observed effect generalizes across random seeds, RL algorithms, games, or other domains.
Atomic Chess game data is derived from the public Lichess database.
The Lichess deployment integration builds on the lichess-bot project; see the files and license under lichess_bot/ for upstream attribution.
Original ALBERTA code is released under the MIT License. See LICENSE.
This repository also contains third-party software and assets under
separate licenses, notably the AGPL-3.0-or-later lichess-bot integration
and the GPL-2.0-or-later Cburnett chess piece set.
See THIRD_PARTY_NOTICES.md for attribution and licensing details.