[MCERQUA]

Hi Brad, here's the PR you asked for on the call, from Mike's team. Our agents prepared it, and separate review passes checked it before we opened it. Reading through rulesmith we found one real bug, in how calibration and .confidence interact. This PR fixes only that.

What's wrong

Calibration and the runtime disagree about what <judgment>.confidence is, in two places:

1. A fitted temperature never reaches .confidence. calibrated() (runtime.py) reshapes a choice's or a score's probabilities but leaves answer["confidence"] at the value the judge first reported. The confidence node reads exactly that value. So for choice questions a fitted temperature can't change what a .confidence rule decides, while docs/rules.md presents calibration as what makes such a threshold meaningful. Example: threat answers ATTACK at 0.9 vs 0.1, and threat.confidence reads 0.80 both uncalibrated and at calibrated 5. Only the reported prediction.confidence moves.
2. Cutoffs on a confidence node are fitted in the wrong number space. observed() (calibrate.py) collects sureness(), the probability of the chosen option, as the values a line can sit between. The runtime compares the reported confidence, which for a choice is (p - 1/K) / (1 - 1/K). On two options p = 0.68 reads 0.36. So for a choice question every candidate line is in the wrong space, worst on 2-option questions. Lines on noul and score values were already consistent.

The fix

• calibrated() now recomputes confidence from the tempered probabilities. A score uses score_confidence. A choice uses the choice_confidence formula on the option it chose. That gives the same number whenever a judge chose its likeliest option. When it chose another option the number is lower, down to 0 once the chosen option falls below 1/K. At temperature 1 nothing changes.
• observed() now collects the reported confidence, the number the rule actually compares. A noul answer has no confidence, so its confidence node is absent, same as at runtime.
• One clause added in docs/rules.md saying threat.confidence moves too.

Another option was to make .confidence return the tempered probability of the chosen option. We didn't do that because it would change what every existing .confidence >= x line means. Happy to switch if you prefer it.

Two things worth knowing:

• Different confidence formulas. This assumes a judge's reported confidence follows the published formula, as the comment in chat_judge.py says the hosted judge's does. If a judge reports something else, the recomputed value replaces it as soon as a temperature other than 1 is set, which rulesmith calibrate does for nearly every choice question it fits. For such a judge, calibrated 1.0001 reads differently from no temperature at all.
• Scores. A score that carries a temperature now has its confidence, and so prediction.confidence, move with it. calibrate only fits temperatures for choices, so this only affects hand-written score temperatures.

Before / after

A constructed 41-example task: a 2-option judge whose confidence follows the published formula, and a threshold on route.confidence starting at 0.95. Accepted requests read 0.42–0.90, reviewed ones 0.10–0.36.

	fitted line	right
main (6a0564b)	0.95 → 0.555	33 / 41
this branch	0.95 → 0.39	41 / 41

threat.confidence >= 0.7 with calibrated 5: on main it reads 0.80 and the rule fires. On this branch it reads 0.22 and the decision falls through to always.

Tests

New tests in tests/test_calibrate.py:

• a property test that after tempering, confidence is recomputed from the tempered probabilities, for choices and scores, and that an answer whose chosen option falls below 1/K reads 0;
• the end-to-end threat.confidence rule above;
• the 41-example cutoff fit. It fails on main with the line at 0.555.

No existing test was changed. Full suite, on macOS, Python 3.12, core deps plus the bench/chess extras (no vizdoom, so the doom/arena modules aren't collected), live and paid markers deselected:

	passed	failed	skipped
main	586	1	3
this branch	589	1	3

The one failure is the same on both: test_invalid_opponent_is_rejected_before_judge_credentials imports test_arena, which pulls in rulesmith.doom and needs vizdoom. ruff check and ruff format --check are clean.

Thanks for building this and for asking us to look. We kept this PR to the one bug. We also have a few small, separate suggestions (docs and defaults) and can send them as their own PRs if you want them.

🤖 Generated with Claude Code