[jimmy-holdout]

Fixes #1551

Summary

There are three commits, one per environment in the issue. qed_math_env no longer sends the answer-mode reference solution to the agent. reasoning_gym_env and finqa_env stop scoring once the episode has ended. Later step() calls return the episode's final observation again, with the same reward, until reset().

• qed_math_env: reset() leaves ProblemObservation.reference_solution empty and get_problem leaves it out in both modes. Proof mode already worked this way. Grading still reads the reference from the server-side problem and the gold cache. The docstrings (the class, reset(), mcp_server.py), the reference_solution field description and the README now say the reference is not sent.
• reasoning_gym_env: the observation of the first scored answer is kept, and later step() calls return a copy of it until reset(), without scoring or counting a step. The episode reward therefore stays the scored answer's reward. That holds for the session's default verify(), which reads the last step, for _resolve_env_reward, which reads the last non-None tool reward, and for any loop that reads the last step. The existing "no current entry" branch is unchanged.
• finqa_env: step() and step_async() keep the terminal observation when an answer is submitted or max_steps is reached, and reset() clears it. Any later step, including ListToolsAction, returns a copy of it: no tool runs, nothing is scored, and step_count does not change. A submission on the last allowed step is still scored. ground_truth stays in the submit observation, which now comes only at the end of the episode.

Behaviour changes for callers:

• Code that read reference_solution from the qed_math_env reset observation or the get_problem payload in answer mode now gets "" or no key.
• In reasoning_gym_env and finqa_env, a step() after done now gets the final observation again instead of a new score or a tool result. This applies to the step() / WebSocket path. The production /mcp route calls tools directly and never issued a step reward.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

Alignment Checklist

Before submitting, verify:

• I have read PRINCIPLES.md and this PR aligns with our principles
• I have checked INVARIANTS.md and no invariants are violated
• I have run the lint hook (lint.sh) and the tests, and addressed all issues

lint.sh stops at its ruff format --check src/ tests/ envs/ step on main too. It lists the same 47 files on main and on this branch: 46 under envs/ and src/openenv/core/harness/README.md. Three of them are READMEs edited here (qed_math, reasoning_gym, finqa). In those, ruff flags code blocks that this PR does not change. usort check reports only the two known files. ruff format --check on the changed Python files, and ruff check src/ tests/, pass. scripts/sync_env_docs.py --check passes. The three doc stubs were regenerated with --fix.

RFC Status

• Not required (bug fix, docs, minor refactoring)
• RFC exists: #___
• RFC needed (will create before merge)

Test Plan

New tests. The ones that check the fix fail on the parent commit and pass on their own commit (qed 2, reasoning_gym 2, finqa 6). The honest-path controls pass on both.

• tests/envs/test_qed_math_environment.py: test_answer_mode_hides_reference_solution (reset observation and get_problem) and test_auto_detected_answer_mode_hides_reference_solution (a row without a rubric). Control: test_answer_mode_correct_answer_still_scores (\boxed{4} still gets reward 1.0). test_answer_mode_includes_reference_solution is replaced.
• tests/envs/test_reasoning_gym_environment.py: wrong, then the revealed answer keeps reward 0.0; correct, then an extra step keeps 1.0. Control: reset() scores the next question normally. This file importorskips reasoning_gym, which the CI job does not install, so these tests run only where reasoning-gym is installed.
• tests/envs/test_finqa_environment.py: new TestEpisodeEnd, with synthetic data like TestResetSeed, so it runs in CI. Fix tests cover a submit after a submit, a submit after max_steps, and a ListToolsAction after a correct submit (reward stays 1.0 and step_count is unchanged), each through step() and step_async(). Controls: a submit on the last allowed step (1.0) and reset().

Runs:

• The CI test command from test.yml, run on a fresh clone with the CI job's extra packages plus reasoning-gym (Python 3.13): 4344 passed, 110 skipped on this branch, and 4331 passed, 110 skipped on main. The touched test files: 153 passed for qed (5 integration tests deselected), 30 passed for reasoning_gym, and 79 passed, 16 skipped for finqa. With the downloaded FinQA data linked at envs/finqa_env/data, all 95 finqa tests pass, including the existing data-dependent TestEnvironment.
• The repro scripts from the issue, before and after. Each env was run in-process under uvicorn. The reasoning_gym_env and finqa_env images were also built from their Dockerfiles and gave the same rewards. The finqa image was driven with raw /ws messages.

Case	main	this branch
qed: reference in reset observation / get_problem	full worked solution	"" / absent
qed: honest \boxed{66} / wrong \boxed{-12345}	1.0 / 0.0	1.0 / 0.0
reasoning_gym: step("x"), then the revealed answer	0.0, then 1.0	0.0, then 0.0 (not rescored)
reasoning_gym harness: wrong, then revealed (verify() / resolved reward)	1.0 / 1.0	0.0 / 0.0
reasoning_gym harness: correct, then one extra call (verify() / resolved reward)	0.0 / 0.0	1.0 / 1.0
finqa: submit_answer("0"), then the returned ground_truth	0.0, then 1.0	0.0, then 0.0 (not rescored)
finqa: submit after FINQA_MAX_STEPS=3 ended the episode	1.0	0.0 (not rescored)
finqa: honest first submission	1.0	1.0

---

Note

Medium Risk
Changes evaluation and training reward semantics plus the qed_math observation/MCP contract; callers that relied on answer-mode references or post-done steps will see different behavior.

Overview
Fixes reward and leakage bugs across three environments (#1551).

qed_math_env no longer exposes the reference solution in reset(), ProblemObservation, or get_problem (including answer mode). The field stays empty and the MCP payload omits it; grading still uses the server-side problem and gold cache. Docs and model field descriptions match this contract.

reasoning_gym_env and finqa_env cache the terminal observation when the episode ends (first scored answer / single step, or submit_answer / FINQA_MAX_STEPS). Later step() / step_async() calls return a copy of that observation with the same done and reward, without running tools, incrementing step_count, or rescoring—so harnesses that read the last step no longer get inflated rewards from a second submit or the revealed correct answer.

Breaking for integrators: answer-mode reference_solution consumers get "" or no key; post-done step() no longer produces new scores or tool results until reset(). New tests cover these paths for all three envs.

^{Reviewed by Cursor Bugbot for commit 9920bc0. Bugbot is set up for automated code reviews on this repo. Configure here.}