[sergiopaniego]

What does this PR do?

grpo_wordle.py and its notebook keep one OpenEnv client for the whole run. A Space closes WebSockets that stay open for hours (about every 2 hours in our runs), and the Wordle run takes about 2 hours 15 minutes. If the drop hits a game in progress, the clients up to OpenEnv 0.11.0 reconnect to a new session and the next step() plays on a board that was never reset, with no error. Since 0.12.0 they raise SessionLostError instead, so the game fails and counts as a loss.

This PR opens a new client at the start of every game, as the Sudoku example does (#8097), so no connection lives long enough to be dropped. The client is .sync(), so close() really closes the previous one. Without it, close() on the async client returns a result that is never run and each game leaves a connection and an event-loop thread open (4 resets leave 5 threads on 0.10.0, 1 with .sync()).

grpo_multi_env.py (and its snippet in openenv.md) already opened a new client per game, but without .sync(), so its close() left the event-loop thread behind. It now uses .sync() as well.

Follow-up from Albert's review of #8097. The same change for the OpenEnv Wordle tutorial script is in openenv/OpenEnv#1560.

Before submitting

• Did you read the contributor guideline?

---

Note

Low Risk
Changes are limited to example scripts, notebook, and documentation around OpenEnv connection handling; core TRL training code is untouched.

Overview
Wordle GRPO (grpo_wordle.py and the notebook) now closes and recreates the OpenEnv client on every reset() so no WebSocket lives long enough for a Space’s periodic disconnect to break an in-progress game. Clients are created with .sync() so close() actually tears down the connection instead of leaking threads.

The multi-env example and openenv.md snippet apply the same .sync() fix for Wordle and Catch clients that were already recreated per game.

Docs and setup instructions bump MAX_CONCURRENT_ENVS from 64 to 96 to cover 64 parallel games plus sessions still closing between games.

^{Reviewed by Cursor Bugbot for commit 2ab0de6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

Thanks, this matches the Sudoku pattern and the .sync() change is right for multi-env too. A few nits:

• docs/source/openenv.md: the WordleEnv snippet in the Wordle section (~L270) still keeps one client for the whole run. Could it get the same reset() change, so it matches the script?
• grpo_wordle.py/notebook reset(): Sudoku has a one-line comment linking to https://huggingface.co/docs/trl/openenv#long-runs-against-a-space. Could Wordle use the same comment, so the copies stay aligned?
• Notebook and openenv.md say MAX_CONCURRENT_ENVS=96 with no reason. The script, plus the Sudoku notebook and README, explain it (64 games plus the sessions being closed). Worth the same short note?

[albertvillanova]

Thanks for adding the comment! Two points from my previous review are still open:

• docs/source/openenv.md: the WordleEnv snippet (L258) still keeps one client for the whole run, so it doesn't match the script.
• Notebook and openenv.md: MAX_CONCURRENT_ENVS=96 still has no reason (64 games plus the sessions being closed).