renderers <-> v1: serializable RenderedTokens

[snimu]

Description

1. RenderedTokens is a plain @DataClass (defined at renderers/base.py:159), which json.dumps cannot serialize.
2. PR feat(renderer-client): give per-token prompt attribution to TrajectoryStep #1414 puts it in state. In verifiers/utils/response_utils.py:131-133, parse_response_tokens writes the raw RenderedTokens dataclass onto the step:

if prompt_attribution is not None:
    out["prompt_attribution"] = prompt_attribution
    tokens.prompt_attribution = None

3. out is the TrajectoryStepTokens dict that MultiTurnEnv.add_model_response then appends to state["trajectory"] (verifiers/envs/multiturn_env.py:151-162).
4. The v1 path then asserts JSON-serializability of the entire state:
   • verifiers/v1/harness.py:184: state.assert_serializable() after rollout completion
   • verifiers/v1/env.py:123: state.assert_serializable() after group run
   • Both eventually call json.dumps(self) on the whole state (verifiers/types.py:898-902), traversing state["trajectory"][i]["tokens"]["prompt_attribution"].

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Test improvement

Testing

• All existing tests pass when running uv run pytest locally.
• New tests have been added to cover the changes

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

Note

Low Risk
Localized serialization change in token parsing with tests; only consumers expecting a RenderedTokens object on trajectory steps need to rehydrate from dict.

Overview
parse_response_tokens now stores renderer prompt_attribution on trajectory step tokens as a JSON-safe dict (dataclasses.asdict), stripping nested multi_modal_data from that dict because tensors already live on out["multi_modal_data"]. The sidecar is still moved off ResponseTokens (not duplicated), matching multi_modal_data policy.

Callers that read tokens["prompt_attribution"] get a dict instead of a RenderedTokens instance; docs note rehydration via RenderedTokens(**d). Tests assert dict equality, dict-shaped truncation fields, and a v1 regression that State.assert_serializable() succeeds when attribution is on a trajectory step.

^{Reviewed by Cursor Bugbot for commit b2b4cbf. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Serialize prompt_attribution as a plain dict in parse_response_tokens

• parse_response_tokens in response_utils.py now converts a RenderedTokens dataclass to a plain dict via dataclasses.asdict, clearing multi_modal_data before conversion to avoid non-serializable values.
• The dict is rehydratable via RenderedTokens(**d). The original tokens.prompt_attribution is still cleared (move semantics preserved).
• A new regression test verifies that a trajectory step with prompt_attribution passes State.assert_serializable.
• Behavioral Change: callers that previously received a RenderedTokens instance from prompt_attribution will now receive a plain dict.

^{Macroscope summarized b2b4cbf.}

[macroscopeapp]

Approvability

Verdict: Needs human review

Unable to check for correctness in b2b4cbf.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7eba25d. Configure here.}