Python: [Bug]: Resuming from a checkpoint discards caller-supplied run kwargs, contrary to the documented contract #9850

Observed Behavior

Resuming a workflow from a checkpoint silently discards run kwargs supplied by the caller on that continuation, contrary to the documented contract.

_workflows/_workflow.py:627-636 stages and commits the run kwargs before initial_executor_fn(); _workflows/_runner.py:334-335 then does state.clear(); state.import_state(checkpoint.state), which restores the previous run's kwargs over them. The contract comment at _workflow.py:596-601 says the prior run's kwargs are preserved "unless the caller explicitly provides new kwargs", and cancel_pending_requests shows the intended order by restoring first (_workflow.py:1553-1561).

Real output from the reproduction below:

run 1 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]
entry checkpoint iteration: 0 state kwargs: {'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}
run 2 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]
EXPECTED run 2 kwargs key == NEW (caller-provided kwargs win on continuation)

The discarded kwargs are also re-persisted into later checkpoints, so the continuation permanently runs with the old value.

Expected Behavior

On run(checkpoint_id=..., function_invocation_kwargs={...}) (and the same for client_kwargs, and for checkpoint_id + responses), the caller-supplied kwargs should win over the checkpoint-restored ones, as the contract comment states. Executors reading WORKFLOW_RUN_KWARGS_KEY should see NEW.

Steps to Reproduce

1. Build a workflow with checkpoint storage whose executor records WORKFLOW_RUN_KWARGS_KEY from state.
2. Run 1 with function_invocation_kwargs={"key": "OLD"} and save the entry checkpoint id.
3. Rebuild the workflow on a fresh instance and call run(checkpoint_id=<entry>, function_invocation_kwargs={"key": "NEW"}).
4. Observe the executor sees OLD, and later checkpoints keep OLD.

Minimal Reproduction

The full runnable script is repro_kwargs_restore.py (pure Python, mock executor, in-memory checkpoint storage): it prints run 1's kwargs, the entry checkpoint state, and run 2's kwargs with the expected NEW. Happy to attach it to the issue.

Error Messages and Stack Traces

No exception. Actual output:

run 2 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]

Package Versions

agent-framework-core: 1.23.0 (main at 85f3fc6, 2026-10-29)

Python Version

Python 3.12.2

Operating System

Windows

Regression

Unknown

Additional Context

• Suggested direction: apply caller-supplied kwargs after the checkpoint state is restored (matching cancel_pending_requests), instead of committing them before initial_executor_fn().
• Searched open and closed issues/PRs: no matching issue or PR. Python: Preserve workflow run kwargs when continuing with run(responses=...) #4296 fixed responses-only continuation kwargs, Python: [Bug]: Nested workflow cancellation retains stale invocation kwargs #8601/Python: Forward invocation kwargs during nested workflow cancellation #8602 fixed nested-cancellation kwargs; Python: Support workflow entry-state updates and per-AgentExecutor run options #8711/Python: [Feature]: Make workflows stateless with caller-owned checkpoints and msgspec snapshots #8522 are vNext features that do not cover this v1 ordering bug.

AI assistance disclosure: analysis and the reproduction script were prepared with AI assistance (Claude); I personally ran the reproduction and verified the output above.

Acknowledgements: I searched existing issues and did not find a duplicate; I personally verified this behavior; I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.