Skip to content

test(e2e): deterministic H3 t2va 2-GPU flowGRPO metric to prevent regression - #217

Merged
Rockdu merged 1 commit into
radixark:mainfrom
Rockdu:h3-2gpu-e2e
Aug 30, 2026
Merged

test(e2e): deterministic H3 t2va 2-GPU flowGRPO metric to prevent regression#217
Rockdu merged 1 commit into
radixark:mainfrom
Rockdu:h3-2gpu-e2e

Conversation

@Rockdu

@Rockdu Rockdu commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

What

  • --deterministic-mode on the H3 t2va 2-GPU flowGRPO recipe, matching the SD3 recipes that back a metric-regression e2e.
  • A new tests/e2e/short/test_h3_t2va_grpo_2xGPU.py (stage-c-3-gpu-h200, 2 rollouts) with its recorded standard, so H3 gets the same bit-for-bit metric regression the other families already have.

CI sizing: the test runs the recipe at --n-samples-per-prompt 4 --eval-interval 0 (8 videos per rollout, no eval). The recipe's own sizes put one tp2 engine at ~60 min for two rollouts, past the suite's timeout; this config keeps both rollouts — so the LoRA-IPC weight sync between them stays covered — at 1492 s in CI.

Why

H3 is the family with the most model-specific rollout code (its own packed-sequence denoise loop, its own SDE update, its own LoRA-IPC weight key mapper), and no e2e guarded it — regressions there only surfaced on the 17/34-GPU production runs.

Validation

Recorded on H200 against main before #215/#216, with stock sglang-miles-h3, --deterministic-mode, 2 rollouts:

Metric rollout 0 rollout 1
rollout/reward/raw_mean 0.7477782 0.7937961
rollout/reward/raw_std 0.0313390 0.0288071
train/log_prob_mean_abs_diff 5.75e-05 4.13e-05

log_prob_mean_abs_diff lands in the e-5 band the H3 recipe is verified against, i.e. the train forward reproduces the rollout trajectory; reward moves in the expected direction across the two rollouts.

The same recipe was also recorded on top of #216 (SDE-window trajectory transport) with its sglang counterpart: all eight metric series came out bitwise identical to this baseline, which is the H3-path evidence that the windowed transport changes nothing observable. CI on this test then passed against the standard, so the run is reproducible across machines, not just locally.

Files

  • scripts/run_diffusion_grpo_h3_t2va_2gpu.py--deterministic-mode.
  • tests/e2e/short/test_h3_t2va_grpo_2xGPU.py + tests/ci/fixtures/e2e_standards/test_h3_t2va_grpo_2xGPU.json — the test and its standard.

Checklist

  • pre-commit run --all-files passes
  • Added/updated tests for new behaviour — this PR is the test
  • pytest -x is green — tests/fast, 239 passed
  • If launch flags changed, python3 train_diffusion.py --help still parses — no new flags
  • If a public flag was added, it appears in the CLI reference docs — n/a
  • If an example was added, it has a real walkthrough — n/a

@Rockdu Rockdu added the run-ci-e2e Run e2e metric-regression tests on this PR label Aug 29, 2026
@Rockdu
Rockdu force-pushed the h3-2gpu-e2e branch 3 times, most recently from e0ca7c9 to 0341b56 Compare August 29, 2026 14:25
@Rockdu Rockdu changed the title test(e2e): deterministic H3 t2va 2-GPU flowGRPO metric regression test(e2e): deterministic H3 t2va 2-GPU flowGRPO metric to prevent regression Aug 29, 2026
@Rockdu
Rockdu marked this pull request as ready for review August 30, 2026 01:01
@Rockdu
Rockdu merged commit c2106a9 into radixark:main Aug 30, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci-e2e Run e2e metric-regression tests on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant