feat(sft): single-card equivalent of the 8-GPU H3 SFT recipe

[Rockdu]

What

scripts/run_diffusion_sft_h3_t2va_1gpu.py — run_diffusion_sft_h3_t2va.py on one card, training the same thing rather than a scaled-down approximation. Same automation level as the 8-GPU recipe: prepare() fetches rockdu/WISA-80K-Practical-Dynamics-254, so a zero-argument launch is a complete run and --data-dir is the only data knob.

Below the docstring the two scripts differ in four lines: the run-name prefix, --fsdp-cpu-offload, --fsdp-master-dtype bf16, and 8 → 1 GPUs. Every *_args block — optimizer, LoRA, batch, sigma grid, reduce dtype — is byte-identical.

Why the two are equivalent

• Batch schedule. global_batch_size is 8 in both. arguments.py derives it as rollout_batch_size × n_samples_per_prompt ÷ num_steps_per_rollout — no world-size term — so one card only changes the layout: 8 dp ranks × 1 micro-batch becomes 1 rank × 8 gradient-accumulation micro-batches at micro_batch_size=1, still 4 optimizer steps per rollout.
• Gradient scale. actor.py backwards loss_sum / num_local_pairs: /8 on the single rank, /1 on each of eight before FSDP2's mesh-mean divides by 8 again.
• Precision. --fsdp-master-dtype bf16 changes no math under --fsdp-cpu-offload. Only the LoRA adapters train and PEFT keeps those in fp32 whatever the base dtype (measured: base bf16 → lora_A/lora_B float32, requires_grad=True); the frozen base is gathered as bf16 for the forward at either setting, since MixedPrecisionPolicy applies param_dtype=bf16 to every wrap and H3 declares no param_dtype_patterns. Step 1 of an otherwise identical run reproduces the fp32 loss to every digit logged, 3.276656e-01. _keep_in_fp32_modules still holds proj_in/proj_out/audio_proj_*/time_embedder/rope in fp32; --fsdp-reduce-dtype stays fp32 because the LoRA gradients do train.

--fsdp-cpu-offload is the one flag that is a requirement rather than a free choice: H3 is 33.5B parameters, so even a bf16 master is ~66 GB and leaves no room for activations on a 139.8 GB H200.

What it costs

Against the 8-GPU reference run on the same dataset, warm cache. Per-step figures are perf/actor_train_time, so encoding is excluded.

	8 GPUs	1 GPU
GPU peak (train step)	~57 GB/rank	39.5 GB — activations plus one block; no weights resident
host RAM	—	217 GB peak at init, 99 GB steady (~95 GB pinned)
init	—	~6 min
per optimizer step	30.5 s	278 s
per rollout (4 steps)	122 s	17.3 min

The 9× gap is 8× rank count plus ~14% for the PCIe round trip the offload adds to every block, on the forward and the gradient-checkpoint recompute alike. A cold .sft_cache costs more and only here — one encode worker instead of eight, ~480–590 s per rollout through the first epoch, during which the ~67 GB encoder rather than the train step owns the GPU peak.

Validation

A zero-argument run on a clean box, end to end:

• prepare() downloaded the dataset (1.1 GB, 254 rows, clips/), and its relative media paths resolved — the path this recipe would otherwise never exercise, since every earlier run of mine used absolute paths.
• FSDP init 6 min; encode 32/32 clips (583.7 s); 4 optimizer steps with per-sigma-bucket metrics; clean exit.
• Separately, an 11-rollout run to iter_0000011 → scripts/export_lora.py → a 624-tensor rank-64/alpha-128 adapter structurally identical to the 8-GPU one, which then served under sglang serve --lora-path and reproduced the lora_sft_guide figure prompts.
• --deterministic-mode was exercised and passes (no op lacks a deterministic implementation), but is not on by default: it costs ~25% per step and ~3.6 GB of GPU, enough to push the peak past a 40 GB card. Documented as --extra-args "--deterministic-mode".

Limits

The two curves are comparable on trend and on the per-sigma buckets, not point by point. prepare_sft_batch seeds the sigma and noise draw on (rollout_id, microbatch_id, dp_rank), so changing topology moves every sample to a different sigma by construction. A long-run trend comparison against the 8-GPU reference is not included here. The run that would have provided it reached 78 steps (19 rollouts) at fp32 master before I stopped it to switch the recipe to bf16; at that length both series still sit inside the 8-GPU run's own warm-up spread, so it would not have supported a conclusion either way.

Status is registered NV: no complete training curve has been run at this topology.

Checklist

• pre-commit run --files <touched> passes
• Added/updated tests — none; the script carries no library code, and its four differing lines are launch flags
• python3 scripts/run_diffusion_sft_h3_t2va_1gpu.py --help parses
• Zero-argument run validated end to end (see above)
• New public flags — none; --fsdp-cpu-offload, --fsdp-master-dtype and --deterministic-mode all already exist and are documented
• Docs updated (docs/models/h3/h3.md §4.1, recipe-verification.md)