radixark · Rockdu · Sep 6, 2026 · Sep 6, 2026
diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -65,11 +65,6 @@ COPY requirements.txt /tmp/requirements.txt
 RUN pip install --ignore-installed PyJWT
 RUN pip install -r /tmp/requirements.txt
 
-# Official HPSv2 reference for the alignment test; its protobuf<4 pin conflicts with wandb.
-# Pinned to git: the PyPI 1.2.0 wheel omits the BPE vocab, so `import hpsv2.src.open_clip` fails.
-RUN pip install --no-deps args==0.1.0 clint==0.5.1 "hpsv2 @ git+https://github.com/tgxs002/HPSv2@866735ecaae9" && \
-    python3 -c "import hpsv2.src.open_clip"
-
 # ====================================== Install main package ============================================
 
 # Install miles_diffusion from a pinned ref (like radixark/miles does for miles),

diff --git a/docs/advanced/streaming-reward.md b/docs/advanced/streaming-reward.md
@@ -76,7 +76,7 @@ seen over the rollout:
 | Metric | Meaning |
 |---|---|
 | `perf/parser_max_queue_depth` | Deepest backlog on a parser actor when a response was handed to it |
-| `perf/reward_max_queue_depth_<pool>` | Deepest backlog on that reward pool's actors when a batch was handed to them, one metric per pool (`hps`, `pickscore`, `ocr`) |
+| `perf/reward_max_queue_depth` | Deepest backlog on a reward actor when a batch was handed to it |
 
 `0` means every dispatch found its worker idle — that stage never made anything wait. A number that climbs with the
 number of concurrent microgroups means the stage is saturated and requests are lining up behind it.
@@ -86,9 +86,9 @@ Read them together with `perf/rollout_time`:
 | Symptom | Look at | Likely fix |
 |---|---|---|
 | `perf/rollout_time` high | `perf/parser_max_queue_depth` > 0 | Deserialization is the bottleneck; raise `--rollout-parser-num-workers` |
-| `perf/rollout_time` high | `perf/reward_max_queue_depth_<pool>` > 0 | That pool's scoring is the bottleneck; add its workers, or give it a dedicated GPU |
+| `perf/rollout_time` high | `perf/reward_max_queue_depth` > 0 | Scoring is the bottleneck; add reward workers, or give them a dedicated GPU |
 | `perf/rollout_time` high, both depths `0` | Neither pool is holding anything up | The engines themselves are the limit — check `--rollout-microgroup-size` and `--sglang-server-concurrency` |
 
 Both metrics are emitted only when the corresponding pool ran, so a reward with no actor pool leaves
-its `perf/reward_max_queue_depth_<pool>` absent rather than zero.
+`perf/reward_max_queue_depth` absent rather than zero.
 
diff --git a/docs/models/sd3/sd3.md b/docs/models/sd3/sd3.md
@@ -13,7 +13,7 @@ DiT transformer with dual text-encoder conditioning (`encoder_hidden_states` +
 
 - Single DiT component — weight sync targets `--update-weight-target-module transformer` (default).
 - LoRA on all attention projections (self-attn + cross-attn add projections).
-- Supports **Flow-GRPO** (OCR or HPS) and **DiffusionNFT** (PickScore) objectives.
+- Supports **Flow-GRPO** (OCR) and **DiffusionNFT** (PickScore) objectives.
 - Gated Hugging Face model — requires `HF_TOKEN`.
 
 ## 2. Supported variants
@@ -51,7 +51,6 @@ Prompt datasets live under
 | Recipe | Subset | Train path |
 |---|---|---|
 | GRPO + OCR | `flowgrpo_ocr` | `.../flowgrpo_ocr/train.jsonl` |
-| GRPO + HPS | `hpdv2` | `.../hpdv2/train.jsonl` |
 | NFT + PickScore | `flowgrpo_pickscore` | `.../flowgrpo_pickscore/train.jsonl` |
 
 Launch scripts download the matching subset automatically via
@@ -103,7 +102,6 @@ All recipes are Python modules under `scripts/`. Each exposes a Typer CLI
 | Script | Reward | GPUs | Algorithm |
 |---|---|---|---|
 | `run_diffusion_grpo_sd3_ocr_sglang.py` | OCR (CPU) | 2 colocate | Flow-GRPO |
-| `run_diffusion_grpo_sd3_hps_sglang.py` | HPS | 2 colocate | Flow-GRPO |
 | `run_diffusion_nft_sd3_pickscore.py` | PickScore | 3 (2+1) | DiffusionNFT |
 
 ### 5.2 Flow-GRPO + OCR (2 GPU colocate)
@@ -122,27 +120,7 @@ Walkthrough: [Quick Start](../../getting-started/quick-start.md).
 
 E2E test: `tests/e2e/short/test_sd3_ocr_grpo_2xGPU.py`.
 
-### 5.3 Flow-GRPO + HPS (2 GPU colocate)
-
-Canonical script: `scripts/run_diffusion_grpo_sd3_hps_sglang.py`
-
-**Status:** [📈 V — Verified](../../user-guide/recipe-verification.md#v) — 600 rollouts
-(2 optimizer steps each, 1,200 in total) on 2×H200; `rollout/reward/raw_mean` 0.284 → 0.349
-(mean of the last 100 rollouts, peak 10-rollout moving average 0.362).
-
-```bash
-export HF_TOKEN=...
-python3 scripts/run_diffusion_grpo_sd3_hps_sglang.py \
-  --cuda-visible-devices 6,7
-```
-
-The recipe uses the SD3 Flow-GRPO SDE, LoRA and precision configuration of the OCR
-recipe, swaps in the deduplicated `hpdv2` prompts and `--rm-type hps`, and colocates one
-HPS reward actor with the train and rollout workers. It keeps Flow-GRPO's own KL weight
-(`--diffusion-kl-beta 0.01`) and group-wise advantage std instead of the OCR recipe's
-`--diffusion-kl-beta 0.04 --globalize-reward-std`.
-
-### 5.4 DiffusionNFT + PickScore (3 GPU)
+### 5.3 DiffusionNFT + PickScore (3 GPU)
 
 Script: `scripts/run_diffusion_nft_sd3_pickscore.py`
 
@@ -162,24 +140,21 @@ MILES_SCRIPT_SMOKE=1 python3 scripts/run_diffusion_nft_sd3_pickscore.py
 
 ### Recipe comparison
 
-| | GRPO + OCR | GRPO + HPS | NFT + PickScore |
-|---|---|---|---|
-| Script | `run_diffusion_grpo_sd3_ocr_sglang.py` | `run_diffusion_grpo_sd3_hps_sglang.py` | `run_diffusion_nft_sd3_pickscore.py` |
-| `--loss-type` | `policy_loss` (default) | `policy_loss` (default) | `nft` |
-| SDE | Full window, noise=0.7, CFG=4.5 | Full window, noise=0.7, CFG=4.5 | ODE, noise=0 |
-| Reference | LoRA base KL (β 0.04) | LoRA base KL (β 0.01) | EMA (`--use-ema`) |
-| Reward placement | CPU OCR | Colocated HPS actor | Dedicated PickScore GPU |
-| Verification | FG | V | FG |
+| | GRPO + OCR | NFT + PickScore |
+|---|---|---|
+| Script | `run_diffusion_grpo_sd3_ocr_sglang.py` | `run_diffusion_nft_sd3_pickscore.py` |
+| `--loss-type` | `policy_loss` (default) | `nft` |
+| SDE | Full window, noise=0.7, CFG=4.5 | ODE, noise=0 |
+| Reference | LoRA base KL | EMA (`--use-ema`) |
+| Reward GPU | None (CPU OCR) | Dedicated (3 GPU total) |
+| Deterministic e2e | `test_sd3_ocr_grpo_2xGPU` | `test_sd3_nft_pickscore_3xGPU` |
 
 ## 6. Recipe configuration
 
 ### GPU layout
 
 **GRPO + OCR (default script):** 2 GPUs colocated (`--colocate`); OCR on CPU Ray actors.
 
-**GRPO + HPS:** 2 GPUs colocated (`--colocate --hps-reward-colocate`); one HPS
-actor shares the rollout GPUs.
-
 **NFT + PickScore:**
 
 | GPU role | Count | Flags |
@@ -188,9 +163,9 @@ actor shares the rollout GPUs.
 | PickScore reward | 1 | `--pickscore-num-workers 1`, `--pickscore-num-gpus-per-worker 1.0` |
 | **Total** | **3** | `--num-gpus-per-node 3` |
 
-PickScore runs as a Ray actor pool on a dedicated GPU. With `--pickscore-reward-colocate`
-(not used in the default script), reward workers share the rollout GPUs
-instead — useful only when GPU count is tight.
+PickScore runs as a Ray actor pool on a dedicated GPU. With `--colocate-reward`
+(not used in the default script), reward workers share rollout GPUs at 0.05 GPU
+per worker — useful only when GPU count is tight.
 
 ### Algorithm flags
 
@@ -206,18 +181,6 @@ instead — useful only when GPU count is tight.
 | Weight sync | `--lora-ipc-weight-sync` (colocate IPC merge) |
 | Determinism | `--deterministic-mode` (CI / e2e parity) |
 
-**Flow-GRPO + HPS:** uses the same SDE, LoRA and precision flags as the OCR recipe,
-with these settings of its own:
-
-| Setting | Value |
-|---|---|
-| Reference | `--diffusion-kl-beta 0.01` |
-| Advantage | Group-wise std (no `--globalize-reward-std`) |
-| Reward | `--rm-type hps --hps-version v2.1` |
-| Reward worker | One actor, batch size 8 |
-| Placement | `--hps-reward-colocate` |
-| Prompt subset | `hpdv2` |
-
 **DiffusionNFT + PickScore:**
 
 | Setting | Value |
@@ -231,7 +194,7 @@ with these settings of its own:
 
 ## 7. LoRA and weight sync
 
-All SD3 recipes use LoRA with IPC weight sync:
+Both SD3 recipes use LoRA with IPC weight sync:
 
 ```bash
 --use-lora \
@@ -247,7 +210,7 @@ merge internals.
 
 ## 8. Precision notes
 
-All SD3 launch scripts use fp16 DiT forward:
+Both SD3 launch scripts use fp16 DiT forward:
 
 ```bash
 --diffusion-forward-dtype fp16 \
@@ -290,7 +253,7 @@ by the E2E fixture.
 ## 10. Pairs well with
 
 - [Quick Start](../../getting-started/quick-start.md) — SD3.5 Flow-GRPO OCR walkthrough.
-- [Rewards](../../user-guide/rewards.md) — OCR, HPS and PickScore scoring.
+- [Rewards](../../user-guide/rewards.md) — OCR and PickScore scoring.
 - [Customization](../../user-guide/customization.md) — `--*-path` plug-points.
 - [SDE step backend](../../advanced/sde-backend.md) — SDE window (GRPO) vs ODE (NFT).
-- [LoRA weight sync](../../advanced/lora.md) — IPC merge used by all SD3 recipes.
+- [LoRA weight sync](../../advanced/lora.md) — IPC merge used by both recipes.
diff --git a/docs/user-guide/cli-reference.md b/docs/user-guide/cli-reference.md
@@ -265,26 +265,18 @@ See [Dtype Control](../advanced/dtype-control.md).
 
 | Flag | Type | Default | Notes |
 |---|---|---|---|
-| `--rm-type` | enum | – | `pickscore` / `hps` / `ocr`. Overridable per sample via `metadata.rm_type`. Ignored when `--custom-rm-path` is set. |
+| `--rm-type` | enum | – | `pickscore` / `ocr`. Overridable per sample via `metadata.rm_type`. |
 | `--reward-key` | str | – | When the reward is a dict. |
 | `--group-rm` | flag | off | Score a whole prompt group at once. |
-| `--custom-rm-path` | str | – | `async def rm(args, samples) -> list[float]`. Batched only; replaces the `--rm-type` dispatch entirely. Shipped: `miles.rollout.rm_hub.weighted_mixture_rm.weighted_mixture_rm` (weighted sum of built-in rewards). |
-| `--custom-rm-args` | str | – | Opaque config string for the custom RM, read as `args.custom_rm_args`; e.g. `"hps=0.7,pickscore=0.3"` for `rm_hub.weighted_mixture_rm`. |
-| `--reward-key` | str | – | For dict-valued rewards: the entry GRPO trains on. Every entry is also logged as `rollout/reward/<key>_mean` and `eval/<dataset>/<key>`. |
+| `--custom-rm-path` | str | – | `async def rm(args, samples) -> list[float]`. Batched only. |
 | `--custom-reward-post-process-path` | str | – | Replace advantage normalisation. |
+| `--colocate-reward` | flag | off | Reward actors onto rollout GPUs (train 0.7 + rollout 0.25 + reward 0.05). Requires `--colocate`. |
 | `--pickscore-model-path` | str | – | Required for `--rm-type pickscore`. |
 | `--pickscore-processor-path` | str | – | Required for `--rm-type pickscore`. |
 | `--pickscore-num-workers` | int | `1` | |
-| `--pickscore-num-gpus-per-worker` | float | `1.0` | Standalone workers only; fractional values allowed. |
-| `--pickscore-reward-colocate` | flag | off | One worker per rollout GPU, sharing it with the train actor and rollout engine. Requires `--colocate`. |
+| `--pickscore-num-gpus-per-worker` | float | `1.0` | Fractional values allowed. |
 | `--pickscore-batch-size` | int | `8` | |
 | `--pickscore-num-frames` | int | – | Frames scored per video; unset = all. |
-| `--hps-num-workers` | int | `1` | |
-| `--hps-num-gpus-per-worker` | float | `1.0` | Standalone workers only; fractional values allowed. |
-| `--hps-reward-colocate` | flag | off | One worker per rollout GPU, sharing it with the train actor and rollout engine. Requires `--colocate`. |
-| `--hps-batch-size` | int | `8` | |
-| `--hps-version` | enum | `v2.1` | `v2.0` / `v2.1`. |
-| `--hps-checkpoint-path` | str | – | Local checkpoint; unset downloads from Hugging Face. |
 | `--ocr-num-workers` | int | `4` | |
 | `--rollout-parser-num-workers` | int | `1` | Ray actors deserializing rollout responses. Raise when trajectory tensors are large. |
 

diff --git a/docs/user-guide/customization.md b/docs/user-guide/customization.md
@@ -126,25 +126,15 @@ async def custom_rm(args, samples: list[Sample], **kwargs) -> list[float]:
 ```
 
 Wired only through `batched_async_rm` — implement per-sample routing inside your
-batched function if needed. Once set, `--rm-type` and `metadata.rm_type` are ignored: the
-custom function is the whole dispatch.
+batched function if needed.
 
 ```bash
 --custom-rm-path my_project.rewards.aesthetic_rm
 ```
 
-`--custom-rm-args` is an opaque string passed through as `args.custom_rm_args` for the
-custom RM to parse.
-
-Shipped custom RMs:
-
-| Path | What |
-|---|---|
-| `miles.rollout.rm_hub.weighted_mixture_rm.weighted_mixture_rm` | Weighted sum of built-in rewards (`hps`, `pickscore`, `ocr`), weights from `--custom-rm-args "hps=0.7,pickscore=0.3"`; returns a dict per sample, train on it with `--reward-key weighted`. See [Rewards](rewards.md) § Combining rewards. |
-
 HTTP / remote scoring: implement a batched custom RM and read `args.rm_url` (or
 your own flags). Encode images from `sample.generated_output` (see
-`generated_output_to_rgb_hwc_uint8_frames` in `miles/utils/processing_utils.py`):
+`_sample_to_rgb_hwc_uint8_frames` in `miles/rollout/rm_hub/pickscore.py`):
 
 ```python
 import aiohttp

diff --git a/docs/user-guide/launch-script.md b/docs/user-guide/launch-script.md
@@ -237,7 +237,7 @@ MILES_SCRIPT_EXTERNAL_RAY=1 python3 scripts/run_diffusion_grpo_wan22_pickscore_1
 ```
 
 Reward workers (`--pickscore-num-workers 4 --pickscore-num-gpus-per-worker 0.25`, no
-`--pickscore-reward-colocate`) are default-scheduled and land on the only free GPU: the reward node.
+`--colocate-reward`) are default-scheduled and land on the only free GPU: the reward node.
 
 ### Verify the run is healthy
 

diff --git a/docs/user-guide/recipe-verification.md b/docs/user-guide/recipe-verification.md
@@ -48,7 +48,6 @@ count as verification.
     full-finetune Flow-GRPO + PickScore.
 - **📈 V**
   - `run_diffusion_sft_h3_t2va.py` — MiniMax H3 8-GPU LoRA SFT.
-  - `run_diffusion_grpo_sd3_hps_sglang.py` — SD3.5 Flow-GRPO + HPSv2.1.
 - **○ NV**
   - `run_diffusion_grpo_wan22_pickscore_5gpu.py` — Wan2.2 5-GPU LoRA
     Flow-GRPO + PickScore.

diff --git a/docs/user-guide/rewards.md b/docs/user-guide/rewards.md
@@ -1,6 +1,6 @@
 ---
 title: Rewards
-description: Built-in reward models (PickScore, HPS, OCR), rm_hub dispatch, and prompt data format.
+description: Built-in reward models (PickScore, OCR), rm_hub dispatch, and prompt data format.
 ---
 Miles-diffusion scores generated images (or video frames) after each rollout
 microgroup. Reward computation lives in `miles/rollout/rm_hub/` and is invoked
@@ -13,7 +13,7 @@ For `--custom-rm-path`, `--custom-reward-post-process-path`, and other
 
 | Stage | Flag | Role |
 |---|---|---|
-| Reward type | `--rm-type` | Selects built-in scorer (`pickscore`, `hps`, `ocr`); ignored when `--custom-rm-path` is set |
+| Reward type | `--rm-type` | Selects built-in scorer (`pickscore`, `ocr`) |
 | Per-sample override | `metadata.rm_type` in JSONL | Overrides global `--rm-type` |
 | Custom reward / norm | see [Customization](customization.md) | `--custom-rm-path`, `--custom-reward-post-process-path` |
 
@@ -49,7 +49,7 @@ batching. For video outputs, frames are uniformly sampled
 | `--pickscore-processor-path` | — | Required for pickscore |
 | `--pickscore-model-path` | — | Required for pickscore |
 | `--pickscore-num-frames` | None | Video frame sampling count |
-| `--pickscore-reward-colocate` | False | One worker per rollout GPU (requires `--colocate`) |
+| `--colocate-reward` | False | Share rollout GPUs (0.05 GPU/worker) |
 
 Example from `scripts/run_diffusion_nft_sd3_pickscore.py`:
 
@@ -62,85 +62,6 @@ Example from `scripts/run_diffusion_nft_sd3_pickscore.py`:
 --pickscore-model-path yuvalkirstain/PickScore_v1
 ```
 
-### HPS (`--rm-type hps`)
-
-Implementation: `miles/rollout/rm_hub/hps.py`.
-
-HPSv2 scores text–image alignment using a preference-tuned CLIP model:
-
-- Model: `ViT-H-14`
-- Version: `--hps-version` (`v2.0` or `v2.1`)
-- Checkpoint: `--hps-checkpoint-path` (optional; defaults to `xswu/HPSv2`)
-
-Scoring formula:
-
-```
-score = diagonal(image_features @ text_features.T)
-```
-
-The diagonal pairs each image with its corresponding prompt in the batch.
-
-HPS runs as a **Ray actor pool** (`HPSRewardActor`) with round-robin batching.
-It currently accepts image outputs only (`generated_output` with one frame).
-Use a custom RM when defining video frame aggregation semantics.
-
-| Flag | Default | Description |
-|---|---|---|
-| `--hps-num-workers` | 1 | Ray actor count |
-| `--hps-num-gpus-per-worker` | 1.0 | GPU per worker (non-colocate) |
-| `--hps-batch-size` | 8 | Batch size per actor |
-| `--hps-version` | `v2.1` | `v2.0` or `v2.1` checkpoint |
-| `--hps-checkpoint-path` | None | Local checkpoint; unset downloads from `xswu/HPSv2` |
-| `--hps-reward-colocate` | False | One worker per rollout GPU (requires `--colocate`) |
-
-Example from `scripts/run_diffusion_grpo_sd3_hps_sglang.py`:
-
-```bash
---rm-type hps \
---hps-num-workers 1 \
---hps-batch-size 8 \
---hps-version v2.1 \
---hps-reward-colocate
-```
-
-### Reward placement
-
-Every GPU reward pool is placed one of two ways:
-
-- `--<rm>-reward-colocate`: every worker takes one **slot** on a rollout placement-group bundle,
-  sharing that GPU with the train actor and the rollout engine. Colocated pools share one slot ledger, so `hps` and `pickscore` can both colocate without
-  overlapping; more workers than bundles is rejected at parse time. Requires `--colocate`.
-- Otherwise the pool is **standalone**: default-scheduled at `--<rm>-num-gpus-per-worker`,
-  which only lands on GPUs outside every placement group (placement groups reserve their
-  GPUs, so Ray never packs these onto rollout GPUs).
-
-`RolloutManager` seats the colocated pools before the first rollout; standalone pools are
-built on first use.
-
-### Combining rewards
-
-`--custom-rm-path` receives `(args, samples)` and can call the built-in scorers
-directly; `--custom-rm-args` is an opaque string the framework hands to that function
-through `args`, so the function owns its own config grammar. The shipped example
-`miles/rollout/rm_hub/weighted_mixture_rm.py` reads `name=weight,name=weight`:
-
-```bash
---custom-rm-path miles.rollout.rm_hub.weighted_mixture_rm.weighted_mixture_rm \
---custom-rm-args "hps=0.7,pickscore=0.3" --reward-key weighted \
---pickscore-processor-path laion/CLIP-ViT-H-14-laion2B-s32B-b79K \
---pickscore-model-path yuvalkirstain/PickScore_v1 \
---hps-reward-colocate --pickscore-reward-colocate   # each reward keeps its own placement flags
-```
-
-The example returns a dict per sample (`{"hps": ..., "pickscore": ..., "weighted": ...}`).
-`--reward-key` picks the entry GRPO trains on, and every entry of a dict reward gets its own
-`rollout/reward/<name>_mean` panel, so the components stay visible while the sum is optimized.
-
-Weights apply to raw scores (HPSv2.1 ≈ 0.25–0.35, PickScore/26 ≈ 0.8–0.9, OCR ∈ [0, 1]), so
-pick them with the scales in mind. Colocated pools share one slot ledger, so several rewards
-can colocate without overlapping. Rewards receive `generated_output` itself, and every reward actor
-quantises it to uint8 on its own terms.
-
 ### OCR (`--rm-type ocr`)
 
 Implementation: `miles/rollout/rm_hub/ocr.py`.
@@ -174,9 +95,7 @@ generate_and_rm_microgroup()
   → batched_async_rm(args, microgroup)          # rm_hub/__init__.py
     → custom_rm_path?  user batched function
     → all pickscore?   pickscore_rm (batched)
-    → all hps?         hps_rm (batched)
-    → all ocr?         ocr_rm (batched, one image per actor call)
-    → else             per-sample async_rm → ocr / pickscore / hps / NotImplementedError
+    → else             per-sample async_rm → ocr / pickscore / NotImplementedError
   → sample.reward = score
   → RolloutManager._post_process_rewards()      # GRPO advantage normalization
 ```
@@ -208,7 +127,6 @@ Dataset repo: [`rockdu/miles-diffusion-datasets`](https://huggingface.co/dataset
 
 | Subset | Used by |
 |---|---|
-| `hpdv2/` | SD3 HPS Flow-GRPO |
 | `flowgrpo_pickscore/` | PickScore recipes (SD3 NFT, Qwen-Image, Wan2.2, LTX) |
 | `flowgrpo_ocr/` | SD3 OCR Flow-GRPO, NFT smoke test |
 
@@ -222,4 +140,4 @@ metadata.get("rm_type") or args.rm_type
 ```
 
 Mixed rm_types within one microgroup fall back to per-sample dispatch (no
-batched PickScore/HPS fast path).
+batched PickScore fast path).