fix: Salt failure is a hard error

[arekay-nv]

What does this PR do?

Makes salt failures a hard error instead of a silent skip. Previously, when --warmup-salt was on (the default) but a sample couldn't be salted, Dataset._apply_salt logged a warning (or silently passed the sample through) and the warmup ran with no cache-busting — the exact condition salt exists to prevent, failing quietly.

Now salt=True requires every sample to be a dict with a text (str) prompt and no input_tokens; anything else raises DatasetValidationError at dataset-load time, before any load is issued.

Key changes:

Area	Change
Validation	New Dataset.validate_saltable() — rejects non-dict samples, input_tokens (pre-tokenized; adapters send these verbatim so a salted prompt never reaches the server), missing prompt, and non-str prompt. Error names the offending sample index + remediation.
Fail-fast placement	Called from _load_datasets when warmup.enabled and warmup.salt, so an unsaltable dataset fails before worker/aggregator subprocesses spawn. Also called from with_salt() so the salting mechanism stays self-protecting.
Hot path	_apply_salt collapses to a single dict-merge ({**data, "prompt": f"[{salt}] {data['prompt']}"}) — the multimodal-list handling, input_tokens warnings, and silent passthroughs are deleted; the contract is guaranteed upstream.

⚠️ Behavior change (note for reviewers)

• Multimodal / pre-tokenized warmup + salt now hard-errors. A prompt that is a list (image/video workloads) or a dataset with input_tokens (e.g. gpt-oss-120b, DeepSeek-R1 via /v1/completions) will fail warmup instead of silently skipping salt. Fix: set --warmup-salt=false / warmup.salt: false.
• Whole-dataset validation. The run is rejected if any sample is non-conforming — including samples the n_requests subset would never issue. Stricter than the old per-sample skip; intended by the hard-error design.

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request introduces early validation for dataset salting during the benchmark setup phase. It adds a validate_saltable method to ensure that all dataset samples are compatible with salting (i.e., they are dictionaries containing a string prompt and no input_tokens) before any subprocesses are spawned. The review feedback highlights a critical bug in validate_saltable where iterating over self.data directly will fail if it is a pandas.DataFrame (as it would iterate over column names instead of rows), and provides a code suggestion to handle this case.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[nv-alicheng]

I have 2 questions:

1. Have we studied whether or not injecting these query-irrelevant random strings at the beginning affects accuracy in a format-sensitive workload like GPT-OSS that uses harmonize?
2. Is it possible to salt a token_ids list by injecting a random list of ints sampled from a set of known tokens that do not represent special markers?

[arekay-nv]

1. • not yet, we can do that but the goal is to eliminate kv-cache reuse between the warmup and performance by changing the prefix of the user prompt. For something like gpt-oss, we need a haromize aware salting mechanism.
2. That is the long term plan - to have each adapter add sensible salt. For now, with the text "prompt" column datasets, we can inject salt, but others require understanding the structure and picking valid tokens.

[nv-alicheng]

Should have been caught earlier, but that is something to address -
exceptions.py is very poorly designed and only contains bare name aliases for particular failure states.

This construction of error message and _salt_violation should be part of the DatasetValidationError class:

class DatasetValidationError:
    ERR_MSG_TEMPLATE = "salt=True requires ... but sample {index} {reason} ..."

    def __init__(self, dataset_idx: int, sample: Sample):
        self.reason = self._violation_reason(sample)
        err_msg = DatasetValidationError.ERR_MSG_TEMPLATE.format(index=dataset_idx, reason=self.reason)
        super().__init__(err_msg)

    def _violation_reason(self, sample: Sample):
        # Body of `_salt_violation` here.

Ideally, reason should be some enum or object rather than a raw string, smth like

class DatasetValidationError:
     ...

    class Reason(Enum):
         TypeMismatch = ...
         InputTokensShadowing = ...
         PromptMissing = ...
         PromptTypeMismatch = ...
         Other = ...

         def fmt_str(self, sample):
              <handle conversion to full error reason string>
     ...

[nv-alicheng]

Review Council — PR #406 (3× Claude, depth: thorough)

Three independent Claude reviewers (bugs/correctness, design/edge, testing/docs) plus a code-quality pass, deduped and severity-recalibrated. "R" marks issues flagged independently by multiple reviewers. All line numbers verified against HEAD (44fc32d7).

Verdict: Solid, focused bug-fix. Fail-fast placement is correct — validation at _load_datasets (execute.py:387) runs before any worker/aggregator subprocess spawns, and warmup's with_salt() (execute.py:546) salts the same ctx.dataloader it re-validates. Transforms apply eagerly in load(), so validate_saltable() scans the exact post-transform shape load_sample() returns — no mismatch. No security issues, green suite. The findings are about strictness + a default, not broken logic.

🟡 Should fix

#	File:line	Cat	R	Issue
1	config/schema.py:762 + execute.py:386	design	3×	warmup.salt defaults True + the new hard error = footgun for first-class pre-tokenized workloads. openai_completions (gpt-oss-120b), sglang, and DeepSeek-R1 run Harmonize()+ColumnFilter(["input_tokens"]) at load, so their samples always carry input_tokens and no prompt. Plain --warmup (without --warmup-salt false) now aborts setup for every such run that previously warned and ran unsalted. Options: default salt=False, auto-downgrade to warning+unsalted when input_tokens is present, or (minimum) state the text-prompt requirement in the field help/description.
2	tests/unit/commands/test_benchmark.py:248	testing		The 3 TestLoadDatasetsSaltValidation tests @patch.object(Dataset, "validate_saltable"), so they only assert called/not-called — they never exercise real validation through _load_datasets. The exact integration this PR adds (unsaltable perf dataset → DatasetValidationError before subprocess spawn) has zero coverage. Add a test that writes an input_tokens/int-prompt JSONL and asserts pytest.raises(DatasetValidationError) through _load_datasets unpatched.
3	execute.py:386	api-contract		Whole-dataset strictness vs the warmup subset. Validation rejects the entire run if any sample is non-saltable, but salt only touches warmup, which issues just a seeded n_requests subset — and the perf phase runs unsalted regardless. A mostly-text dataset with one anomalous row now hard-fails at load even if that row is never issued. Consider scoping validation to the warmup-issued subset.
4	dataset_manager/dataset.py:470	error-handling		if self.data is None: return silently passes validation on an unloaded dataset — contradicts the PR's own fail-loud thesis and masks a "dataset not loaded" ordering bug. A truly-empty loaded dataset is [] (already a no-op via the loop), not None; EmptyDataset is never actually instantiated. Distinguish: raise on None (not-loaded), no-op on [].
5	tests/unit/dataset_manager/test_salted_dataset.py:196	testing		match=r"\b1\b" is a weak/brittle index assertion — re.search passes on any word-bounded 1 anywhere in the message and fails for two-digit indices. Anchor to match=r"sample 1\b".

🔵 Consider

• dataset.py:270 — both-keys (input_tokens + valid str prompt) is now rejected, but the deleted code salted the prompt in that case. A prompt-consuming adapter on a mixed-schema dataset loses a salt it used to get. Confirm no dataset+adapter pairing sends prompt while input_tokens is present.
• dataset.py:495 / execute.py:387 (2×) — the full dataset is scanned twice (load-time guard + with_salt re-scan) over unchanged shared data. Cold-path and fine, but for the 50k+-sample corpora this project targets, add a one-line comment marking the early call as the intentional pre-spawn fail-fast so a future reader doesn't "dedupe" it away.
• dataset.py:477 — the error names sample {i}, an index into post-transform/filtered loaded order, not the user's source-file line; a JSONL with column filters won't map cleanly. Clarify "index into loaded order."
• Testing — the online/agentic path and accuracy_only (dataloader None → skip) are uncovered; pin the intended skip so a refactor can't start validating a None dataloader.

🧹 Code quality

• dataset.py:464 & :490 — docstrings narrate removed behavior ("rejected rather than skipped", "fails here rather than silently issuing unsalted prompts"). Per AGENTS.md ("describe current state, not development history"), reframe positively and drop the "rather than …" clauses.
• dataset.py:506 — assert self._salt_rng is not None vanishes under python -O → a bare AttributeError on the warmup path. For a PR about making salt failures loud, a raised error fits better (same applies to the pre-existing assert self.data is not None on the hot access path).

---

Convergence: all three reviewers independently landed on the salt-default-True footgun (#1) — the one decision worth the author's attention; the rest is polish.