Skip to content

Add reproducible seeded submission order - #74

Merged
xujustinj merged 1 commit into
mainfrom
feat/reproducible-shuffle
Oct 12, 2026
Merged

xujustinj merged 1 commit into
mainfrom
feat/reproducible-shuffle

Conversation

@xujustinj

Copy link
Copy Markdown

A sorted dataset can cluster similar documents during concurrent inference. Add optional --shuffle_seed to the benchmark and inference commands so comparisons share a reproducible order independent of filesystem discovery. Save the seed and ordered IDs in _submission_order.json.

Schedule async tasks in input order before observing completion order. Omitting the seed preserves discovery order; cached results are still skipped normally.

Validation:

  • 9 focused tests: shuffle reproducibility across pipelines/discovery order, different seeds, unchanged unseeded order, async scheduling, CLI forwarding, existing external filename behavior.
  • Repository-wide Ruff lint and format checks.
  • git diff --check.
  • No provider requests or paid inference.

@xujustinj
xujustinj merged commit 15fd6cf into main Oct 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant