Skip to content

Keep one current V2.1 dataset in the checkout - #44

Merged
MagMueller merged 2 commits into
mainfrom
codex/v21-only-dataset
Sep 25, 2026
Merged

MagMueller merged 2 commits into
mainfrom
codex/v21-only-dataset

Conversation

@MagMueller

@MagMueller MagMueller commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Main still carried two 55-task selector files and an obsolete 200-task snapshot alongside the current V2.1 dataset. Remove those three files and make the README point to one current 200-task V2.1 dataset, retaining the compatible BU_Bench_V2.enc filename.

Revision validation now reads its hash-pinned original baseline from Git history, with --base-artifact for archives or shallow clones. CI fetches the required history. The current dataset and encrypted judge review cases are byte-identical; V1/Stealth, scoring, runner settings and historical release assets are unchanged.

Validation: 68 unittest cases (65 passed, 3 optional skips); rubric validator confirms all 200 IDs, weights and canaries; Ruff, YAML parsing and diff checks pass. Verified the default loads all 200 tasks and V1/Stealth still load.


Summary by cubic

Keeps one current 200-task V2.1 dataset in the checkout. Removes the two 55-task selector files and the obsolete August 25 snapshot, and updates the README to reference only BU_Bench_V2.enc.

Refactors

  • Revision validation reads its hash-pinned original baseline from Git history instead of the removed snapshot, verifying the SHA against rubric_revision.json.
  • A new --base-artifact flag supplies the original encrypted baseline for source archives or shallow clones, with the same hash check.
  • CI fetches the pinned baseline commit so validation runs there.
  • The current dataset and encrypted judge review cases are byte-identical; V1/Stealth, scoring, runner settings, and historical release assets are unchanged. Normal benchmark runs need neither Git history nor the baseline.

Written for commit e5b79bd. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Fix all with cubic | Re-trigger cubic

Comment thread tests/test_rubric_revision.py Outdated
Comment thread .github/workflows/test.yml Outdated
Comment thread RUBRIC_REVISION.md
Comment thread tests/test_rubric_revision.py
@MagMueller
MagMueller merged commit 6ddf6d6 into main Sep 25, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant