Skip to content

refactor(benchmarks): retire Tau3 Banking from Kepler (2/2) - #140

Draft
ayush-or wants to merge 4 commits into
devin/1790205187-remove-harbor-benchmarksfrom
codex/remove-native-mmmu-tau3
Draft

ayush-or wants to merge 4 commits into
devin/1790205187-remove-harbor-benchmarksfrom
codex/remove-native-mmmu-tau3

Conversation

@ayush-or

@ayush-or ayush-or commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

Removes only Tau3 Banking's duplicate TypeScript implementation after its native Harbor replacement is validated and available. Draft / blocked on cutover; do not merge yet. MMMU Pro Vision and GPQA stay in Kepler.

What changed?

  • Removes tau3_bench_banking from the registry, metadata, config union, option schemas, and CLI dispatch, along with its simulator, tools, retrieval, evaluator, and tests.
  • Preserves MMMU Pro Vision, its original media manifest and mirror tooling, --image-detail, and all other remaining Kepler benchmarks. MMMU files and tooling are unchanged from the base branch.
  • Adds regression coverage that Tau3 is no longer registered and MMMU, GPQA, MMLU Pro, Tau Airline, and VGI remain available.
  • Does not change deployed routing, regenerate MMMU media, delete historical results, or publish datasets.

Stack and rollout

  1. #113 removes the original six Harbor-backed implementations; base c6351231d8666e99b25c8f3b6e386bbb0283d4ff.
  2. This PR removes Tau3 Banking only.

Native adapter prerequisite: Harbor #3595, which extends the existing Tau3 adapter for revision-pinned banking generation and BM25+grep.

Merge gates: finish native model/runtime validation, publish the pinned dataset, support Tau3's agent and runtime service in the deployed Harbor runner/catalog, and drain existing Kepler Tau3 jobs. Then sync the harness subtree and update monorepo registrations/callers together. Merging removal first would break existing callers; generation alone does not satisfy these gates.

How to test

At bdec2cf:

  • Passed locally: bun run format:check, bun run check (existing warnings only), bun run typecheck, bun test (977 passed, 0 failed), and bun run build.
  • Passed: CI validation.
  • The first narrowing commit failed formatting on an extra blank line in CLI coverage; bdec2cf fixes it and all checks were rerun.
  • Native smoke passed, not statistical parity: the same task_001 at the pinned source and BM25+grep scored 1/1 in Kepler (a3f44415) and 1/1 in Harbor (fea9e195), using openai/gpt-5.4-mini for agent/user with medium effort. Harbor also passed its Docker oracle. Step/token ceilings and simulator API/seed behavior differ; broader parity and production integration remain pending.

To reproduce, run bun install --frozen-lockfile, then the five validation commands above. Confirm the migration regression tests retain MMMU/GPQA and reject Tau3.

Reviewer focus

Upstream Tau3 defaults to BM25 without grep; Kepler uses BM25+grep. Match the source revision, retrieval, prompts, user model/effort, grading, and seeds before comparing scores. This PR deliberately remains blocked; no production benchmark is removed by opening it.

Security and privacy

No new credentials, model calls, logs, or ePHI flows in the diff. Deletes Tau3 implementation code and synthetic fixtures; preserves historical result/storage infrastructure.

@ayush-or ayush-or changed the title refactor(benchmarks): retire MMMU Pro and Tau3 from Kepler (2/2) refactor(benchmarks): retire Tau3 Banking from Kepler (2/2) Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant