Repository navigation
Conversation
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Removes only Tau3 Banking's duplicate TypeScript implementation after its native Harbor replacement is validated and available. Draft / blocked on cutover; do not merge yet. MMMU Pro Vision and GPQA stay in Kepler.
What changed?
tau3_bench_bankingfrom the registry, metadata, config union, option schemas, and CLI dispatch, along with its simulator, tools, retrieval, evaluator, and tests.--image-detail, and all other remaining Kepler benchmarks. MMMU files and tooling are unchanged from the base branch.Stack and rollout
c6351231d8666e99b25c8f3b6e386bbb0283d4ff.Native adapter prerequisite: Harbor #3595, which extends the existing Tau3 adapter for revision-pinned banking generation and BM25+grep.
Merge gates: finish native model/runtime validation, publish the pinned dataset, support Tau3's agent and runtime service in the deployed Harbor runner/catalog, and drain existing Kepler Tau3 jobs. Then sync the harness subtree and update monorepo registrations/callers together. Merging removal first would break existing callers; generation alone does not satisfy these gates.
How to test
At
bdec2cf:bun run format:check,bun run check(existing warnings only),bun run typecheck,bun test(977 passed, 0 failed), andbun run build.bdec2cffixes it and all checks were rerun.task_001at the pinned source and BM25+grep scored 1/1 in Kepler (a3f44415) and 1/1 in Harbor (fea9e195), usingopenai/gpt-5.4-minifor agent/user with medium effort. Harbor also passed its Docker oracle. Step/token ceilings and simulator API/seed behavior differ; broader parity and production integration remain pending.To reproduce, run
bun install --frozen-lockfile, then the five validation commands above. Confirm the migration regression tests retain MMMU/GPQA and reject Tau3.Reviewer focus
Upstream Tau3 defaults to BM25 without grep; Kepler uses BM25+grep. Match the source revision, retrieval, prompts, user model/effort, grading, and seeds before comparing scores. This PR deliberately remains blocked; no production benchmark is removed by opening it.
Security and privacy
No new credentials, model calls, logs, or ePHI flows in the diff. Deletes Tau3 implementation code and synthetic fixtures; preserves historical result/storage infrastructure.