Repository navigation
TITAN integration: negative cached METEOR score and cache-only submission requirements #210
Description
Hello RouterArena maintainers,
We are integrating TITAN, Theus's commercial model router, and completed a preregistered, cache-only routing pilot on the 809 shared sub_10 cases. The pool uses the exact three historical model IDs in your official caches. Selection received only each question; we froze the policy/state and sealed every prediction before accessing scores. No component was trained or tuned on RouterArena, and no inference, embedding or model-judge calls were made.
Our evaluator stopped on one cached value outside our preregistered [0,1] quality domain:
- Repository revision:
cff9659dd6a3a07815f4842fcdadea5a597a1df1 - Cache:
claude-3-haiku-20240307.jsonl, SHA-2569f5bed666a524d74e629091b57ef2fdf644181dcc1039ef44b44460f254d150a global_index:NarrativeQA_4102evaluation_result.metric:meteor_scoreevaluation_result.score:-0.0330188679245283
We retained all 809 routing decisions and have published no aggregate quality/cost for this failed run. We did not clip the value, drop the case, or impute a replacement.
Evidence: public report and downloadable artifacts, preregistered protocols and prediction seals.
Could you clarify:
- Is this negative cached METEOR value intentional, and how should the current official aggregator treat it?
- For a future commercial-router submission, may the complete 8400-case and 420-case robustness predictions reuse exact public cached generations with fixed, disclosed provenance? Which artifacts and redistribution permissions are required when the router implementation and operational snapshot remain private?
We understand that this 809-case pilot is not eligible as a full leaderboard result and are not requesting that it be listed as one. We want to establish a valid zero-new-inference path to a complete submission.
Thank you.
Activity
Cap the METEOR fragmentation penalty so scores stay in [0, 1] (RouteW…
yl231 commentedon Oct 31, 2026
Thanks for the careful report, @AstratechPe, and sorry for the slow reply.
NarrativeQA_4102now scores 0. All leaderboard routers have since been re-graded with the fix (Re-grade the 25 leaderboard routers with the current evaluator #253).cached_results/instead of calling the model again, and the cached answers and their scores must not inform the choice. Answers from other submissions' prediction files remain excluded.A complete submission needs the config file, all 8,400 rows in the prediction file and the 420-row robustness file, as in the README's submission section. Closed-source routers are welcome; the router's code and snapshot can stay private.
AstratechPe commentedon Oct 31, 2026