[nilam-divyam]

Re-submission of divyam-router with affiliation Divyam (https://divyam.ai)

Changes:

• model_cost/model_cost.json (added gemini-3.8-flash)
• router_inference/config/divyam-router.json
• router_inference/predictions/divyam-router.json
• router_inference/predictions/divyam-router-robustness.json
• universal_model_names.py (added gemini-3.8-flash)

[nilam-divyam]

/evaluate

[nilam-divyam]

/evaluate

[github-actions]

Router Evaluation Results

Router: divyam-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7685
Accuracy	80.39%
Total Cost	$5.490983
Avg Cost per Query	$0.000654
Avg Cost per 1K Queries	$0.6537
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7571

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.0642
Opt.Cost (Cost Efficiency)	0.1684
Opt.Acc (Accuracy vs Optimal)	0.9270

---

Evaluation completed by RouterArena automated workflow

[nilam-divyam]

Thanks for the evaluation. FYI, when we re-graded the same prediction file locally with upstream main (llm_evaluation/run.py divyam-router full ), we got a RouterArena score of 0.7689 (accuracy 80.44%, cost per 1K queries $0.6537), versus 0.7685 (80.39%) from the workflow. The cost is identical, so the difference is about 4 of 8,400 queries graded wrong on CI but correct locally. We think it comes from LiveCodeBench, where generated code runs against test cases with time limits: a few solutions that finish just in time on a local machine may time out on the slower GitHub runner. The other benchmarks are graded by text matching, so they shouldn't vary between machines. We just wanted to flag that LiveCodeBench grading can vary slightly depending on the machine.

[yl231]

/evaluate

[github-actions]

Router Evaluation Results

Router: divyam-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7688
Accuracy	80.43%
Total Cost	$5.490983
Avg Cost per Query	$0.000654
Avg Cost per 1K Queries	$0.6537
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7571

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.0657
Opt.Cost (Cost Efficiency)	0.1691
Opt.Acc (Accuracy vs Optimal)	0.9285

---

Evaluation completed by RouterArena automated workflow

[yl231]

Thanks, @nilam-divyam. Merged. We re-ran /evaluate after merging the LiveCodeBench scorer fixes (#230, #234, #239): Arena 0.7688, accuracy 80.43%. The leaderboard row update follows in a separate PR. The small local-vs-runner difference you noted will be evened out by the full re-grade, which runs every router on the same runners. Prices for all routers, including the new gemini-3.8-flash entry, will be revisited together when we rebuild the price table.

[yl231]

A correction to my note above: the /evaluate re-run graded with this branch's own copy of the evaluator, so it did not include the LiveCodeBench fixes after all. The full re-grade in #253 uses the current evaluator for every router. There, Divyam scores 77.22 (accuracy 80.83%), mostly from the sys.stdin.buffer fix (#234), which puts it at #1 once #253 merges.