[lalitkumarps]

Add SapientAI Auto Router v2 submission

Router Display Name: SapientAI Auto Router v2
Router ID (Config / Predictions): PSAI-Auto-Router-v2
Organisation: Publicis Sapient — proprietary, closed-source
Author: Lalit Kumar (@lalitkumarps)
Split evaluated: full (8,400 queries) + robustness (420 queries)

---

Summary

SapientAI Auto Router v2 (PSAI-Auto-Router-v2) is the second generation of Publicis Sapient's production LLM router. Like v1, it is a proprietary, privately hosted service: routing decisions are produced by an internal endpoint, so classifier weights, the routing index, and serving code are not included in this repository. This pull request submits router configuration and prediction artefacts for evaluation and inclusion on the official RouterArena leaderboard.

All fourteen candidate models in our routing pool are already registered in model_cost/model_cost.json and mapped in universal_model_names.py, so this PR modifies no core harness files and is strictly limited to its three submission files.

Results

Metric	Value
Arena Score	76.41
Accuracy	79.33%
Cost	$0.4992 / 1K queries ($4.19 total)
Robustness	81.43

Methodology

1. Three-head prompt classifier

v2 replaces v1's single scalar complexity score with a multi-head non-autoregressive prompt classifier (a ModernBERT-large encoder with a custom classification head) fine-tuned on internal telemetry to predict three attributes per prompt in a single forward pass:

• Difficulty tier: easy / medium / hard
• Task: 10 task classes (e.g., math reasoning, coding, expert knowledge QA, translation)
• Domain: 9 subject domains (e.g., Science, Literature, History, Technology)

The three predictions form a composite routing key tier|task|domain. Prompts exceeding 4,000 characters are truncated (head + tail preservation) prior to classification to maintain strict routing latency bounds.

2. Prior-telemetry routing index

Each tier|task|domain key maps to a ranked list of candidate models in a static routing index. Candidate rankings are pre-computed strictly from internal production evaluation logs and offline calibration benchmarks (measuring empirical accuracy and realised token cost per slice), prioritizing the lowest-cost deployment among models within an equivalent quality band.

When a specific slice has insufficient prior support, lookup backs off hierarchically (tier|task|domain → tier|task → tier → *) until a statistically validated routing entry is resolved.

3. Serving pool

Fourteen models are registered in router_inference/config/PSAI-Auto-Router-v2.json. Across the 8,400 full split queries, the router dispatched traffic across six active models:

Canonical Model Name (prediction)	Queries Routed	Share
gemini-3-flash-preview	6,377	75.92%
gpt-4o-mini	581	6.92%
gemini-3.8-flash	505	6.01%
google/gemini-3.1-flash-lite	489	5.82%
qwen/qwen3-next-80b-a3b-instruct	395	4.70%
gemini-3.1-pro-preview	53	0.63%
Total	8,400	100.00%

Reproducibility & Evaluation

Because the routing endpoint resides on private infrastructure, the CI workflow evaluates the pre-computed prediction artefacts directly:

• Routing decisions (prediction): Every row in both PSAI-Auto-Router-v2.json (8,400 queries) and PSAI-Auto-Router-v2-robustness.json (420 perturbed queries) reflects a live, uncached routing decision produced by the v2 classifier endpoint.
• Model generations (generated_result): In accordance with RouterArena's response-reuse protocol, whenever v2 selected a (query, model) pair that already had a verified completion in cached_results/ or committed baseline predictions (including our v1 submission), that existing completion and token usage were reused (8,038 rows). The remaining 362 (query, model) pairs not covered by existing cache artefacts were executed live against the respective model endpoints. All 420 rows in the robustness split also include populated generated_result payloads (417 live, 3 cached) for full schema consistency.

All metrics in the table above were produced by running RouterArena's local evaluation pipeline (compute_scores.py) using the repository's native dataset verifiers and model_cost/model_cost.json pricing.

Compliance

• Zero Benchmark Leakage: The prompt classifier and routing index were trained and calibrated exclusively on internal Publicis Sapient data prior to evaluation. No RouterArena queries, ground-truth answers, accuracy scores, or split files (sub_10, full, or robustness) were used during training, threshold tuning, or index construction.
• Zero-Touch PR: Only the three submission files listed below are added.

Files Added

• router_inference/config/PSAI-Auto-Router-v2.json
• router_inference/predictions/PSAI-Auto-Router-v2.json
• router_inference/predictions/PSAI-Auto-Router-v2-robustness.json

[lalitkumarps]

/evaluate

[github-actions]

Router Evaluation Results

Router: PSAI-Auto-Router-v2
Dataset Split: full
Evaluator: main @ 9f6ae25

RouterArena Metrics

Metric	Value
RouterArena Score	0.7636
Accuracy	79.27%
Total Cost	$4.193242
Avg Cost per Query	$0.000499
Avg Cost per 1K Queries	$0.4992
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.8143

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	1.0000
Opt.Cost (Cost Efficiency)	1.0000
Opt.Acc (Accuracy vs Optimal)	1.0000

---

Evaluation completed by RouterArena automated workflow

[lalitkumarps]

@yl231 Can you review this and if accepted get this on leaderboard ?