Allow dashboard customization

[akshaylive]

Different runs may require different aggregations and visualizations that are native.

Specifically, given list of runs, how can we help customers customize the visualizations? Ex: pareto-optimal graphs, timelines, bar graphs etc? The data model can be owned by coder-eval and it can create default "views" that can be imported into custom dashboard.

coder-eval evalboard --template? Brainstorm here to customize dashbaord.

[bai-uipath]

Proposal

Three additive changes, no change to the scoring model or to run.json's existing shape:

1. Carry the criterion aggregate through to the dashboard. Thread the metric map into the run and task projections, then let the rollup and the daily chart select a metric by name, defaulting to pass rate. Selection is view state and lives in the URL, so there is no new config format.
2. Let any criterion emit a structured metric map. Only the two classification criteria produce one today. Making it part of the criterion contract is the general version of "custom aggregation", and it puts the definition next to the thing that computes it.
3. pass@k as a reduction over replicates. The replicate index is already in the data model, so this is a rollup rather than a new dimension.

Why this is the whole fix

Most of what is being asked for already exists and is thresholdable by name through suite_thresholds.

Every criterion on a dataset-backed task gets, for free: count, mean, median, std, min, max.

Label-emitting criteria (classification_match, skill_triggered) overlay on top of that:

• accuracy
• macro_f1, weighted_f1, micro_f1 (real micro pooling of tp/fp/fn, not an average of per-row ratios)
• precision.<label>, recall.<label>, f1.<label> for every label
• plus the full confusion matrix, per-label support, and label list in the aggregate details

What is missing is that none of it reaches the dashboard. The run and task projections carry the weighted score and status only, so every rollup and the daily chart can render pass rate and nothing else. From the outside that reads as "the framework only does pass/fail", even though the numbers are already sitting in run.json. That is a surfacing gap rather than a customization gap, and the two are worth separating before we design for the second one.

Minor: the suite_thresholds docstring advertises f1.macro, but the aggregator writes macro_f1. As written, f1.macro only resolves if a label is literally named macro.

Out of scope

Chart templating (evalboard --template, pareto plots, timelines, bar graphs). This is a visualization language: a schema, a renderer per chart type, validation, versioning, and docs to match. That is a large surface to own, it overlaps with tools people already point at exported data, and none of it pays off until the metrics are reachable. Worth revisiting once we can see which metrics people actually select.

Per-experiment dashboard config and post-run scripts. Config that travels with the run has no answer for cross-run views, where N runs on one chart arrive with N conflicting configs and something has to arbitrate. Declaring scripts to run after an experiment and rendering their output also adds an execution surface to a board fed from CI. If we want per-run authority later, the smaller version is to stamp metric descriptors rather than view config.

[uipreliga]

I am a bit concerned about "Selection is view state and lives in the URL" - does it mean we always start with a 'default view' and one has to click what metrics should be displayed? I would still think some kind of config file would be useful here.

I like "Label-emitting criteria (classification_match, skill_triggered)" - so we get it for free, but what happens if there is criteria overlap and we only want to select some of them?

Also, providing hook for post processing results after experiment would be useful too.

For charts, we could provide few simple, out of the box variants - what Nick mentioned; again, they could be saved in config file and when modified in UX, modification stored in your browser cache.

[akshaylive]

pass@k as a reduction over replicates

Yep. However, we need custom aggregators. pass@k reduction is also a sort of custom aggregation.

Let any criterion emit a structured metric map

This is very intensive change right?

Selection is view state and lives in the URL

I think the view should be tied to the eval board itself. Something like this coder-eval evalboard --configuration ...?

Out of scope: Chart templating (evalboard --template, pareto plots, timelines, bar graphs).

IMO this should be in scope. Different customers would want to have different dashboards. That's the meat of this issue.

[Sugarc0de]

Following up on our meeting, the chart Nick was referring to might look something like this: https://www.wiz.io/cyber-model-arena
Scatter of pass@k against cost / time / steps per solve, switchable, with an optional harness filter.