Replace curated Dashboard filters with inferred Variables and batched option queries #447

Depends on: #427

Blocks the Variables assignment part of #428. Panel assignment in #428 may be implemented independently.

Supersedes the curated-filter/provider direction and #450.

Phases

• 1 — Hard removal and Variables foundation
• 2 — Controls and batched option execution
• 3 — Assignment, examples, and integration — unblocked: Add Library drag assignment to Dashboard panels and variables #428's drag/drop item shipped in PR feat(#428): assign a Library query to a Dashboard by dragging it #484 (merged 2026-07-27); six items remain, see below

Owner decisions

The existing curated/free filter model is experimental and must be removed rather than adapted.

The replacement is a Dashboard-local Variables model:

• variables are inferred from {name:Type} placeholders in queries owned by Dashboard panels;
• the variable name is matched exactly and case-sensitively;
• all declarations of the same name in one Dashboard must use the same ClickHouse type;
• a variable may have optional Dashboard-local option SQL;
• empty option SQL means the variable uses an automatic direct-input control;
• configured option SQL produces exactly two String columns: value and visible label;
• configured option queries are compiled into one UNION ALL request per refresh;
• option SQL may not contain {name:Type} parameters in this issue;
• cascading/dependent variables are explicitly out of scope;
• no production-grade migration of experimental curated-filter documents is required.

Phase 1 is intentionally subtractive: remove the complete old curated-filter/provider implementation before building the new runtime, so later agents cannot accidentally extend the abandoned architecture.

Goal

Make Dashboard variables derive directly from the panel SQL that consumes them.

Example panel query:

SELECT
    event_time,
    amount
FROM sales
WHERE country = {country:String}
  AND city = {city:String}
  AND event_date >= {from:Date}
  AND event_date <= {to:Date}

The Dashboard derives:

country : String
city    : String
from    : Date
to      : Date

The user may optionally provide option SQL for country and city. from and to may remain direct inputs.

There are no persisted filter objects, provider objects, target-panel lists, filter-owned saved queries, filter roles, or query-name-based bindings.

Terminology

Inferred variable

A distinct exact variable name found in one or more panel-owned Dashboard queries.

Variable configuration

Dashboard-local optional SQL stored under a variable name. It provides predefined values for that variable.

Active variable

An inferred variable whose declarations have one consistent type.

Conflicted variable

An inferred variable name declared with two or more different types.

Orphaned configuration

Stored variable configuration whose name is no longer referenced by any Dashboard panel query.

Persisted model

Store only Dashboard-local option-query configuration. Exact generated type names may follow repository conventions, but the semantics are:

interface DashboardVariableConfig {
  sql: string;

  // Informational only, used to display an orphan after its last declaration
  // disappears. A newly inferred declaration always wins.
  lastKnownType?: string;
}

interface DashboardDocument {
  // Existing fields...
  variableConfigs?: Record<string, DashboardVariableConfig>;
}

Do not persist:

• a second variable/filter ID;
• a label separate from the variable name;
• panel targets;
• control type;
• visualisation type;
• selection mode;
• saved-query ID;
• ownership role;
• provider/source metadata.

The authoritative variable name and type come from panel query placeholders.

Current selected values are Dashboard runtime/session state, not new document configuration introduced by this issue.

Use a clear next document/schema version. Generated schemas, codecs, validators, fixtures, examples, and types must move together.

Variable inference

For every panel tile in a Dashboard:

1. resolve its owned query;
2. parse all {name:Type} placeholders from the executable SQL;
3. record variable name, declared type, Dashboard ID, tile ID, query ID, and source location where available;
4. aggregate declarations by exact variable name.

Adding or removing a panel query automatically updates the effective Variables list. Creating a variable row is never a separate user action.

Type consistency

These declarations are compatible:

-- Panel A
WHERE country = {country:String}

-- Panel B
WHERE billing_country = {country:String}

These declarations create a conflict:

-- Panel A
WHERE customer_id = {customer_id:UInt64}

-- Panel B
WHERE customer_id = {customer_id:String}

A conflicted variable:

• renders as one Variables-tree row;
• shows every distinct declared type;
• is marked with error styling, an error icon, and accessible error text;
• lists the conflicting panel/query usages in its hover diagnostic;
• does not render a working Dashboard control;
• is excluded from option-query execution;
• blocks only panels that require that variable;
• resolves automatically when panel declarations become consistent.

Do not rely on colour alone.

Variables hierarchy

Replace the old filter hierarchy with a Variables subtree:

Dashboard
  Variables · 5
    country : String
    city : String
    from : Date
    customer_id : String | UInt64   error
    region : String                 unused   [trash]
  Panels · 3

Requirements:

• show every distinct inferred variable name;
• show the inferred ClickHouse type next to the name;
• a conflicted name counts as one row and displays all distinct types;
• include preserved orphaned configurations;
• the group count is distinct inferred names plus orphaned configuration names;
• row search includes variable name, type, and stored SQL where practical;
• clicking anywhere on a row opens that variable's option SQL (a main-editor tab since Remove the variable drawer and edit option SQL in a dedicated main-editor tab #457);
• do not add an overflow (...) menu;
• show a trailing trash icon only for orphaned configurations;
• clicking trash must not also open the variable's tab;
• active and conflicted inferred variables never show trash.

Conflict tooltip

Example:

Variable “customer_id” has incompatible types:

Orders panel:  {customer_id:UInt64}
Revenue panel: {customer_id:String}

Orphaned configuration

When the final panel reference to a configured variable disappears:

• preserve its SQL;
• retain its last inferred type as lastKnownType for display only;
• show it in the Variables subtree with warning styling, warning icon, and unused status;
• hide it from the normal Dashboard controls;
• exclude it from the generated batch query;
• allow opening its editor;
• show a trailing trash icon.

Tooltip:

Variable “region” is not referenced by any Dashboard panel.
Its option SQL is preserved but will not be executed.

Deleting an orphan removes only variableConfigs.region, including SQL and lastKnownType. It does not edit panel queries. Require confirmation because SQL will be lost.

If no inferred declaration and no stored configuration remain, remove the row immediately.

Variable SQL editor

Superseded by #457 (shipped). Phase 1 delivered this as a dedicated right-side
drawer. That was the wrong direction — it duplicated the editing machinery the
application already has — so #457 deleted the drawer and moved option SQL into the
main editor as a dedicated Variable: <name> tab, identified by
(dashboardId, variableName). The requirements below still hold except where
noted; "the editor" now means that tab, and the drawer-local Run/Test action is
gone (see #465).

Clicking any Variables row switches to the Query surface and opens that Dashboard
variable's option SQL in its own main-editor tab.

Requirements:

• a newly inferred variable opens with empty SQL;
• the editor identifies the variable name and inferred type — the tab is titled
  Variable: <name>; the type read-out went with the drawer (Remove the variable drawer and edit option SQL in a dedicated main-editor tab #457);
• the editor contains no panel visualisation selector;
• the editor contains no Dashboard role selector;
• SQL may be typed, pasted, or copied from a Library query by drag/drop integration in Add Library drag assignment to Dashboard panels and variables #428;
• saving blank SQL removes the active variable's stored option configuration and returns it to direct-input mode;
• editing an orphan remains possible until it is deleted;
• provide a Run/Test action that executes only this variable's option query and
  validates its result shape — the tab uses the ordinary Run action; re-hosting the
  result-shape validation on it is tracked in Validate dashboard-variable option SQL on editor Run #465 (Remove the variable drawer and edit option SQL in a dedicated main-editor tab #457);
• diagnostics belong to this variable rather than to a global provider system.

Option SQL contract

Configured option SQL must be one embeddable read query (SELECT or WITH ... SELECT) and return zero or more rows with exactly two columns:

column 1: String value
column 2: String visible label

Column position defines meaning. Column names do not.

Example:

SELECT
    toString(country_id),
    country_name
FROM countries
ORDER BY country_name

If value and label are the same:

SELECT
    environment,
    environment
FROM environments
ORDER BY environment

Reject or diagnose locally:

• zero or one result column;
• three or more result columns;
• either result column not being String;
• multiple SQL statements;
• non-embeddable output clauses such as FORMAT or INTO OUTFILE;
• any {name:Type} placeholder in option SQL;
• syntax or execution errors.

Trailing semicolon handling must be deterministic: strip one harmless final semicolon before embedding or reject it consistently.

Duplicate values collapse deterministically in application code, preserving the first returned row and label. Option order follows query result order.

Use the application's bounded result policy; option queries must not create an unbounded client-side result.

No cascading variables in this issue

Option SQL must contain no Dashboard/query parameters.

Reject this for now:

SELECT toString(id), name
FROM cities
WHERE country = {country:String}

Diagnostic:

Variable option queries cannot reference Dashboard variables yet.

This deliberately avoids dependency graphs, cycle detection, upstream waiting states, invalidating downstream selections, and partial batch execution.

Cascading variables may be designed in a separate follow-up issue.

Batched option-query execution

During Dashboard refresh, include every variable that is:

• inferred;
• type-consistent;
• non-orphaned;
• configured with non-empty, locally acceptable option SQL.

Compile one ClickHouse query:

SELECT 'country' AS __variable_name,
       tupleElement(tuple(*), 1) AS __variable_value,
       tupleElement(tuple(*), 2) AS __variable_label
FROM (
    SELECT toString(country_id), country_name
    FROM countries
) LIMIT 1001

UNION ALL

SELECT 'city' AS __variable_name,
       tupleElement(tuple(*), 1) AS __variable_value,
       tupleElement(tuple(*), 2) AS __variable_label
FROM (
    SELECT toString(city_id), city_name
    FROM cities
) LIMIT 1001

Corrected while shipping phase 2. This snippet originally read
SELECT 'country' AS __variable_name, * FROM ( … ). That cannot be read
correctly: the streaming transport sends each row as a JSON object keyed by
column name, so two identically-named columns collapse on parse — and
SELECT environment, environment FROM environments, this issue's own
value==label example, is exactly that shape. In a UNION ALL the result column
names come from the FIRST branch alone, so one branch with a duplicate name
silently rewrote a LATER branch's value into its label (reproduced on ClickHouse
26.6). Taking the user's columns BY POSITION is what this issue actually
specifies, and it yields names that cannot collide. tuple/tupleElement exist
since ClickHouse 20.4; the positional wire format that would have avoided the
SQL change first shipped in 25.2 and was therefore rejected. The per-branch
LIMIT is the bounded-results policy applied per variable, so one long list
cannot crowd out the variables after it. See the ADR-0003 phase-2 addendum.

Requirements:

• generate branch tags from exact variable names using safe SQL literal escaping/parameters;
• branch order is deterministic and follows Variables ordering;
• user SQL is always nested as a subquery;
• all branches are compatible because user queries return exactly two String columns;
• parse the response by column position as variable name, value, label;
• partition rows by exact variable name;
• execute no options request when no valid configured variables exist;
• normal operation uses one request regardless of the number of configured variables.

A combined query failure is a batch-level failure in this issue. Show the failure in the Variables/filter-control area and make option-backed controls unavailable. Running ONE variable's SQL on its own — in its main-editor tab (#457) — is the diagnostic path. Do not automatically fall back to N separate runtime queries in the initial implementation.

Dashboard controls and binding

A Dashboard variable automatically binds to every panel query on the same Dashboard that declares the exact same variable name.

There is no persisted target list.

Variable without option SQL

Render a direct-input control inferred from its ClickHouse type, for example:

• String -> text input;

• Date -> date input;

• DateTime / DateTime64 -> datetime input;

• integer/float types -> numeric input;

• Bool -> checkbox;

• unsupported compound types such as Array, Tuple, Map, or Nested -> local unsupported-type diagnostic in this issue.

  Narrowed by Restore the searchable multi-select for Array(scalar T) Dashboard variables with option SQL #468 (shipped). An Array of a SCALAR that carries option SQL
  now renders the searchable multi-select instead — several option rows combine
  into one bound array. The unsupported-type diagnostic still stands for
  Tuple, Map, Nested, a nested Array(Array(T)), and for an
  Array(scalar T) with no option SQL configured.

Variable with option SQL

Render one single-select control from the batched result rows.

Rules:

• do not silently select the first row;
• a variable begins unset unless existing runtime behaviour already supplies a value explicitly;
• allow clearing a selection back to unset;
• panels wait rather than execute when a required variable is unset;
• changing a variable reruns only panels that reference that variable;
• option-query refresh remains one compiled request for all configured variables in the initial implementation.

Conflicted and orphaned variables do not render active controls.

Copy/paste and #428 integration

Panel assignment in #428 remains unchanged: copying a Library query to Panels creates a panel-owned saved-query copy and tile.

Variable assignment is different:

Library query
  -> drag onto an existing Variables row
  -> copy only its SQL text into dashboard.variableConfigs[variableName].sql

Requirements:

• the target variable must already exist through panel-query inference;
• do not create a saved-query clone for option SQL;
• do not create a filter/query role;
• do not store a source query ID or live Library reference;
• later Library edits do not mutate the Dashboard variable SQL;
• dropping or pasting parameterised SQL produces the no-cascading diagnostic;
• the same Library query may separately be copied to a panel and copied as SQL text to a variable.

No configuration dialog, parameter picker, target picker, default picker, selection-mode picker, or provider mapping is needed.

Remove the old curated-filter system completely

Phase 1 must remove, not deprecate or wrap, code whose only purpose is the abandoned filter/provider design.

Remove from persisted models, generated schemas, validators, codecs, imports/exports, and migrations:

• curated/free DashboardFilter definitions;
• filter IDs and source query IDs;
• targets and selection-mode fields;
• filter-owned saved-query references;
• saved-query filter ownership role;
• shared filter-source allowances;
• filter-provider/source metadata.

Remove from runtime and application services:

• implicit filter synthesis from Library or favourite queries;
• provider discovery by output-column name;
• global provider/helper collection;
• provider merging and duplicate-provider arbitration;
• identical-provider collapse;
• source execution deduplication for several filters;
• one query providing several filter arrays;
• array/multiselect parsing and state;
• per-filter panel targets;
• old filter attribution and ownership commands;
• old filter import/export dependency closure.

Remove from UI:

• the old Dashboard curated-filter visualisation/container;
• rendering filter-role queries as panels or previews;
• filter-role badges and icons used only by the old model;
• old add/edit/filter configuration dialogs;
• old free-filter hierarchy decisions and rows;
• provider/duplicate/multiselect toasts and diagnostics;
• any fallback copy telling users to select a Filter panel type.

Remove or rewrite:

• old examples and demo workspaces;
• fixtures and snapshots;
• CSS and menu items;
• ADR/design text;
• unit, integration, and E2E tests for the old topology.

There must be no compatibility branch that allows both old filters and new Variables at runtime.

Panel visualisation selector cleanup

Filter is not a panel visualisation and no longer exists as a saved-query role.

Remove from both Dashboard view and the panel-type selector:

• the Filter option;
• the Dashboard role category;
• category headings such as Preview…, Panel, and Dashboard role when they no longer distinguish selectable kinds;
• any rolePreviewView or equivalent mapping;
• filter-specific panel previews.

The selector becomes a flat list of actual panel visualisations only.

Migration policy

There is no production legacy to preserve.

Requirements:

• do not build a complex lossless migration from curated filters/providers;
• update bundled examples, test documents, and development fixtures to the new schema;
• experimental documents containing the removed filter representation may be rejected as unsupported and recreated/reset;
• document the development reset path where needed;
• do not retain old runtime types merely to decode obsolete test data.

Delivery phases

Phase 1 / PR 1 — Hard removal and Variables foundation

This phase must remove the old architecture before the new option runtime is introduced.

Implement:

• next document/schema version with variableConfigs;
• complete removal of curated-filter/provider persisted types and saved-query filter role;
• complete removal of old filter runtime, visualisation, selector, examples, tests, and documentation;
• flat panel visualisation selector containing only real visualisations;
• placeholder parser and variable aggregation from panel-owned queries;
• exact-name/type consistency validation;
• active, conflicted, and orphaned variable model;
• Variables subtree showing name : Type;
• conflict and orphan styling/tooltips;
• row click opens the variable SQL editor shell;
• orphan-only trash icon and atomic configuration deletion;
• editor supports saving/removing Dashboard-local SQL, but option execution may remain stubbed until Phase 2.

Phase 1 acceptance gate:

No reachable schema, runtime, UI, example, test, documentation, icon, menu, diagnostic, or generated type refers to curated filters, filter providers, filter-role saved queries, shared sources, arrays, or multiselect.

Note (#468). The "arrays, or multiselect" clause was a phase-1 REMOVAL gate
for the curated model, and it held. #468 later re-introduced a multiselect
control for Array(scalar T) variables on the inferred-variable model — no
curated filter, provider, target list or selection mode came back with it.

Phase 2 / PR 2 — Controls and batched option execution

Implement:

• direct controls inferred from variable types;
• single-select controls for configured variables;
• individual editor Run/Test;
• strict two-String-column validation;
• parameter-free option-query validation;
• deterministic UNION ALL compiler;
• one-request execution and response partitioning;
• unset/selection state;
• panel waiting and selective panel refresh;
• batch-level and per-variable diagnostics;
• bounded option results.

Phase 3 / PR 3 — Assignment, examples, and integration

Implement:

• Add Library drag assignment to Dashboard panels and variables #428 drag/drop of Library SQL onto existing Variables rows — shipped in PR feat(#428): assign a Library query to a Dashboard by dragging it #484 (merged 2026-07-27). Add Library drag assignment to Dashboard panels and variables #428 owns only this item; the six below stay here.
• paste/copy workflow in the variable editor;
• rewritten shipped examples using inferred variables and Dashboard-local option SQL;
• import/export coverage for variableConfigs;
• complete docs and E2E coverage;
• final dead-code audit for removed filter terminology and paths;
• reconcile or close superseded Attribute filter option columns per filter, not globally by name (duplicate-provider errors once sources are copied) #450 work.

Tests

Phase 1

• adding a panel query containing {country:String} creates one country : String tree row;
• several panels using {country:String} still produce one row;
• exact names remain distinct (country and Country);
• conflicting types produce one invalid row with all types and usages;
• resolving panel SQL clears the conflict automatically;
• removing the last usage of a configured variable creates an orphan row;
• removing the last usage of an unconfigured variable removes the row;
• trash appears only for orphaned configurations;
• deleting an orphan removes only its stored configuration;
• row click opens the variable editor;
• panel selector has no Filter option or category headings;
• old curated-filter/provider types and code paths are absent.

Phase 2

• empty SQL renders a direct control inferred from type;
• configured SQL renders a single-select control;
• exactly two String columns are accepted;
• wrong column counts and non-String columns fail locally;
• parameterised option SQL is rejected;
• no first-row auto-selection occurs;
• duplicate values keep the first label and stable order;
• two or more configured variables produce one deterministic batch request;
• no configured variables produce no batch request;
• orphaned and conflicted variables are excluded from the batch;
• response rows are partitioned by exact variable name;
• panels wait for required unset variables;
• changing a value reruns only panels that reference it;
• batch failure is visible and individual editor Run/Test remains available.

Phase 3

• dragging a Library query onto a variable copies SQL text only;
• no saved query, owner role, filter ID, or source reference is created;
• later Library edits do not change copied variable SQL;
• the same source can be copied independently to a panel and a variable;
• parameterised dragged SQL receives the no-cascading diagnostic;
• examples, import/export, and E2E flows use only the Variables model.

Acceptance criteria

• Dashboard Variables are derived from panel {name:Type} declarations.
• Exact variable names bind automatically to every matching panel query in the Dashboard.
• Same-name type conflicts are visible in the tree and block only affected variables/panels.
• Optional Dashboard-local SQL provides exactly two String columns per configured variable.
• All configured option queries execute through one generated UNION ALL request.
• Variables without option SQL use direct controls inferred from type.
• Cascading/parameterised option queries are rejected in this issue.
• Orphaned configurations are preserved, visibly marked, excluded from execution, and deletable with a trash icon.
• The Variables tree has only two row operations: open by clicking the row and delete for orphans.
• The old curated-filter/provider architecture is fully removed before the new runtime is implemented.
• Filter is absent from Dashboard rendering and the panel visualisation selector.
• Add Library drag assignment to Dashboard panels and variables #428 can copy Library SQL text onto an existing variable without creating another saved-query resource.

Non-goals

• Cascading/dependent variable option queries.
• Dependency graphs or cycle detection.
• Multiselect or Array-valued variable controls. Reversed by Restore the searchable multi-select for Array(scalar T) Dashboard variables with option SQL #468 (shipped)
  for the narrow case of an Array of a SCALAR that carries option SQL; every
  other container type remains a non-goal.
• One option query producing several variables directly; batching is generated by the application from independent per-variable SQL.
• Several provider queries merged into one variable.
• Per-panel target selection.
• Fuzzy or case-insensitive name matching.
• Manually creating, renaming, deleting, or changing the type of an active inferred variable from the tree.
• Production-grade migration of experimental curated-filter documents.
• Automatic N-query fallback after a combined options-query failure.
• Touch-specific assignment UI.