Skip to content

docs(langsmith): document the search bar filter editor in evaluators, automations, dashboards, Gateway traces, and Insights - #7960

Draft
eric-langchain wants to merge 7 commits into
mainfrom
robin/v2-filter-docs
Draft

eric-langchain wants to merge 7 commits into
mainfrom
robin/v2-filter-docs

Conversation

@eric-langchain

@eric-langchain eric-langchain commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Why

Evaluators, automations, dashboard charts, the Gateway traces table, and Insights now use the search-bar filter editor (with scopes) when the V2 filter rollout is on. These pages only described the old Add filter chip builder. Each page now covers both experiences and links to Filter traces for syntax instead of repeating it.

Changes

  • online-evaluations-llm-as-judge.mdx: Trigger On (Threads/Traces/Runs), search bar steps, Saved filters, Clear, legacy conversion, corrected backfill labels.
  • online-evaluations-multi-turn.mdx: thread evaluators filter with Thread, Root run, and Any run scopes; Trace Count converts to turn_count; Thread-scope filters turn off backfill.
  • online-evaluations-code.mdx, online-evaluations-decision-models.mdx, evaluators.mdx: search bar notes and feedback filter examples (feedback.correctness.score:<0.5).
  • dashboards.mdx: per-metric search bar filters, Traces/Runs, saved filters, ratio operands; old chip steps kept and label fixes (+ Add Filter).
  • rules.mdx, webhooks.mdx: automation Trigger On tabs, search bar filters, Update legacy filters, Apply to Past … labels.
  • llm-gateway-traces.mdx: search bar in Gateway Traces.
  • insights.mdx: Insights Filters / Add filters (optional) filter box; group fields such as run_count are rejected on save.

Review notes

  • Insights depends on unreleased product changes (the Insights V2 filter backend and frontend). Don't publish that section before they ship. It also conflicts with the open continuous Insights docs PR, which rewrites insights.mdx.
  • Claims come from reading the product source, not from a running app. No screenshots added; decision-at-node.png (dashboards example) still shows the old filter UI.
  • The Tip on the LLM-as-a-judge page saying runs-table filters carry over to the evaluator may be stale; left in place.

Prepared with Robin.

Made by Robin · view thread · anthropic:claude-opus-5-5 (high)

@github-actions github-actions Bot added langsmith For docs changes to LangSmith internal labels Nov 20, 2026

@robin-agent robin-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Robin Review found 6 potential issues.

Open in Robin • View Robin trace

Comment on lines +179 to +184
The **Filters** field, and **Add filters (optional)** when you configure online Insights, has one of two experiences:

- **A filter box**: Type a query in the [search bar syntax](/langsmith/filter-traces#query-syntax), using the [run fields](/langsmith/filter-traces#run-fields). The box has no scope selector. Insights applies the query when you press `Enter` or click outside the box, and only if the query is valid. A filter saved with the chip-based editor appears translated into this syntax.
- **A chip-based filter editor**: Add each condition as a filter chip. The **Is Trace** condition is fixed and cannot be removed. For more information, see [Filter traces (ClickHouse)](/langsmith/filter-traces-in-application).

In either experience, LangSmith rejects a filter on group-level fields, such as `run_count` and `thread_idle_seconds`, when you save the config or create a report.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Insights filter box and save-time rejection not on main

Confirmed problem (blocking until the product change ships). On current langchainplus main, neither Insights surface has the filter box. Pages/InsightsJobConfig/components/SelectRunsSection.tsx always renders the chip editor (FilterDialogContent with lockedFieldIds={['is_root']}). Pages/OnlineInsights/ConfigFields.tsx always renders FilterBarOperandsV2 plus AddFilterButton. Neither one reads the V2 filter flag (useDashboardV2FiltersEnabled / enable_v2_filter_surfaces).

The group-field claim on line 184 also doesn't match main. The only check is in the report sampler: smith-backend/app/models/clio/generate.py (_filter_references_group_attrs, which fetch_runs_by_thread calls) raises InsightsReportError while the report job runs. Nothing rejects the filter when you save a config or create a report.

The PR description already says this depends on unreleased work. Hold this section, or split it into a follow-up PR, until the Insights V2 backend and frontend ship. When they do, confirm the Enter/blur behavior and the save-time rejection against the merged code.

(Refers to lines 179-184)


A filter in the **Thread** scope turns off backfill for the evaluator, so the evaluator applies only to new threads.

When you open a thread-level evaluator saved with the filter builder, LangSmith converts its **Trace Count** conditions to `turn_count` clauses in the **Thread** scope, and its other conditions to the **Any run** scope. A filter that combines **Trace Count** with other conditions in one expression is not converted automatically.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Combined Trace Count AND filters are converted, not rejected

Confirmed problem (major). The filter builder saves FQL, and FQL that joins Trace Count and run conditions with and(...) is split automatically. In components/InlineFilters/FiltersRevamp/applySavedView.ts, splitLegacyThreadFilter throws only when operator !== 'and' && threads.length && runs.length. ThreadEvaluationFilterBar.test.tsx checks that and(eq(name, "agent"),gte(run_count,2)) becomes turn_count:>=2 in the Thread scope plus the run predicates in Any run.

Conversion fails in two cases. One is an or/not mix of count and run predicates. The other is a filter already in search-bar text that mixes trace_count with other clauses. That case shows "Separate the legacy thread count and run conditions before editing" (EvaluatorCrudPane/utils/threadEvaluatorFilters.ts). As written, the sentence tells readers their most common legacy filter (count AND condition) needs manual work, which it doesn't.

Also, when trajectory evaluators are available, Threads keeps the filter builder, so no conversion happens at all (EvaluatorReuseSourceControls.tsx, legacyThreads).

(Refers to line 99)

Suggested change
When you open a thread-level evaluator saved with the filter builder, LangSmith converts its **Trace Count** conditions to `turn_count` clauses in the **Thread** scope, and its other conditions to the **Any run** scope. A filter that combines **Trace Count** with other conditions in one expression is not converted automatically.
When you open a thread-level evaluator saved with the filter builder, LangSmith converts its **Trace Count** conditions to `turn_count` clauses in the **Thread** scope, and its other conditions to the **Any run** scope. A filter that combines **Trace Count** with other conditions using OR is not converted automatically.


Use filters or sampling to control evaluator cost. For example, evaluate only threads under *N* turns or sample 10% of all threads.

With the search bar, click **Filter** and choose a scope for each condition. For details, refer to [Filter threads with the search bar](#filter-threads-with-the-search-bar). With the filter builder, filter on **Trace Count** to limit thread length.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Trace Count missing from trajectory filter builder

Confirmed problem (major). The page says that where trajectory evaluators are available, Threads keeps the filter builder (line 89). In that case the builder is TrajectoryFilterBar: EvaluatorReuseSourceControls.tsx renders EvaluatorReuseThreadFilter with legacyTrajectory, which sets isTrajectory. components/FilterBar/TrajectoryFilterBar.tsx offers only is_root, name, run_type, and metadata. Trace Count (run_count) is only in the plain thread filter bar.

Readers in that state, which the panel shows when the V2 filter rollout is on and trajectory is available, won't find Trace Count. State that condition, or limit the sentence to panels without trajectory support.

(Refers to line 53)


- **Single metric**: one **+ Filter** in the **Filter & group** panel (the `where` slot). It applies to that metric.
- **Multiple metrics or a ratio**: each metric (or ratio) gets its own **+ Filter** inline in its card under **Pick a metric**. There is no separate chart-wide filter.
- **Traces and Runs tabs above a search bar with a scope selector**: Refer to [Filter with the search bar](#filter-with-the-search-bar).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Dashboard search bar is hidden until + Filter is clicked

Suggestion. In Pages/SingleDashboard/crud/DashboardV2FilterControls.tsx, the search bar is collapsed behind a + Filter button until the metric has a filter (showBar = expanded && !!sessionId). The Traces/Runs tabs are always visible. A reader with an unfiltered metric sees no search bar, and the legacy ratio numerator/denominator also has a + Filter button. Use the tabs as the signal for which experience applies.

(Refers to line 103)

Suggested change
- **Traces and Runs tabs above a search bar with a scope selector**: Refer to [Filter with the search bar](#filter-with-the-search-bar).
- **Traces and Runs tabs in each metric card**: Refer to [Filter with the search bar](#filter-with-the-search-bar).


- **Traces** and **Runs**: Choose what the metric counts. **Traces**, the default, matches root runs only. **Runs** matches any run, including child runs.
- **Filter**: Opens the search bar. The scope selector sets which runs a query matches: **Traces** offers the **Root run** and **Any run** scopes, and **Runs** adds the **Single run** scope. For the query syntax and the fields you can filter on, refer to [Query syntax](/langsmith/filter-traces#query-syntax) and [Fields](/langsmith/filter-traces#fields). For what each scope matches, refer to [Choose what a filter matches](/langsmith/filter-traces#choose-what-a-filter-matches).
- **Saved filters**: Lists the [saved filters](/langsmith/filter-traces#save-a-filter) of each selected tracing project, grouped by project. Selecting one replaces the metric's filter and its **Traces** or **Runs** selection. Saved filters that use the thread scope are not listed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Saved filters list drops more than thread-scope views

Suggestion. Pages/SingleDashboard/utils/dashboardFilterEditor.ts skips more than thread views. It skips a view that is not type === 'runs' or that has a thread_filter_string, a view that fails validateDashboardSeriesFilters, and a view whose translation throws. DashboardV2FilterControls.tsx then removes those views from the menu. Saying that only thread-scope filters are left out suggests every other saved filter appears.

(Refers to line 125)

Suggested change
- **Saved filters**: Lists the [saved filters](/langsmith/filter-traces#save-a-filter) of each selected tracing project, grouped by project. Selecting one replaces the metric's filter and its **Traces** or **Runs** selection. Saved filters that use the thread scope are not listed.
- **Saved filters**: Lists the [saved filters](/langsmith/filter-traces#save-a-filter) of each selected tracing project, grouped by project. Selecting one replaces the metric's filter and its **Traces** or **Runs** selection. Only run-level saved filters that a chart can use are listed.

Comment thread src/langsmith/rules.mdx
- **Saved filters**: Applies a [saved filter](/langsmith/filter-traces#save-a-filter) from the tracing project or agent environment. Selecting one replaces the rule's filter and switches to the tab the saved filter was created for.
- **Clear**: Removes the rule's filter.

When you create a rule from a tracing project or agent environment, the form starts with the filter and the **Threads**, **Traces**, or **Runs** selection currently applied to its table.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Rule form inherits table filter only from the runs tab

Suggestion. In components/Rules/SessionRulesCrudFormV2.tsx, table filters seed a new rule only when there is no initialData and the project's URL tab is the runs table (null, '0', or 'runs'). Otherwise the form starts empty. The Threads/Traces/Runs selection is passed through only when the revamped panel is on (Project.tsx, runTableViewMode={revampPanelEnabled ? tableViewMode : undefined}). Without the panel, the selection comes from is_root:true and is never Threads. Consider limiting this sentence to rules created from the runs table.

(Refers to line 91)

@robin-agent robin-agent Bot changed the title docs(langsmith): document the search bar filter editor in evaluators, automations, dashboards, runs tables, and Insights docs(langsmith): document the search bar filter editor in evaluators, automations, dashboards, Gateway traces, and Insights Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Mintlify preview branch generated: preview-robin-v2-filter-docs-a3fd64b3

Site preview: https://langchain-5e9cc07a-preview-robin-v2-filter-docs-a3fd64b3.mintlify.site

Important

Preview links may take a few minutes to start working while the deployment finishes.

Changed documentation pages (preview deep links):

Only the top 5 changed markdown files by diff size are listed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal langsmith For docs changes to LangSmith

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant