Repository navigation
docs(langsmith): document the search bar filter editor in evaluators, automations, dashboards, Gateway traces, and Insights - #7960
eric-langchain wants to merge 7 commits into
Conversation
Describe the search bar and the legacy Add filter chips in the experiment traces pane, an example's Linked Runs tab, the LLM Gateway Traces tab, and Insights config filters, linking to Filter traces for syntax.
… into robin/v2-filter-docs
…o robin/v2-filter-docs
| The **Filters** field, and **Add filters (optional)** when you configure online Insights, has one of two experiences: | ||
|
|
||
| - **A filter box**: Type a query in the [search bar syntax](/langsmith/filter-traces#query-syntax), using the [run fields](/langsmith/filter-traces#run-fields). The box has no scope selector. Insights applies the query when you press `Enter` or click outside the box, and only if the query is valid. A filter saved with the chip-based editor appears translated into this syntax. | ||
| - **A chip-based filter editor**: Add each condition as a filter chip. The **Is Trace** condition is fixed and cannot be removed. For more information, see [Filter traces (ClickHouse)](/langsmith/filter-traces-in-application). | ||
|
|
||
| In either experience, LangSmith rejects a filter on group-level fields, such as `run_count` and `thread_idle_seconds`, when you save the config or create a report. |
There was a problem hiding this comment.
🟠 Insights filter box and save-time rejection not on main
Confirmed problem (blocking until the product change ships). On current langchainplus main, neither Insights surface has the filter box. Pages/InsightsJobConfig/components/SelectRunsSection.tsx always renders the chip editor (FilterDialogContent with lockedFieldIds={['is_root']}). Pages/OnlineInsights/ConfigFields.tsx always renders FilterBarOperandsV2 plus AddFilterButton. Neither one reads the V2 filter flag (useDashboardV2FiltersEnabled / enable_v2_filter_surfaces).
The group-field claim on line 184 also doesn't match main. The only check is in the report sampler: smith-backend/app/models/clio/generate.py (_filter_references_group_attrs, which fetch_runs_by_thread calls) raises InsightsReportError while the report job runs. Nothing rejects the filter when you save a config or create a report.
The PR description already says this depends on unreleased work. Hold this section, or split it into a follow-up PR, until the Insights V2 backend and frontend ship. When they do, confirm the Enter/blur behavior and the save-time rejection against the merged code.
(Refers to lines 179-184)
|
|
||
| A filter in the **Thread** scope turns off backfill for the evaluator, so the evaluator applies only to new threads. | ||
|
|
||
| When you open a thread-level evaluator saved with the filter builder, LangSmith converts its **Trace Count** conditions to `turn_count` clauses in the **Thread** scope, and its other conditions to the **Any run** scope. A filter that combines **Trace Count** with other conditions in one expression is not converted automatically. |
There was a problem hiding this comment.
🟡 Combined Trace Count AND filters are converted, not rejected
Confirmed problem (major). The filter builder saves FQL, and FQL that joins Trace Count and run conditions with and(...) is split automatically. In components/InlineFilters/FiltersRevamp/applySavedView.ts, splitLegacyThreadFilter throws only when operator !== 'and' && threads.length && runs.length. ThreadEvaluationFilterBar.test.tsx checks that and(eq(name, "agent"),gte(run_count,2)) becomes turn_count:>=2 in the Thread scope plus the run predicates in Any run.
Conversion fails in two cases. One is an or/not mix of count and run predicates. The other is a filter already in search-bar text that mixes trace_count with other clauses. That case shows "Separate the legacy thread count and run conditions before editing" (EvaluatorCrudPane/utils/threadEvaluatorFilters.ts). As written, the sentence tells readers their most common legacy filter (count AND condition) needs manual work, which it doesn't.
Also, when trajectory evaluators are available, Threads keeps the filter builder, so no conversion happens at all (EvaluatorReuseSourceControls.tsx, legacyThreads).
(Refers to line 99)
| When you open a thread-level evaluator saved with the filter builder, LangSmith converts its **Trace Count** conditions to `turn_count` clauses in the **Thread** scope, and its other conditions to the **Any run** scope. A filter that combines **Trace Count** with other conditions in one expression is not converted automatically. | |
| When you open a thread-level evaluator saved with the filter builder, LangSmith converts its **Trace Count** conditions to `turn_count` clauses in the **Thread** scope, and its other conditions to the **Any run** scope. A filter that combines **Trace Count** with other conditions using OR is not converted automatically. |
|
|
||
| Use filters or sampling to control evaluator cost. For example, evaluate only threads under *N* turns or sample 10% of all threads. | ||
|
|
||
| With the search bar, click **Filter** and choose a scope for each condition. For details, refer to [Filter threads with the search bar](#filter-threads-with-the-search-bar). With the filter builder, filter on **Trace Count** to limit thread length. |
There was a problem hiding this comment.
🟡 Trace Count missing from trajectory filter builder
Confirmed problem (major). The page says that where trajectory evaluators are available, Threads keeps the filter builder (line 89). In that case the builder is TrajectoryFilterBar: EvaluatorReuseSourceControls.tsx renders EvaluatorReuseThreadFilter with legacyTrajectory, which sets isTrajectory. components/FilterBar/TrajectoryFilterBar.tsx offers only is_root, name, run_type, and metadata. Trace Count (run_count) is only in the plain thread filter bar.
Readers in that state, which the panel shows when the V2 filter rollout is on and trajectory is available, won't find Trace Count. State that condition, or limit the sentence to panels without trajectory support.
(Refers to line 53)
|
|
||
| - **Single metric**: one **+ Filter** in the **Filter & group** panel (the `where` slot). It applies to that metric. | ||
| - **Multiple metrics or a ratio**: each metric (or ratio) gets its own **+ Filter** inline in its card under **Pick a metric**. There is no separate chart-wide filter. | ||
| - **Traces and Runs tabs above a search bar with a scope selector**: Refer to [Filter with the search bar](#filter-with-the-search-bar). |
There was a problem hiding this comment.
🔵 Dashboard search bar is hidden until + Filter is clicked
Suggestion. In Pages/SingleDashboard/crud/DashboardV2FilterControls.tsx, the search bar is collapsed behind a + Filter button until the metric has a filter (showBar = expanded && !!sessionId). The Traces/Runs tabs are always visible. A reader with an unfiltered metric sees no search bar, and the legacy ratio numerator/denominator also has a + Filter button. Use the tabs as the signal for which experience applies.
(Refers to line 103)
| - **Traces and Runs tabs above a search bar with a scope selector**: Refer to [Filter with the search bar](#filter-with-the-search-bar). | |
| - **Traces and Runs tabs in each metric card**: Refer to [Filter with the search bar](#filter-with-the-search-bar). |
|
|
||
| - **Traces** and **Runs**: Choose what the metric counts. **Traces**, the default, matches root runs only. **Runs** matches any run, including child runs. | ||
| - **Filter**: Opens the search bar. The scope selector sets which runs a query matches: **Traces** offers the **Root run** and **Any run** scopes, and **Runs** adds the **Single run** scope. For the query syntax and the fields you can filter on, refer to [Query syntax](/langsmith/filter-traces#query-syntax) and [Fields](/langsmith/filter-traces#fields). For what each scope matches, refer to [Choose what a filter matches](/langsmith/filter-traces#choose-what-a-filter-matches). | ||
| - **Saved filters**: Lists the [saved filters](/langsmith/filter-traces#save-a-filter) of each selected tracing project, grouped by project. Selecting one replaces the metric's filter and its **Traces** or **Runs** selection. Saved filters that use the thread scope are not listed. |
There was a problem hiding this comment.
🔵 Saved filters list drops more than thread-scope views
Suggestion. Pages/SingleDashboard/utils/dashboardFilterEditor.ts skips more than thread views. It skips a view that is not type === 'runs' or that has a thread_filter_string, a view that fails validateDashboardSeriesFilters, and a view whose translation throws. DashboardV2FilterControls.tsx then removes those views from the menu. Saying that only thread-scope filters are left out suggests every other saved filter appears.
(Refers to line 125)
| - **Saved filters**: Lists the [saved filters](/langsmith/filter-traces#save-a-filter) of each selected tracing project, grouped by project. Selecting one replaces the metric's filter and its **Traces** or **Runs** selection. Saved filters that use the thread scope are not listed. | |
| - **Saved filters**: Lists the [saved filters](/langsmith/filter-traces#save-a-filter) of each selected tracing project, grouped by project. Selecting one replaces the metric's filter and its **Traces** or **Runs** selection. Only run-level saved filters that a chart can use are listed. |
| - **Saved filters**: Applies a [saved filter](/langsmith/filter-traces#save-a-filter) from the tracing project or agent environment. Selecting one replaces the rule's filter and switches to the tab the saved filter was created for. | ||
| - **Clear**: Removes the rule's filter. | ||
|
|
||
| When you create a rule from a tracing project or agent environment, the form starts with the filter and the **Threads**, **Traces**, or **Runs** selection currently applied to its table. |
There was a problem hiding this comment.
🔵 Rule form inherits table filter only from the runs tab
Suggestion. In components/Rules/SessionRulesCrudFormV2.tsx, table filters seed a new rule only when there is no initialData and the project's URL tab is the runs table (null, '0', or 'runs'). Otherwise the form starts empty. The Threads/Traces/Runs selection is passed through only when the revamped panel is on (Project.tsx, runTableViewMode={revampPanelEnabled ? tableViewMode : undefined}). Without the panel, the selection comes from is_root:true and is never Threads. Consider limiting this sentence to rules created from the runs table.
(Refers to line 91)
|
Mintlify preview branch generated: Site preview: https://langchain-5e9cc07a-preview-robin-v2-filter-docs-a3fd64b3.mintlify.site Important Preview links may take a few minutes to start working while the deployment finishes. Changed documentation pages (preview deep links):
Only the top 5 changed markdown files by diff size are listed. |
Why
Evaluators, automations, dashboard charts, the Gateway traces table, and Insights now use the search-bar filter editor (with scopes) when the V2 filter rollout is on. These pages only described the old Add filter chip builder. Each page now covers both experiences and links to Filter traces for syntax instead of repeating it.
Changes
online-evaluations-llm-as-judge.mdx: Trigger On (Threads/Traces/Runs), search bar steps, Saved filters, Clear, legacy conversion, corrected backfill labels.online-evaluations-multi-turn.mdx: thread evaluators filter with Thread, Root run, and Any run scopes; Trace Count converts toturn_count; Thread-scope filters turn off backfill.online-evaluations-code.mdx,online-evaluations-decision-models.mdx,evaluators.mdx: search bar notes and feedback filter examples (feedback.correctness.score:<0.5).dashboards.mdx: per-metric search bar filters, Traces/Runs, saved filters, ratio operands; old chip steps kept and label fixes (+ Add Filter).rules.mdx,webhooks.mdx: automation Trigger On tabs, search bar filters, Update legacy filters, Apply to Past … labels.llm-gateway-traces.mdx: search bar in Gateway Traces.insights.mdx: Insights Filters / Add filters (optional) filter box; group fields such asrun_countare rejected on save.Review notes
insights.mdx.decision-at-node.png(dashboards example) still shows the old filter UI.Prepared with Robin.
Made by Robin · view thread · anthropic:claude-opus-5-5 (high)