Skip to content

docs(langsmith): document continuous Insights - #7885

Open
Andy Young (ayoung19) wants to merge 5 commits into
mainfrom
andy/online-insights-ga-docs
Open

Andy Young (ayoung19) wants to merge 5 commits into
mainfrom
andy/online-insights-ga-docs

Conversation

@ayoung19

@ayoung19 Andy Young (ayoung19) commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Insights now runs continuously on LangSmith Cloud, and on self-hosted v0.18 or later with SmithDB. Each insight builds categories from recent traces, then categorizes new conversations as they arrive. Before, Insights generated one-time or scheduled batch reports, and the Insights page still described only batch reports.

  • insights.mdx: rewritten for continuous Insights. It covers:

    • How insights work: category builds, sampling, thread idle time, and the three automation rules each insight uses.
    • Creating an insight from the UI, SDK, or CLI.
    • Reading an insight: top categories, categories over time, the breakdown table, statuses, and what happens to configurations from earlier batch reports.
    • Configuring an insight: runs and sampling rate, models, categories, and attributes.
    • Deleting an insight, refreshing categories, sharing, Slack updates, and cost drivers.
    • Limitations: 100 insights per agent environment, and up to three category builds at a time per workspace.

    Every workspace is agent-based on Cloud and on self-hosted v0.18, so the page opens Insights from the agent switcher and refers to agent environments rather than tracing projects.

  • New insights-batch-reports.mdx: the previous batch report content, for self-hosted v0.18 or later installations that query ClickHouse instead of SmithDB. It uses the agent and environment navigation from docs(langsmith): remove the self-hosted split from agent docs #7523, which edited the old Insights page after this branch was cut. The merge kept this branch's version of that page, so docs(langsmith): remove the self-hosted split from agent docs #7523's guidance moves here. The screenshot and GIF showed the old project-based UI, so this PR removes them and deletes the two image files. Listed in nav after Insights.

  • langsmith-cli.mdx: "Manage insights" explains what configurations and jobs mean under continuous Insights:

    • schedule_cron jobs do not run.
    • insights run refreshes categories.
    • config update applies to conversations analyzed afterward.
    • config delete stops categorization.
    • The CLI cannot set the sampling rate or Slack updates.
  • self-host-insights-chat.mdx: a new section on when self-hosted runs continuous Insights (v0.18 or later, with SmithDB queries and ingestion enabled for every workspace) and when it generates batch reports.

  • self-host-smithdb-features.mdx: a row for continuous Insights.

The old cost estimate is removed. It assumed a one-time report capped at 1,000 threads, which no longer applies. The Insights page describes what drives cost instead.

Review carefully

  • Behavior statements were checked against product source: statuses, Slack update contents, which attributes appear in the breakdown table, the rule and concurrency limits, the self-hosted SmithDB requirement, and the SDK saving a passed API key as a workspace secret.
  • The batch page no longer documents self-hosted versions before v0.18, consistent with docs(langsmith): remove the self-hosted split from agent docs #7523.
  • Open questions:
    • Can customers still open past batch reports on Cloud, and where?
    • generate_insights() uploads chat histories "to a new LangSmith project". How that maps to an agent on v0.18 is unverified.

Test plan

  • make lint_prose clean on all changed pages.
  • Links and anchors into the Insights pages resolve after merging main.
  • make check-mdx and make broken-links-with-anchors pass in CI (Documentation Build & Link Check).

Refs DOC-1996

Prepared with Claude Code.

Rewrite the Insights page for continuous Insights, which builds
categories from recent traces and then categorizes new conversations as
they arrive. Move the batch report docs to their own page for
self-hosted installations that still generate batch reports, and update
the CLI and self-hosted pages to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix statements about continuous Insights that did not match product
behavior: which attributes appear in the breakdown table, what the Off
status means, what Slack updates contain, the digest time zone, and
which sharing options need SmithDB. Add what readers need to plan
around: three automation rules per insight, three concurrent builds per
workspace, the SDK saving the passed key as a workspace secret, the CLI
not setting a sampling rate, and the full self-hosted SmithDB
requirement.

Also fix style on the moved batch-report page (future tense,
contractions, em dashes, report capitalization, gerund headings), move
the delete steps out of "Add attributes", and update stale "report"
wording in the CLI section.
…ga-docs

# Conflicts:
#	src/langsmith/insights.mdx
Self-hosted LangSmith v0.18 makes every workspace agent-based, so drop
the agent-based versus tracing-project split from the Insights pages.
Open Insights from the agent switcher, and refer to agent environments
instead of tracing projects.

The batch reports page now covers v0.18 or later installations that
query ClickHouse, and no longer documents earlier versions. It carries
over the environment guidance from #7523, which the merge dropped
because that text moved here.

Remove the batch page's screenshot and GIF, which showed the old
project-based UI, along with the sentence that referred to the GIF.
@github-actions

Copy link
Copy Markdown
Contributor

Mintlify preview branch generated: preview-andy-online-insights-ga-f48a96ea

Site preview: https://langchain-5e9cc07a-preview-andy-online-insights-ga-f48a96ea.mintlify.site

Important

Preview links may take a few minutes to start working while the deployment finishes.

Changed documentation pages (preview deep links):

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal langsmith For docs changes to LangSmith

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants