Repository navigation
docs(langsmith): reword project-scoped settings for agent environments - #7745
Merged
Merged
Conversation
Self-hosted LangSmith v0.18 moves every workspace to agents, so the docs no longer need to describe two kinds of workspace. - Rename the "Agent-based workspaces" nav group to "Agent Registry". - Delete the project-based, agent-based, and availability note snippets. - Turn the workspace migration page into a self-hosted v0.18 upgrade page. - Remove the application framing from the admin overview, resource hierarchy, and workload isolation pages. - Remove "agent-based workspace" qualifiers and project-based alternatives across the tracing and setup pages. - Update AGENTS.md to match. Hold until self-hosted v0.18 ships. Part of DOC-1797.
- Replace "tracing project, or agent environment" alternatives with agent environments across the evaluator, rules, Engine, dashboard, alert, and annotation queue pages. - Start the billing automation step from Tracing in the agent's environment, and select an Agent when creating an annotation queue. - Document the Agent resource tag key and its permission exception. - Move the self-hosted upgrade page to Self-hosted > Manage an instance and rename it to self-host-agent-migration, with a redirect from the old URL. Part of DOC-1797.
An agent environment's evaluator tab is labeled Online Evaluators, not Evaluators. Use that name wherever a page describes an agent environment, and name both tabs where a page covers datasets and environments. Also describe the evaluator spend widgets on those tabs without quoting the per-view chart titles, and select an agent environment instead of a tracing project in the decision model evaluator steps. Part of DOC-1797.
- The daily evaluator spend chart has no Project / Dataset toggle in an agent workspace, so describe it as broken down by evaluator. - Use the widget's title, Weekly evaluator spend limits, and describe each row as an evaluator and the agent environment or dataset it is attached to. - Say where the scoped widgets appear: a dataset's Evaluators tab and an agent environment's Online Evaluators tab on the Tracing page. Part of DOC-1797.
When an evaluator reaches its weekly spend limit, describe what stops in terms of the attached agent environment or dataset instead of a project. Part of DOC-1797.
- Correct the dashboard chart source labels for agent workspaces: the button reads "+ Select agent or dataset" and the tabs are Tracing and Datasets. - On the self-hosted upgrade page, say that the migration runs on the existing queue workers, and name LangSmith as the actor in three sentences. - Remove a contraction and a future tense on lines this PR changed. Part of DOC-1797.
Apply a persona review of the upgrade page: - Lead with the benefit, name the four environments, and note that new installations with authentication start with agents. - Put the no-downgrade rule in a warning, and move the one-minor-version rule into "Before you upgrade". - Describe the migration steps without internal job terms, add a Helm values snippet for pausing, and say that pausing does not end it. - Add a section on checking whether a workspace has migrated. - Spell out the identifier suffix, name the projects that stay as they are, and say what happens to tracing projects created later. - Use "agent switcher", fix two link labels, rename the sidebar entry, and drop the icon. Two TODO(verify) comments remain for open product questions. Part of DOC-1797.
- Rename trace-ingestion-project to create-agent-for-tracing, which says what the snippet does, and update the quickstart and the tracing tutorial that import it. - Replace the nonexistent Observability option with the real path: Connect > Connect an existing agent, which opens the Connect an agent dialog. Name both ways into the new agent menu: + Agent with All agents selected, or the + button in the agent switcher. - Link the create-an-agent section and the agent switcher, and tighten the closing paragraph. Part of DOC-1797.
Link the first mention of a deployment, and remove "Experiments stay experiments," which repeated the sentence before it. Part of DOC-1797.
… ABAC Replace the agent-based workspace qualifier that came in from main with a plain statement: the project resource type covers agent environments, and a policy for it applies to each environment separately. Part of DOC-1797.
- Upgrade page: replace the two TODO(verify) comments with what engineering confirmed. Tracing keeps working during the migration, and an authType: none install that later enables basic auth or SSO then runs the migration. New installations on v0.18 or later start with agents, with or without authentication. - Alerts: give the real path to the Alerts bell (agent switcher, Tracing, environment) and the Monitoring > Alerts route, and replace "Alerts > Create Alert" with "+ Alert" in the integration steps. - Remove leftover project alternatives in annotation-queues, evaluators, audit-logs, and deploy-to-cloud. Relabel the audit log "Tracing projects" row as "Agent environments" and add create_tracer_session. - Say what attach-user-feedback's linked requirements are, and drop a vague sentence in annotate-code. Part of DOC-1797.
…d keys - billing: rewrite default retention as per-environment steps, rename the section anchor, and remove the outdated retention screenshot - usage-and-billing, data-purging-compliance, evaluators, administration-overview: set retention per agent environment and update links to the renamed anchor - composite-evaluators-ui: start from an agent environment or dataset - create-account-api-key: drop legacy project upsert and destination sentences, link ABAC policies - dashboards: match the Monitoring > Analytics tab and the Workspace > Dashboards page, use environment labels, and note the Feedback Key limit
…ataset pages - Use "select your agent, go to Tracing, and select the environment" on the online evaluator, rules, insights, and dataset pages - Name the Online Evaluators tab on the Tracing page in decision-model, LLM-as-a-judge, and tuned evaluator steps - Note that Logs is in the far-right Actions column of the evaluators table, and tighten the backfill and log sentences - tuned-evaluators: add an evaluator to an agent environment, not a tracing project - manage-datasets-in-application: add data from an agent environment - Fix "+New", "click on", contractions, and future tense where touched
…d webhooks - Claude Code, Cowork, Cursor, and Codex: note that the plugins always send to the agent's Production environment, give the Tracing path, and scope the ABAC deny policy to the agent environment - set-up-resource-tags: list taggable resources by their picker labels, starting with Agent Environments - set-up-hierarchy: drop the unsupported ABAC claim and the upgrade pointer, and fix a paragraph break and tense - view-traces: open with the path to an environment's traces - webhooks: scope rules to the agent environment and describe session_id as the environment's tracer_session_id
…tting - online-evaluations-multi-turn: idle time is set on the agent environment and shown read-only under Advanced in the evaluator form, so step 6 checks it and points to Edit tracing settings. Use Trigger On for the Runs/Threads choice, split the source step, click Create, and tidy step formatting - online-evaluations-llm-as-judge: use Trigger On in the backfill steps - rules: describe the idle time as an agent environment setting
…nt environments - tuned-evaluators: attach to agent environments, and rename the two troubleshooting headings - administration-overview: Agent Environments scope row, retention and evaluator opt-out per environment, per-environment usage limits with the UI's Project label explained, and drop "create projects" from tracing keys - view-traces, manage-datasets-in-application: format selection persists per environment; rename the dataset headings and update inbound anchors - evaluator-spend: attached agent environments and datasets, and rename the override heading and anchor - online-evaluations-composite, annotation-queues-sdk: replace tracing project links; project_id is the environment's Tracing Project ID - Engine pages: one first step for Configure Engine, per-environment Engine configuration in byoc-migration, and the issues:create roles in engine-setup
- administration-overview: replace the resource tags vs. commit tags note with a pointer to commit tags, say the Project list shows each environment's tracer_session_name, and use "per-agent-environment" - engine-setup, engine: configuring Engine needs issues:update and the environment spend limit needs an Organization Admin; label the agent environment limit consistently - rules: say where to change the shared thread idle time - annotation-queues-sdk: give the Tracing path for project_id and thread_id - tuned-evaluators: Workspace Admins and Editors attach tuned evaluators - byoc-migration, manage-datasets-in-application: agent wording
Florence Morris (fjmorris)
left a comment
Contributor
Author
There was a problem hiding this comment.
Lauren Hirata Singh (@lnhsingh) ready for you to review
This was referenced Nov 16, 2026
Lauren Hirata Singh (lnhsingh)
approved these changes
Nov 16, 2026
Lauren Hirata Singh (lnhsingh)
left a comment
Collaborator
There was a problem hiding this comment.
nits
# Conflicts: # src/docs.json # src/langsmith/audit-logs.mdx # src/langsmith/deploy-to-agent-environment.mdx # src/langsmith/workload-isolation.mdx
…ations Apply the reviewer suggestions on the upgrade page, the agent creation snippet, the evaluator log steps, and the feedback version floors. Replace the audit log note about operations missing from stable self-hosted with a 0.18.0 entry in the version list. Refs DOC-1797
…tions Keep the short UI steps as a numbered list and move the idle time, model, prompt, and feedback settings into subsections linked from step 6, per review. Refs DOC-1797
# Conflicts: # src/langsmith/administration-overview.mdx # src/langsmith/billing.mdx # src/langsmith/data-purging-compliance.mdx # src/langsmith/evaluators.mdx # src/langsmith/usage-and-billing.mdx
Drop "as a compatibility value" from the tracer_session_name sentence on all six copies, link the agent variables directly in annotate-code, make the Connect an agent dialog the link text in agent-registry, and name the enabled value in self-host-agent-migration. Refs DOC-1797
# Conflicts: # src/langsmith/administration-overview.mdx
6 tasks done
Florence Morris (fjmorris)
added a commit
that referenced
this pull request
Nov 22, 2026
## Why Self-hosted LangSmith v0.18 moves every workspace to agents and environments, so the docs no longer need to describe two kinds of workspace. Today, many pages split their content between "agent-based" and "project-based" workspaces, offer a tracing project as an alternative to an agent environment, or carry a "Cloud only, not self-hosted" callout. After v0.18, all of these are wrong for every reader. Self-hosted v0.18.0 was released on November 20, 2026. This is the first of four stacked PRs for DOC-1797, which merge in order: this PR and PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on November 24 or 25. ## Changes - **Upgrade page:** rewrote the workspace migration page as "What changes in LangSmith v0.18", for self-hosted administrators: before upgrading, how the migration runs and how to pause it, checking whether a workspace has migrated, how tracing projects map to agents, and what changes in each workspace. Renamed it to `self-host-agent-migration` (with a redirect) and moved it to Self-hosted > Manage an instance, after "Upgrade an installation". - **Navigation:** renamed the "Agent-based workspaces" group to "Agent Registry", with the `agent-registry` page labeled "Overview". - **Workspace-type split removed:** deleted the `project-based-note`, `agent-based-note`, and `agent-availability-note` snippets. Across about 65 tracing, evaluator, and setup pages, removed "agent-based workspace" qualifiers, project-based alternatives, and `LANGSMITH_PROJECT` fallbacks, and replaced "tracing project, or agent environment" wording with agent environments. No "agent-based workspace" or "project-based" wording is left, apart from the old migration page redirect. - **UI labels checked in the product:** the evaluator tab is **Online Evaluators** on an agent environment, attachments are **Environments & Datasets**, the Engine limit is **Environment limit**, the dashboard chart source is **+ Select agent or dataset**, and the **Evaluators** page spend widgets match an agent workspace. Steps to an environment now consistently say "select your agent, go to **Tracing**, and select the environment". - **Errors already on `main`:** the create-an-agent snippet (renamed `create-agent-for-tracing`) named an **Observability** option that does not exist and now gives **Connect** > **Connect an existing agent**. The alerts steps now give the real path to **Alerts** and say **+ Alert**. The dashboards page named the chart button and tabs incorrectly, and now places custom dashboards under **Workspace** > **Dashboards**. - **Settings now per environment:** retention (`billing`, `usage-and-billing`, `data-purging-compliance`, `evaluators`), resource tags (the **Agent** tag key and the **Agent Environments** picker label), and the tracing plugins, which always send to the agent's **Production** environment. - **Reference links:** `audit-logs` relabels the operations row as "Agent environments", adds `create_tracer_session`, and lists v0.18 for `create_agent_environment` and `mint_agent_access_token`. Its `operations` link and the `abac` API link now point to the docs API reference instead of the Swagger UI. - **Smaller fixes:** `set-up-hierarchy` and `workload-isolation` anchors renamed to `#set-up-agents` and `#isolate-by-agent`, the 403 troubleshooting paragraph removed from `log-traces-to-agent`, `webhooks` describes `session_id` as the environment's `tracer_session_id`, `attach-user-feedback` states the v0.18 and SDK minimums, the `observability-quickstart` intro no longer points to compatibility instructions that the old snippet held, and `AGENTS.md` updates the navigation map. ## Review notes - **Upgrade page:** the migration behavior was checked against the final self-hosted v0.18.0 release (Helm chart `langsmith-0.18.0`, app version `0.18.2026112100`): the roughly 10-minute start delay that restarts do not reset, each workspace switching once its migration is verified, the stop after 24 quiet hours and a clean full check, `authType: none` installs not being migrated, and pausing with `config.agentMigration.enabled`, which defaults to `true` and is unchanged from the release candidates. The check corrected the pause section: the 24 quiet hours count time spent paused, so after a long pause the migration can stop soon after it resumes rather than running another 24 hours. Robin's review then corrected the identifier rule: the eight-digit suffix is also added when a tracing project already uses one of the agent's future environment names, such as `checkout-staging` for `checkout`, which matches the v0.18.0 migrator. - **Experiments in the migration:** the Helm comment for `config.agentMigration.enabled` says experiments migrate to agents. They are only classified as experiments, so the page is correct. langchain-ai/helm#1357 fixes the comment. - **Left alone on purpose:** the default **Application** and **Environment** resource-tag keys, which LangSmith still creates for every workspace. - **Out of scope, handled by the next PRs:** project-tied settings (usage limits, thread idle time, project resource tags, per-attachment spend limits in `evaluator-spend`), the remaining `LANGSMITH_PROJECT` mentions, and the remaining "tracing project" prose. - The full per-page change list and review history are on DOC-1797. ## Related - Refs [DOC-1797](https://linear.app/langchain/issue/DOC-1797/scrub-legacy-concepts-from-docs-after-self-hosted-update). - Follows #7522, which removed the sidebar icons from the same group. ## Checklist - [x] I read the [contributing guidelines](https://docs.langchain.com/oss/contributing/documentation), including the [language policy](https://docs.langchain.com/oss/python/contributing/overview#language-policy) - [x] I checked the rendered pages locally with `docs dev` or on the preview deployment - [x] I ran every changed code example, or none changed - [x] `make lint_prose` passes on the changed files - [x] `make broken-links-with-anchors` passes, or no links, pages, or navigation changed - [x] I updated `src/docs.json` navigation and redirects, or no page was added, moved, or removed Prepared with Claude Code.
# Conflicts: # src/langsmith/administration-overview.mdx # src/langsmith/audit-logs.mdx # src/langsmith/engine-repositories.mdx # src/langsmith/engine-self-hosted-github.mdx # src/langsmith/engine-setup.mdx # src/langsmith/engine.mdx # src/langsmith/evaluator-spend.mdx # src/langsmith/online-evaluations-multi-turn.mdx # src/langsmith/self-host-agent-migration.mdx # src/langsmith/usage-and-billing.mdx
Florence Morris (fjmorris)
marked this pull request as ready for review
November 22, 2026 00:28
Florence Morris (fjmorris)
requested a review
from Sean Angiolillo (seanangio)
as a code owner
November 22, 2026 00:28
Contributor
|
Mintlify preview branch generated: Site preview: https://langchain-5e9cc07a-preview-fjmorris-doc-1797-2-99d9b71b.mintlify.site Important Preview links may take a few minutes to start working while the deployment finishes. Changed documentation pages (preview deep links):
Only the top 5 changed markdown files by diff size are listed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Several LangSmith settings belong to an agent environment, but the docs still describe them as tracing project settings. Some steps also no longer match the UI. This PR rewords those settings in agent and environment terms and updates the steps.
Second of four stacked PRs. PR 1 (#7523) merged on November 21, so this PR now targets
main. Self-hosted v0.18.0 was released on November 20, 2026. The four stacked PRs for DOC-1797 merge in order: PR 1 (#7523) and PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on November 24 or 25.Changes
online-evaluations-multi-turn,rules,tuned-evaluators): one setting per agent environment, shown read-only in the evaluator and rule forms and changed from the Edit tracing settings icon. The Runs/Threads choice is under Trigger On (also inonline-evaluations-llm-as-judge).online-evaluations-multi-turnconfiguration: the short UI steps stay a numbered list ending with Create. The idle time, model, prompt, and feedback settings are now subsections, linked from step 6, instead of dense bold-titled steps (per review). The model and prompt subsections are tightened to active, present-tense wording, andtuned-evaluatorslinks straight to the idle-time subsection (per Robin's review).administration-overview: Agent Environments scope row. Removed the resource tags vs. commit tags note, which recommendedEnvironmenttags.usage-and-billing: the evaluator opt-out depends on the environment's default retention, the usage limit scopes list an Agent environment scope, and the section is now "Set environment and user usage limits". The usage limit dialog still says Project, so the page keeps that label, explains it, and says the Project list shows each environment by itstracer_session_name.evaluator-spend: "agent environment or dataset" throughout, with the override heading and anchor renamed.view-traces,manage-datasets-in-application,annotation-queues-sdk,online-evaluations-composite: environment wording, renamed dataset headings with inbound anchors updated, and thetracing-projectslinks fixed. The anchor itself stays on the "Compatibility with older APIs" accordion.issues:create(Workspace Admin or Editor), configuring it takesissues:update, and the environment spend limit takes an Organization Admin.Review notes
engine-setup,engine,engine-fix-validation,engine-notifications,engine-red-teaming,engine-repositories, andengine-self-hosted-github. A Cloud-only feature flag can replace this page with a goals board that has no switcher. It is always off on self-hosted, so the step holds for every self-hosted reader. Cloud rollout of that flag is not confirmed.mainwas merged into this branch. Ten files conflicted because this branch still held PR 1's original commits. Each was resolved by keepingmain's version and reapplying only this PR's changes, so the diff againstmainis now this PR's 22 files and nothing from PR 1 is applied twice. Earlier, docs(langsmith): clarify trace retention periods by hosting option #7723 onmainmoved the retention and usage limit content fromadministration-overviewtousage-and-billing, so this PR's retention and usage limit edits live there.engine-red-teaming"enabled for your project" (unclear what access is granted for), backfill for the Traces trigger, and the remaining "tracing project" prose, which the next PRs cover.Related
Checklist
docs devor on the preview deploymentmake lint_prosepasses on the changed filesmake broken-links-with-anchorspasses, or no links, pages, or navigation changedsrc/docs.jsonnavigation and redirects, or no page was added, moved, or removedPrepared with Claude Code.