Repository navigation
docs(langsmith): replace remaining tracing project wording with agent environments - #7784
Florence Morris (fjmorris) wants to merge 3 commits into
Conversation
… environments Reword "tracing project" across Engine, evaluator, filter, export, cost, retention, Terraform, and control plane pages. Describe session_id and session_name as the agent environment's tracer_session_id and tracer_session_name, rename the Engine setup and evaluator attachment headings with their inbound links, and fix the evaluator alignment labels for agent environments. Refs DOC-1797
| <Step title="Create the destination agent environments"> | ||
| The tool does not create agents or environments. Create the [agent environments](/langsmith/agent-environments) you need in the destination workspace before you migrate automation rules or charts, or those mappings have no target. | ||
| </Step> |
There was a problem hiding this comment.
🟡 New agent environments will not match the tool's name-based project mapping
Confirmed problem (Major). If a reader follows this step, the rule and chart mappings this step exists to fix still have no target.
The migration tool maps source projects to destination projects only by exact session name. See _build_project_mapping in langsmith_migrator/core/migrators/chart.py and build_project_mapping in rules.py in langchain-ai/langsmith-data-migration-tool. Both list /sessions on each side and match on name. The old step worked because the reader created destination projects with the same names as the source projects.
A new agent environment's backing session gets a tracer_session_name that LangSmith assigns. create-an-agent.mdx#use-an-environment-project says to "use the returned value rather than deriving it from the agent ID or display name". That name will not match the source project name, so the exact-name match fails.
There is a second problem. "The tool does not create agents or environments" sits next to line 117, which says the issues command "creates any tracing project that has Engine data". rules.py build_project_mapping(create_missing=...) can also create missing sessions. Readers will not know whether to pre-create anything.
Either tell readers to supply an explicit project mapping (source project ID to the environment's tracer_session_id), using the tool's project-mapping option that the README covers, or keep the original instruction to create same-named projects. Confirm with the migration tool owners which one applies on v0.18.
(Refers to lines 52-54)
Your feedback helps Robin learn. React with 👍 or 👎 to tell us if this review comment was useful.
…is/DOC-1797-4 # Conflicts: # src/langsmith/usage-and-billing.mdx
…ments on PR 4 - cost-tracking, data-export, filter-traces, insights, manage-evaluators-sdk: stats, exports, filters, Insights, and evaluator attachment describe agent environments. ID placeholders point to the resolved session_id, and the Insights CLI example uses the tracer_session_name placeholder. - Engine pages (notifications, red teaming, self-hosted), byoc-migration, self-host-slack, self-host-ttl: per-environment wording. - manage-with-terraform: example description strings say agent environments. Refs DOC-1797
|
Mintlify preview branch generated: Site preview: https://langchain-5e9cc07a-preview-fjmorris-doc-1797-4-a23f53fa.mintlify.site Important Preview links may take a few minutes to start working while the deployment finishes. Changed documentation pages (preview deep links):
Only the top 5 changed markdown files by diff size are listed. |
## Why Self-hosted LangSmith v0.18 moves every workspace to agents and environments, so the docs no longer need to describe two kinds of workspace. Today, many pages split their content between "agent-based" and "project-based" workspaces, offer a tracing project as an alternative to an agent environment, or carry a "Cloud only, not self-hosted" callout. After v0.18, all of these are wrong for every reader. Self-hosted v0.18.0 was released on November 20, 2026. This is the first of four stacked PRs for DOC-1797, which merge in order: this PR and PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on November 24 or 25. ## Changes - **Upgrade page:** rewrote the workspace migration page as "What changes in LangSmith v0.18", for self-hosted administrators: before upgrading, how the migration runs and how to pause it, checking whether a workspace has migrated, how tracing projects map to agents, and what changes in each workspace. Renamed it to `self-host-agent-migration` (with a redirect) and moved it to Self-hosted > Manage an instance, after "Upgrade an installation". - **Navigation:** renamed the "Agent-based workspaces" group to "Agent Registry", with the `agent-registry` page labeled "Overview". - **Workspace-type split removed:** deleted the `project-based-note`, `agent-based-note`, and `agent-availability-note` snippets. Across about 65 tracing, evaluator, and setup pages, removed "agent-based workspace" qualifiers, project-based alternatives, and `LANGSMITH_PROJECT` fallbacks, and replaced "tracing project, or agent environment" wording with agent environments. No "agent-based workspace" or "project-based" wording is left, apart from the old migration page redirect. - **UI labels checked in the product:** the evaluator tab is **Online Evaluators** on an agent environment, attachments are **Environments & Datasets**, the Engine limit is **Environment limit**, the dashboard chart source is **+ Select agent or dataset**, and the **Evaluators** page spend widgets match an agent workspace. Steps to an environment now consistently say "select your agent, go to **Tracing**, and select the environment". - **Errors already on `main`:** the create-an-agent snippet (renamed `create-agent-for-tracing`) named an **Observability** option that does not exist and now gives **Connect** > **Connect an existing agent**. The alerts steps now give the real path to **Alerts** and say **+ Alert**. The dashboards page named the chart button and tabs incorrectly, and now places custom dashboards under **Workspace** > **Dashboards**. - **Settings now per environment:** retention (`billing`, `usage-and-billing`, `data-purging-compliance`, `evaluators`), resource tags (the **Agent** tag key and the **Agent Environments** picker label), and the tracing plugins, which always send to the agent's **Production** environment. - **Reference links:** `audit-logs` relabels the operations row as "Agent environments", adds `create_tracer_session`, and lists v0.18 for `create_agent_environment` and `mint_agent_access_token`. Its `operations` link and the `abac` API link now point to the docs API reference instead of the Swagger UI. - **Smaller fixes:** `set-up-hierarchy` and `workload-isolation` anchors renamed to `#set-up-agents` and `#isolate-by-agent`, the 403 troubleshooting paragraph removed from `log-traces-to-agent`, `webhooks` describes `session_id` as the environment's `tracer_session_id`, `attach-user-feedback` states the v0.18 and SDK minimums, the `observability-quickstart` intro no longer points to compatibility instructions that the old snippet held, and `AGENTS.md` updates the navigation map. ## Review notes - **Upgrade page:** the migration behavior was checked against the final self-hosted v0.18.0 release (Helm chart `langsmith-0.18.0`, app version `0.18.2026112100`): the roughly 10-minute start delay that restarts do not reset, each workspace switching once its migration is verified, the stop after 24 quiet hours and a clean full check, `authType: none` installs not being migrated, and pausing with `config.agentMigration.enabled`, which defaults to `true` and is unchanged from the release candidates. The check corrected the pause section: the 24 quiet hours count time spent paused, so after a long pause the migration can stop soon after it resumes rather than running another 24 hours. Robin's review then corrected the identifier rule: the eight-digit suffix is also added when a tracing project already uses one of the agent's future environment names, such as `checkout-staging` for `checkout`, which matches the v0.18.0 migrator. - **Experiments in the migration:** the Helm comment for `config.agentMigration.enabled` says experiments migrate to agents. They are only classified as experiments, so the page is correct. langchain-ai/helm#1357 fixes the comment. - **Left alone on purpose:** the default **Application** and **Environment** resource-tag keys, which LangSmith still creates for every workspace. - **Out of scope, handled by the next PRs:** project-tied settings (usage limits, thread idle time, project resource tags, per-attachment spend limits in `evaluator-spend`), the remaining `LANGSMITH_PROJECT` mentions, and the remaining "tracing project" prose. - The full per-page change list and review history are on DOC-1797. ## Related - Refs [DOC-1797](https://linear.app/langchain/issue/DOC-1797/scrub-legacy-concepts-from-docs-after-self-hosted-update). - Follows #7522, which removed the sidebar icons from the same group. ## Checklist - [x] I read the [contributing guidelines](https://docs.langchain.com/oss/contributing/documentation), including the [language policy](https://docs.langchain.com/oss/python/contributing/overview#language-policy) - [x] I checked the rendered pages locally with `docs dev` or on the preview deployment - [x] I ran every changed code example, or none changed - [x] `make lint_prose` passes on the changed files - [x] `make broken-links-with-anchors` passes, or no links, pages, or navigation changed - [x] I updated `src/docs.json` navigation and redirects, or no page was added, moved, or removed Prepared with Claude Code.
#7745) ## Why Several LangSmith settings belong to an agent environment, but the docs still describe them as tracing project settings. Some steps also no longer match the UI. This PR rewords those settings in agent and environment terms and updates the steps. Second of four stacked PRs. PR 1 (#7523) merged on November 21, so this PR now targets `main`. Self-hosted v0.18.0 was released on November 20, 2026. The four stacked PRs for DOC-1797 merge in order: PR 1 (#7523) and PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on November 24 or 25. ## Changes - **Thread idle time** (`online-evaluations-multi-turn`, `rules`, `tuned-evaluators`): one setting per agent environment, shown read-only in the evaluator and rule forms and changed from the **Edit tracing settings** <Icon icon="settings"/> icon. The Runs/Threads choice is under **Trigger On** (also in `online-evaluations-llm-as-judge`). - **`online-evaluations-multi-turn` configuration:** the short UI steps stay a numbered list ending with **Create**. The idle time, model, prompt, and feedback settings are now subsections, linked from step 6, instead of dense bold-titled steps (per review). The model and prompt subsections are tightened to active, present-tense wording, and `tuned-evaluators` links straight to the idle-time subsection (per Robin's review). - **`administration-overview`:** **Agent Environments** scope row. Removed the resource tags vs. commit tags note, which recommended `Environment` tags. - **`usage-and-billing`:** the evaluator opt-out depends on the environment's default retention, the usage limit scopes list an **Agent environment** scope, and the section is now "Set environment and user usage limits". The usage limit dialog still says **Project**, so the page keeps that label, explains it, and says the **Project** list shows each environment by its `tracer_session_name`. - **`evaluator-spend`:** "agent environment or dataset" throughout, with the override heading and anchor renamed. - **`view-traces`, `manage-datasets-in-application`, `annotation-queues-sdk`, `online-evaluations-composite`:** environment wording, renamed dataset headings with inbound anchors updated, and the `tracing-projects` links fixed. The anchor itself stays on the "Compatibility with older APIs" accordion. - **Engine pages:** one shared first step for **Configure Engine**. Turning on Engine takes `issues:create` (Workspace Admin or Editor), configuring it takes `issues:update`, and the environment spend limit takes an Organization Admin. ## Review notes - UI labels were checked in the product. Where each setting is stored and the Engine permissions were checked in the product source. - **Engine environment selection:** an earlier note here said the Engine page had no environment selector. Checked against the v0.18.0 release: the agent's **Engine** page shows an environment switcher at the top, so every step that opens **Configure Engine** now says to select the environment there first: `engine-setup`, `engine`, `engine-fix-validation`, `engine-notifications`, `engine-red-teaming`, `engine-repositories`, and `engine-self-hosted-github`. A Cloud-only feature flag can replace this page with a goals board that has no switcher. It is always off on self-hosted, so the step holds for every self-hosted reader. Cloud rollout of that flag is not confirmed. - **Brought up to date:** after #7523 was squash-merged, `main` was merged into this branch. Ten files conflicted because this branch still held PR 1's original commits. Each was resolved by keeping `main`'s version and reapplying only this PR's changes, so the diff against `main` is now this PR's 22 files and nothing from PR 1 is applied twice. Earlier, #7723 on `main` moved the retention and usage limit content from `administration-overview` to `usage-and-billing`, so this PR's retention and usage limit edits live there. - **Not changed here:** `engine-red-teaming` "enabled for your project" (unclear what access is granted for), backfill for the **Traces** trigger, and the remaining "tracing project" prose, which the next PRs cover. ## Related - Refs [DOC-1797](https://linear.app/langchain/issue/DOC-1797/scrub-legacy-concepts-from-docs-after-self-hosted-update). - Follows #7523, merged November 21. ## Checklist - [x] I read the [contributing guidelines](https://docs.langchain.com/oss/contributing/documentation), including the [language policy](https://docs.langchain.com/oss/python/contributing/overview#language-policy) - [x] I checked the rendered pages locally with `docs dev` or on the preview deployment - [x] I ran every changed code example, or none changed - [x] `make lint_prose` passes on the changed files - [x] `make broken-links-with-anchors` passes, or no links, pages, or navigation changed - [x] I updated `src/docs.json` navigation and redirects, or no page was added, moved, or removed Prepared with Claude Code.
Why
Many pages still describe traces, Engine, evaluators, exports, and settings as belonging to a "tracing project". After self-hosted v0.18, every workspace uses agents and environments, so this wording points readers at a container they no longer see. This PR rewords the remaining "tracing project" prose to agent environments, and describes API fields that still take a project ID in terms of the environment.
Do not merge before November 24. Fourth of four stacked PRs, based on #7757, brought up to date with
mainbefore merging. Self-hosted v0.18.0 was released on November 20, 2026. The four stacked PRs for DOC-1797 merge in order: PR 1 (#7523) and PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on November 24 or 25.Changes
engine-setup: the heading is now "Turn on Engine for an agent environment" (the 4 inbound links are updated). The first step says to select your agent, go to Engine, and pick the environment under Choose an environment to analyze. Production is preselected when Engine is not set up for it yet. The rest of the page's "project" wording now refers to the environment or the agent.engine,engine-issues,engine-red-teaming,engine-fix-validation,engine-repositories,engine-self-hosted,engine-self-hosted-github, andself-host-slack: Engine runs on, scans, and is paused per agent environment. The evidence link label is View all in tracing.engine-self-hosted: turning on Engine takes a Workspace Admin, a Workspace Editor, or a custom role withissues:create, matchingengine-setup. Previously the page said a role that "can update tracing projects".engine-notifications:session_idandsession_nameare the agent environment'stracer_session_idandtracer_session_name.engine-issues: the CLI example uses the<environment-tracer-session-name>placeholder from docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757.evaluation-concepts,evaluators,manage-evaluators-sdk,decision-model-evaluator,online-evaluations-decision-models,online-evaluations-composite, andusage-and-billing: evaluators attach to agent environments and datasets.evaluation-conceptsheading is renamed, and its link fromevaluator-spendis updated.improve-judge-evaluator-feedback: the environment column uses the agent navigation. To align an existing evaluator, open it from the Online Evaluators tab. The box for environment evaluators is Align Evaluator with run data, and its button is Select Runs. Before this PR the page said "experiment data" and Select Experiments, which are the dataset labels.session_id:big-query-bulk-export,data-export, anddata-purging-compliancesay to copy the environment's Tracing Project ID from the ID badge, as docs(langsmith): reword project-scoped settings for agent environments #7745 does onannotation-queues-sdk. Thesession_iddescriptions are updated the same way on these pages:run-data-formatfeedback-data-fieldsrun-evals-api-onlymanage-with-terraformcontrol-planeanddeploy-with-control-planesay each deployment sends traces to an agent environment. Without--agent-id, LangSmith creates an agent whose ID is derived from the deployment name. Deleting a deployment keeps its environment and traces.custom-output-rendering: settings are opened from Edit tracing settings on the environment's Tracing page.filtering-experience-note: the noun is now agent environment.cost-trackinglog-llm-traceinsightsmultiple-messagesdataset-transformationssmithtuneredact-secretscoding-agent-secret-redactionself-host-ttl, retention tiers are set per environment.self-host-scaleuses the new noun. Inbyoc-migration, the step to create destination agent environments replaces the step to create projects.cost-tracking: agent environment stats, including the heading.data-export: exports come from an agent environment, andsession_idis described as the environment'stracer_session_id.filter-traces: the common values come from the agent environment. The AI search SDK samples use a<session-id>placeholder, with a sentence pointing to Resolve an agent environment.insights: the CLI example uses--project <environment-tracer-session-name>, and the thread variables apply to agent environments.manage-evaluators-sdk: evaluators attach to an agent environment or dataset.engine-notifications,engine-red-teaming,engine-self-hosted,byoc-migration,self-host-slack, andself-host-ttl: these say agent environment.manage-with-terraform: the example description strings say agent environments.Review notes
issues:createroles, the Engine webhooksession_name, the Edit tracing settings and Custom Output Rendering labels, Create from labeled data, the alignment box labels, the agent created for a deployment without one, and that deleting a deployment keeps its environment by default.gatewayis an internal project, not an agent environment.byoc-migration'sissuessentence: the migration tool creates tracing projects, not agents.abac,set-up-resource-tags, and the SmithDB migration snippets.project_namecompatibility parameter intrace-with-google-adk.observability-concepts,granular-usage,self-host-upgrades, andself-host-agent-migration.langsmith project listcomment inlangsmith-cli.annotation-queues-sdk,composite-evaluators-sdk,export-traces, andoss/deepagents/subagentswere scrubbed in docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757 instead of here, so this PR does not change them. The only remaining match on those pages is the UI's Tracing Project ID badge label.usage-and-billing: docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757 (through docs(langsmith): remove the self-hosted split from agent docs #7523's review) had folded the evaluator opt-out Note into the paragraph above it, which already says environment, so the merge keeps docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757's version.make lint_prose,make check-mdx, andmake broken-links-with-anchorspass after the merge.engine-self-hosted.engine-red-teamingdoes not say that external red-team targets are unavailable on self-hosted. That gap predates this PR.Related
Checklist
docs devor on the preview deploymentmake lint_prosepasses on the changed filesmake broken-links-with-anchorspasses, or no links, pages, or navigation changedsrc/docs.jsonnavigation and redirects, or no page was added, moved, or removedPrepared with Claude Code.