Skip to content

docs(langsmith): replace remaining tracing project wording with agent environments - #7784

Draft
Florence Morris (fjmorris) wants to merge 3 commits into
fjmorris/DOC-1797-3from
fjmorris/DOC-1797-4
Draft

Florence Morris (fjmorris) wants to merge 3 commits into
fjmorris/DOC-1797-3from
fjmorris/DOC-1797-4

Conversation

@fjmorris

@fjmorris Florence Morris (fjmorris) commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Why

Many pages still describe traces, Engine, evaluators, exports, and settings as belonging to a "tracing project". After self-hosted v0.18, every workspace uses agents and environments, so this wording points readers at a container they no longer see. This PR rewords the remaining "tracing project" prose to agent environments, and describes API fields that still take a project ID in terms of the environment.

Do not merge before November 24. Fourth of four stacked PRs, based on #7757, brought up to date with main before merging. Self-hosted v0.18.0 was released on November 20, 2026. The four stacked PRs for DOC-1797 merge in order: PR 1 (#7523) and PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on November 24 or 25.

Changes

  • Engine pages:
    • engine-setup: the heading is now "Turn on Engine for an agent environment" (the 4 inbound links are updated). The first step says to select your agent, go to Engine, and pick the environment under Choose an environment to analyze. Production is preselected when Engine is not set up for it yet. The rest of the page's "project" wording now refers to the environment or the agent.
    • engine, engine-issues, engine-red-teaming, engine-fix-validation, engine-repositories, engine-self-hosted, engine-self-hosted-github, and self-host-slack: Engine runs on, scans, and is paused per agent environment. The evidence link label is View all in tracing.
    • engine-self-hosted: turning on Engine takes a Workspace Admin, a Workspace Editor, or a custom role with issues:create, matching engine-setup. Previously the page said a role that "can update tracing projects".
    • engine-notifications: session_id and session_name are the agent environment's tracer_session_id and tracer_session_name.
    • engine-issues: the CLI example uses the <environment-tracer-session-name> placeholder from docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757.
  • Evaluators:
    • evaluation-concepts, evaluators, manage-evaluators-sdk, decision-model-evaluator, online-evaluations-decision-models, online-evaluations-composite, and usage-and-billing: evaluators attach to agent environments and datasets.
    • The evaluation-concepts heading is renamed, and its link from evaluator-spend is updated.
    • improve-judge-evaluator-feedback: the environment column uses the agent navigation. To align an existing evaluator, open it from the Online Evaluators tab. The box for environment evaluators is Align Evaluator with run data, and its button is Select Runs. Before this PR the page said "experiment data" and Select Experiments, which are the dataset labels.
  • IDs passed as session_id: big-query-bulk-export, data-export, and data-purging-compliance say to copy the environment's Tracing Project ID from the ID badge, as docs(langsmith): reword project-scoped settings for agent environments #7745 does on annotation-queues-sdk. The session_id descriptions are updated the same way on these pages:
    • run-data-format
    • feedback-data-fields
    • run-evals-api-only
    • manage-with-terraform
  • Deployments: control-plane and deploy-with-control-plane say each deployment sends traces to an agent environment. Without --agent-id, LangSmith creates an agent whose ID is derived from the deployment name. Deleting a deployment keeps its environment and traces.
  • UI steps and labels:
    • custom-output-rendering: settings are opened from Edit tracing settings on the environment's Tracing page.
    • Filter pages and filtering-experience-note: the noun is now agent environment.
    • The following pages get the same noun change:
      • cost-tracking
      • log-llm-trace
      • insights
      • multiple-messages
      • dataset-transformations
      • smithtune
      • redact-secrets
      • coding-agent-secret-redaction
  • Self-hosted and BYOC: in self-host-ttl, retention tiers are set per environment. self-host-scale uses the new noun. In byoc-migration, the step to create destination agent environments replaces the step to create projects.
  • Final "project" pass: a search of this PR's pages for "project", "legacy", and "application" found more legacy wording, now changed:
    • cost-tracking: agent environment stats, including the heading.
    • data-export: exports come from an agent environment, and session_id is described as the environment's tracer_session_id.
    • filter-traces: the common values come from the agent environment. The AI search SDK samples use a <session-id> placeholder, with a sentence pointing to Resolve an agent environment.
    • insights: the CLI example uses --project <environment-tracer-session-name>, and the thread variables apply to agent environments.
    • manage-evaluators-sdk: evaluators attach to an agent environment or dataset.
    • engine-notifications, engine-red-teaming, engine-self-hosted, byoc-migration, self-host-slack, and self-host-ttl: these say agent environment.
    • manage-with-terraform: the example description strings say agent environments.
    • Left as they are: GitLab, BigQuery, Jira, and Linear projects, code-project wording, CLI commands, URLs, and API literals.

Review notes

  • Checked in the product source: the Engine environment picker and its Production default, View all in tracing, the issues:create roles, the Engine webhook session_name, the Edit tracing settings and Custom Output Rendering labels, Create from labeled data, the alignment box labels, the agent created for a deployment without one, and that deleting a deployment keeps its environment by default.
  • Needs a UI check: that clicking an evaluator on the Online Evaluators tab opens the page with the alignment box.
  • Left unchanged on purpose:
    • LLM Gateway pages: gateway is an internal project, not an agent environment.
    • byoc-migration's issues sentence: the migration tool creates tracing projects, not agents.
    • API-level descriptions in abac, set-up-resource-tags, and the SmithDB migration snippets.
    • The project_name compatibility parameter in trace-with-google-adk.
    • Deliberate explanations of the backing project in observability-concepts, granular-usage, self-host-upgrades, and self-host-agent-migration.
    • The langsmith project list comment in langsmith-cli.
    • The changelogs.
  • Shared pages handled in docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757: annotation-queues-sdk, composite-evaluators-sdk, export-traces, and oss/deepagents/subagents were scrubbed in docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757 instead of here, so this PR does not change them. The only remaining match on those pages is the UI's Tracing Project ID badge label.
  • Brought up to date: merged docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757 into this branch, including its review fixes. One conflict in usage-and-billing: docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757 (through docs(langsmith): remove the self-hosted split from agent docs #7523's review) had folded the evaluator opt-out Note into the paragraph above it, which already says environment, so the merge keeps docs(langsmith): replace or scope LANGSMITH_PROJECT for agent addressing #7757's version. make lint_prose, make check-mdx, and make broken-links-with-anchors pass after the merge.
  • Reviews run: a style review and a source verification ran before commit, and their fixes are included. A second source verification covered the final pass and found no contradictions. Not verified: the usage-attribution identifiers listed on engine-self-hosted.
  • Not changed: engine-red-teaming does not say that external red-team targets are unavailable on self-hosted. That gap predates this PR.

Related

Checklist

  • I read the contributing guidelines, including the language policy
  • I checked the rendered pages locally with docs dev or on the preview deployment
  • I ran every changed code example, or none changed
  • make lint_prose passes on the changed files
  • make broken-links-with-anchors passes, or no links, pages, or navigation changed
  • I updated src/docs.json navigation and redirects, or no page was added, moved, or removed

Prepared with Claude Code.

… environments

Reword "tracing project" across Engine, evaluator, filter, export, cost,
retention, Terraform, and control plane pages. Describe session_id and
session_name as the agent environment's tracer_session_id and
tracer_session_name, rename the Engine setup and evaluator attachment
headings with their inbound links, and fix the evaluator alignment labels
for agent environments.

Refs DOC-1797
@github-actions github-actions Bot added langsmith For docs changes to LangSmith internal labels Nov 16, 2026

@robin-agent robin-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Robin Review found 1 potential issue.

2 additional findings can be viewed in the web app.

Open in Robin • View Robin trace

Comment on lines +52 to 54
<Step title="Create the destination agent environments">
The tool does not create agents or environments. Create the [agent environments](/langsmith/agent-environments) you need in the destination workspace before you migrate automation rules or charts, or those mappings have no target.
</Step>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 New agent environments will not match the tool's name-based project mapping

Confirmed problem (Major). If a reader follows this step, the rule and chart mappings this step exists to fix still have no target.

The migration tool maps source projects to destination projects only by exact session name. See _build_project_mapping in langsmith_migrator/core/migrators/chart.py and build_project_mapping in rules.py in langchain-ai/langsmith-data-migration-tool. Both list /sessions on each side and match on name. The old step worked because the reader created destination projects with the same names as the source projects.

A new agent environment's backing session gets a tracer_session_name that LangSmith assigns. create-an-agent.mdx#use-an-environment-project says to "use the returned value rather than deriving it from the agent ID or display name". That name will not match the source project name, so the exact-name match fails.

There is a second problem. "The tool does not create agents or environments" sits next to line 117, which says the issues command "creates any tracing project that has Engine data". rules.py build_project_mapping(create_missing=...) can also create missing sessions. Readers will not know whether to pre-create anything.

Either tell readers to supply an explicit project mapping (source project ID to the environment's tracer_session_id), using the tool's project-mapping option that the README covers, or keep the original instruction to create same-named projects. Confirm with the migration tool owners which one applies on v0.18.

(Refers to lines 52-54)


Your feedback helps Robin learn. React with 👍 or 👎 to tell us if this review comment was useful.

…is/DOC-1797-4

# Conflicts:
#	src/langsmith/usage-and-billing.mdx
…ments on PR 4

- cost-tracking, data-export, filter-traces, insights, manage-evaluators-sdk:
  stats, exports, filters, Insights, and evaluator attachment describe agent
  environments. ID placeholders point to the resolved session_id, and the
  Insights CLI example uses the tracer_session_name placeholder.
- Engine pages (notifications, red teaming, self-hosted), byoc-migration,
  self-host-slack, self-host-ttl: per-environment wording.
- manage-with-terraform: example description strings say agent environments.

Refs DOC-1797
@github-actions

Copy link
Copy Markdown
Contributor

Mintlify preview branch generated: preview-fjmorris-doc-1797-4-a23f53fa

Site preview: https://langchain-5e9cc07a-preview-fjmorris-doc-1797-4-a23f53fa.mintlify.site

Important

Preview links may take a few minutes to start working while the deployment finishes.

Changed documentation pages (preview deep links):

Only the top 5 changed markdown files by diff size are listed.

Florence Morris (fjmorris) added a commit that referenced this pull request Nov 22, 2026
## Why

Self-hosted LangSmith v0.18 moves every workspace to agents and
environments, so the docs no longer need to describe two kinds of
workspace. Today, many pages split their content between "agent-based"
and "project-based" workspaces, offer a tracing project as an
alternative to an agent environment, or carry a "Cloud only, not
self-hosted" callout. After v0.18, all of these are wrong for every
reader.

Self-hosted v0.18.0 was released on November 20, 2026. This is the first
of four stacked PRs for DOC-1797, which merge in order: this PR and PR 2
(#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on
November 24 or 25.

## Changes

- **Upgrade page:** rewrote the workspace migration page as "What
changes in LangSmith v0.18", for self-hosted administrators: before
upgrading, how the migration runs and how to pause it, checking whether
a workspace has migrated, how tracing projects map to agents, and what
changes in each workspace. Renamed it to `self-host-agent-migration`
(with a redirect) and moved it to Self-hosted > Manage an instance,
after "Upgrade an installation".
- **Navigation:** renamed the "Agent-based workspaces" group to "Agent
Registry", with the `agent-registry` page labeled "Overview".
- **Workspace-type split removed:** deleted the `project-based-note`,
`agent-based-note`, and `agent-availability-note` snippets. Across about
65 tracing, evaluator, and setup pages, removed "agent-based workspace"
qualifiers, project-based alternatives, and `LANGSMITH_PROJECT`
fallbacks, and replaced "tracing project, or agent environment" wording
with agent environments. No "agent-based workspace" or "project-based"
wording is left, apart from the old migration page redirect.
- **UI labels checked in the product:** the evaluator tab is **Online
Evaluators** on an agent environment, attachments are **Environments &
Datasets**, the Engine limit is **Environment limit**, the dashboard
chart source is **+ Select agent or dataset**, and the **Evaluators**
page spend widgets match an agent workspace. Steps to an environment now
consistently say "select your agent, go to **Tracing**, and select the
environment".
- **Errors already on `main`:** the create-an-agent snippet (renamed
`create-agent-for-tracing`) named an **Observability** option that does
not exist and now gives **Connect** > **Connect an existing agent**. The
alerts steps now give the real path to **Alerts** and say **+ Alert**.
The dashboards page named the chart button and tabs incorrectly, and now
places custom dashboards under **Workspace** > **Dashboards**.
- **Settings now per environment:** retention (`billing`,
`usage-and-billing`, `data-purging-compliance`, `evaluators`), resource
tags (the **Agent** tag key and the **Agent Environments** picker
label), and the tracing plugins, which always send to the agent's
**Production** environment.
- **Reference links:** `audit-logs` relabels the operations row as
"Agent environments", adds `create_tracer_session`, and lists v0.18 for
`create_agent_environment` and `mint_agent_access_token`. Its
`operations` link and the `abac` API link now point to the docs API
reference instead of the Swagger UI.
- **Smaller fixes:** `set-up-hierarchy` and `workload-isolation` anchors
renamed to `#set-up-agents` and `#isolate-by-agent`, the 403
troubleshooting paragraph removed from `log-traces-to-agent`, `webhooks`
describes `session_id` as the environment's `tracer_session_id`,
`attach-user-feedback` states the v0.18 and SDK minimums, the
`observability-quickstart` intro no longer points to compatibility
instructions that the old snippet held, and `AGENTS.md` updates the
navigation map.

## Review notes

- **Upgrade page:** the migration behavior was checked against the final
self-hosted v0.18.0 release (Helm chart `langsmith-0.18.0`, app version
`0.18.2026112100`): the roughly 10-minute start delay that restarts do
not reset, each workspace switching once its migration is verified, the
stop after 24 quiet hours and a clean full check, `authType: none`
installs not being migrated, and pausing with
`config.agentMigration.enabled`, which defaults to `true` and is
unchanged from the release candidates. The check corrected the pause
section: the 24 quiet hours count time spent paused, so after a long
pause the migration can stop soon after it resumes rather than running
another 24 hours. Robin's review then corrected the identifier rule: the
eight-digit suffix is also added when a tracing project already uses one
of the agent's future environment names, such as `checkout-staging` for
`checkout`, which matches the v0.18.0 migrator.
- **Experiments in the migration:** the Helm comment for
`config.agentMigration.enabled` says experiments migrate to agents. They
are only classified as experiments, so the page is correct.
langchain-ai/helm#1357 fixes the comment.
- **Left alone on purpose:** the default **Application** and
**Environment** resource-tag keys, which LangSmith still creates for
every workspace.
- **Out of scope, handled by the next PRs:** project-tied settings
(usage limits, thread idle time, project resource tags, per-attachment
spend limits in `evaluator-spend`), the remaining `LANGSMITH_PROJECT`
mentions, and the remaining "tracing project" prose.
- The full per-page change list and review history are on DOC-1797.

## Related

- Refs
[DOC-1797](https://linear.app/langchain/issue/DOC-1797/scrub-legacy-concepts-from-docs-after-self-hosted-update).
- Follows #7522, which removed the sidebar icons from the same group.

## Checklist

- [x] I read the [contributing
guidelines](https://docs.langchain.com/oss/contributing/documentation),
including the [language
policy](https://docs.langchain.com/oss/python/contributing/overview#language-policy)
- [x] I checked the rendered pages locally with `docs dev` or on the
preview deployment
- [x] I ran every changed code example, or none changed
- [x] `make lint_prose` passes on the changed files
- [x] `make broken-links-with-anchors` passes, or no links, pages, or
navigation changed
- [x] I updated `src/docs.json` navigation and redirects, or no page was
added, moved, or removed

Prepared with Claude Code.
Florence Morris (fjmorris) added a commit that referenced this pull request Nov 22, 2026
#7745)

## Why

Several LangSmith settings belong to an agent environment, but the docs
still describe them as tracing project settings. Some steps also no
longer match the UI. This PR rewords those settings in agent and
environment terms and updates the steps.

Second of four stacked PRs. PR 1 (#7523) merged on November 21, so this
PR now targets `main`. Self-hosted v0.18.0 was released on November 20,
2026. The four stacked PRs for DOC-1797 merge in order: PR 1 (#7523) and
PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4
(#7784) on November 24 or 25.

## Changes

- **Thread idle time** (`online-evaluations-multi-turn`, `rules`,
`tuned-evaluators`): one setting per agent environment, shown read-only
in the evaluator and rule forms and changed from the **Edit tracing
settings** <Icon icon="settings"/> icon. The Runs/Threads choice is
under **Trigger On** (also in `online-evaluations-llm-as-judge`).
- **`online-evaluations-multi-turn` configuration:** the short UI steps
stay a numbered list ending with **Create**. The idle time, model,
prompt, and feedback settings are now subsections, linked from step 6,
instead of dense bold-titled steps (per review). The model and prompt
subsections are tightened to active, present-tense wording, and
`tuned-evaluators` links straight to the idle-time subsection (per
Robin's review).
- **`administration-overview`:** **Agent Environments** scope row.
Removed the resource tags vs. commit tags note, which recommended
`Environment` tags.
- **`usage-and-billing`:** the evaluator opt-out depends on the
environment's default retention, the usage limit scopes list an **Agent
environment** scope, and the section is now "Set environment and user
usage limits". The usage limit dialog still says **Project**, so the
page keeps that label, explains it, and says the **Project** list shows
each environment by its `tracer_session_name`.
- **`evaluator-spend`:** "agent environment or dataset" throughout, with
the override heading and anchor renamed.
- **`view-traces`, `manage-datasets-in-application`,
`annotation-queues-sdk`, `online-evaluations-composite`:** environment
wording, renamed dataset headings with inbound anchors updated, and the
`tracing-projects` links fixed. The anchor itself stays on the
"Compatibility with older APIs" accordion.
- **Engine pages:** one shared first step for **Configure Engine**.
Turning on Engine takes `issues:create` (Workspace Admin or Editor),
configuring it takes `issues:update`, and the environment spend limit
takes an Organization Admin.

## Review notes

- UI labels were checked in the product. Where each setting is stored
and the Engine permissions were checked in the product source.
- **Engine environment selection:** an earlier note here said the Engine
page had no environment selector. Checked against the v0.18.0 release:
the agent's **Engine** page shows an environment switcher at the top, so
every step that opens **Configure Engine** now says to select the
environment there first: `engine-setup`, `engine`,
`engine-fix-validation`, `engine-notifications`, `engine-red-teaming`,
`engine-repositories`, and `engine-self-hosted-github`. A Cloud-only
feature flag can replace this page with a goals board that has no
switcher. It is always off on self-hosted, so the step holds for every
self-hosted reader. Cloud rollout of that flag is not confirmed.
- **Brought up to date:** after #7523 was squash-merged, `main` was
merged into this branch. Ten files conflicted because this branch still
held PR 1's original commits. Each was resolved by keeping `main`'s
version and reapplying only this PR's changes, so the diff against
`main` is now this PR's 22 files and nothing from PR 1 is applied twice.
Earlier, #7723 on `main` moved the retention and usage limit content
from `administration-overview` to `usage-and-billing`, so this PR's
retention and usage limit edits live there.
- **Not changed here:** `engine-red-teaming` "enabled for your project"
(unclear what access is granted for), backfill for the **Traces**
trigger, and the remaining "tracing project" prose, which the next PRs
cover.

## Related

- Refs
[DOC-1797](https://linear.app/langchain/issue/DOC-1797/scrub-legacy-concepts-from-docs-after-self-hosted-update).
- Follows #7523, merged November 21.

## Checklist

- [x] I read the [contributing
guidelines](https://docs.langchain.com/oss/contributing/documentation),
including the [language
policy](https://docs.langchain.com/oss/python/contributing/overview#language-policy)
- [x] I checked the rendered pages locally with `docs dev` or on the
preview deployment
- [x] I ran every changed code example, or none changed
- [x] `make lint_prose` passes on the changed files
- [x] `make broken-links-with-anchors` passes, or no links, pages, or
navigation changed
- [x] I updated `src/docs.json` navigation and redirects, or no page was
added, moved, or removed

Prepared with Claude Code.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal langsmith For docs changes to LangSmith

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant