Skip to content

docs(langsmith): reword project-scoped settings for agent environments - #7745

Merged
Florence Morris (fjmorris) merged 32 commits into
mainfrom
fjmorris/DOC-1797-2
Nov 22, 2026
Merged

Florence Morris (fjmorris) merged 32 commits into
mainfrom
fjmorris/DOC-1797-2

Conversation

@fjmorris

@fjmorris Florence Morris (fjmorris) commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

Several LangSmith settings belong to an agent environment, but the docs still describe them as tracing project settings. Some steps also no longer match the UI. This PR rewords those settings in agent and environment terms and updates the steps.

Second of four stacked PRs. PR 1 (#7523) merged on November 21, so this PR now targets main. Self-hosted v0.18.0 was released on November 20, 2026. The four stacked PRs for DOC-1797 merge in order: PR 1 (#7523) and PR 2 (#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on November 24 or 25.

Changes

  • Thread idle time (online-evaluations-multi-turn, rules, tuned-evaluators): one setting per agent environment, shown read-only in the evaluator and rule forms and changed from the Edit tracing settings icon. The Runs/Threads choice is under Trigger On (also in online-evaluations-llm-as-judge).
  • online-evaluations-multi-turn configuration: the short UI steps stay a numbered list ending with Create. The idle time, model, prompt, and feedback settings are now subsections, linked from step 6, instead of dense bold-titled steps (per review). The model and prompt subsections are tightened to active, present-tense wording, and tuned-evaluators links straight to the idle-time subsection (per Robin's review).
  • administration-overview: Agent Environments scope row. Removed the resource tags vs. commit tags note, which recommended Environment tags.
  • usage-and-billing: the evaluator opt-out depends on the environment's default retention, the usage limit scopes list an Agent environment scope, and the section is now "Set environment and user usage limits". The usage limit dialog still says Project, so the page keeps that label, explains it, and says the Project list shows each environment by its tracer_session_name.
  • evaluator-spend: "agent environment or dataset" throughout, with the override heading and anchor renamed.
  • view-traces, manage-datasets-in-application, annotation-queues-sdk, online-evaluations-composite: environment wording, renamed dataset headings with inbound anchors updated, and the tracing-projects links fixed. The anchor itself stays on the "Compatibility with older APIs" accordion.
  • Engine pages: one shared first step for Configure Engine. Turning on Engine takes issues:create (Workspace Admin or Editor), configuring it takes issues:update, and the environment spend limit takes an Organization Admin.

Review notes

  • UI labels were checked in the product. Where each setting is stored and the Engine permissions were checked in the product source.
  • Engine environment selection: an earlier note here said the Engine page had no environment selector. Checked against the v0.18.0 release: the agent's Engine page shows an environment switcher at the top, so every step that opens Configure Engine now says to select the environment there first: engine-setup, engine, engine-fix-validation, engine-notifications, engine-red-teaming, engine-repositories, and engine-self-hosted-github. A Cloud-only feature flag can replace this page with a goals board that has no switcher. It is always off on self-hosted, so the step holds for every self-hosted reader. Cloud rollout of that flag is not confirmed.
  • Brought up to date: after docs(langsmith): remove the self-hosted split from agent docs #7523 was squash-merged, main was merged into this branch. Ten files conflicted because this branch still held PR 1's original commits. Each was resolved by keeping main's version and reapplying only this PR's changes, so the diff against main is now this PR's 22 files and nothing from PR 1 is applied twice. Earlier, docs(langsmith): clarify trace retention periods by hosting option #7723 on main moved the retention and usage limit content from administration-overview to usage-and-billing, so this PR's retention and usage limit edits live there.
  • Not changed here: engine-red-teaming "enabled for your project" (unclear what access is granted for), backfill for the Traces trigger, and the remaining "tracing project" prose, which the next PRs cover.

Related

Checklist

  • I read the contributing guidelines, including the language policy
  • I checked the rendered pages locally with docs dev or on the preview deployment
  • I ran every changed code example, or none changed
  • make lint_prose passes on the changed files
  • make broken-links-with-anchors passes, or no links, pages, or navigation changed
  • I updated src/docs.json navigation and redirects, or no page was added, moved, or removed

Prepared with Claude Code.

Self-hosted LangSmith v0.18 moves every workspace to agents, so the
docs no longer need to describe two kinds of workspace.

- Rename the "Agent-based workspaces" nav group to "Agent Registry".
- Delete the project-based, agent-based, and availability note snippets.
- Turn the workspace migration page into a self-hosted v0.18 upgrade page.
- Remove the application framing from the admin overview, resource
  hierarchy, and workload isolation pages.
- Remove "agent-based workspace" qualifiers and project-based
  alternatives across the tracing and setup pages.
- Update AGENTS.md to match.

Hold until self-hosted v0.18 ships. Part of DOC-1797.
- Replace "tracing project, or agent environment" alternatives with
  agent environments across the evaluator, rules, Engine, dashboard,
  alert, and annotation queue pages.
- Start the billing automation step from Tracing in the agent's
  environment, and select an Agent when creating an annotation queue.
- Document the Agent resource tag key and its permission exception.
- Move the self-hosted upgrade page to Self-hosted > Manage an instance
  and rename it to self-host-agent-migration, with a redirect from the
  old URL.

Part of DOC-1797.
An agent environment's evaluator tab is labeled Online Evaluators, not
Evaluators. Use that name wherever a page describes an agent environment,
and name both tabs where a page covers datasets and environments.

Also describe the evaluator spend widgets on those tabs without quoting
the per-view chart titles, and select an agent environment instead of a
tracing project in the decision model evaluator steps.

Part of DOC-1797.
- The daily evaluator spend chart has no Project / Dataset toggle in an
  agent workspace, so describe it as broken down by evaluator.
- Use the widget's title, Weekly evaluator spend limits, and describe
  each row as an evaluator and the agent environment or dataset it is
  attached to.
- Say where the scoped widgets appear: a dataset's Evaluators tab and
  an agent environment's Online Evaluators tab on the Tracing page.

Part of DOC-1797.
When an evaluator reaches its weekly spend limit, describe what stops in
terms of the attached agent environment or dataset instead of a project.

Part of DOC-1797.
- Correct the dashboard chart source labels for agent workspaces: the
  button reads "+ Select agent or dataset" and the tabs are Tracing and
  Datasets.
- On the self-hosted upgrade page, say that the migration runs on the
  existing queue workers, and name LangSmith as the actor in three
  sentences.
- Remove a contraction and a future tense on lines this PR changed.

Part of DOC-1797.
Apply a persona review of the upgrade page:

- Lead with the benefit, name the four environments, and note that new
  installations with authentication start with agents.
- Put the no-downgrade rule in a warning, and move the one-minor-version
  rule into "Before you upgrade".
- Describe the migration steps without internal job terms, add a Helm
  values snippet for pausing, and say that pausing does not end it.
- Add a section on checking whether a workspace has migrated.
- Spell out the identifier suffix, name the projects that stay as they
  are, and say what happens to tracing projects created later.
- Use "agent switcher", fix two link labels, rename the sidebar entry,
  and drop the icon.

Two TODO(verify) comments remain for open product questions.

Part of DOC-1797.
- Rename trace-ingestion-project to create-agent-for-tracing, which
  says what the snippet does, and update the quickstart and the tracing
  tutorial that import it.
- Replace the nonexistent Observability option with the real path:
  Connect > Connect an existing agent, which opens the Connect an agent
  dialog. Name both ways into the new agent menu: + Agent with All
  agents selected, or the + button in the agent switcher.
- Link the create-an-agent section and the agent switcher, and tighten
  the closing paragraph.

Part of DOC-1797.
Link the first mention of a deployment, and remove "Experiments stay
experiments," which repeated the sentence before it.

Part of DOC-1797.
… ABAC

Replace the agent-based workspace qualifier that came in from main with
a plain statement: the project resource type covers agent environments,
and a policy for it applies to each environment separately.

Part of DOC-1797.
- Upgrade page: replace the two TODO(verify) comments with what
  engineering confirmed. Tracing keeps working during the migration,
  and an authType: none install that later enables basic auth or SSO
  then runs the migration. New installations on v0.18 or later start
  with agents, with or without authentication.
- Alerts: give the real path to the Alerts bell (agent switcher,
  Tracing, environment) and the Monitoring > Alerts route, and replace
  "Alerts > Create Alert" with "+ Alert" in the integration steps.
- Remove leftover project alternatives in annotation-queues,
  evaluators, audit-logs, and deploy-to-cloud. Relabel the audit log
  "Tracing projects" row as "Agent environments" and add
  create_tracer_session.
- Say what attach-user-feedback's linked requirements are, and drop a
  vague sentence in annotate-code.

Part of DOC-1797.
…d keys

- billing: rewrite default retention as per-environment steps, rename the
  section anchor, and remove the outdated retention screenshot
- usage-and-billing, data-purging-compliance, evaluators,
  administration-overview: set retention per agent environment and update
  links to the renamed anchor
- composite-evaluators-ui: start from an agent environment or dataset
- create-account-api-key: drop legacy project upsert and destination
  sentences, link ABAC policies
- dashboards: match the Monitoring > Analytics tab and the Workspace >
  Dashboards page, use environment labels, and note the Feedback Key limit
…ataset pages

- Use "select your agent, go to Tracing, and select the environment" on
  the online evaluator, rules, insights, and dataset pages
- Name the Online Evaluators tab on the Tracing page in decision-model,
  LLM-as-a-judge, and tuned evaluator steps
- Note that Logs is in the far-right Actions column of the evaluators
  table, and tighten the backfill and log sentences
- tuned-evaluators: add an evaluator to an agent environment, not a
  tracing project
- manage-datasets-in-application: add data from an agent environment
- Fix "+New", "click on", contractions, and future tense where touched
…d webhooks

- Claude Code, Cowork, Cursor, and Codex: note that the plugins always
  send to the agent's Production environment, give the Tracing path, and
  scope the ABAC deny policy to the agent environment
- set-up-resource-tags: list taggable resources by their picker labels,
  starting with Agent Environments
- set-up-hierarchy: drop the unsupported ABAC claim and the upgrade
  pointer, and fix a paragraph break and tense
- view-traces: open with the path to an environment's traces
- webhooks: scope rules to the agent environment and describe
  session_id as the environment's tracer_session_id
…tting

- online-evaluations-multi-turn: idle time is set on the agent
  environment and shown read-only under Advanced in the evaluator form,
  so step 6 checks it and points to Edit tracing settings. Use Trigger On
  for the Runs/Threads choice, split the source step, click Create, and
  tidy step formatting
- online-evaluations-llm-as-judge: use Trigger On in the backfill steps
- rules: describe the idle time as an agent environment setting
@github-actions github-actions Bot added langsmith For docs changes to LangSmith internal labels Nov 15, 2026

@robin-agent robin-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Robin Review found 1 potential issue.

Open in Robin • View Robin trace

Comment thread src/langsmith/rules.mdx Resolved
…nt environments

- tuned-evaluators: attach to agent environments, and rename the two
  troubleshooting headings
- administration-overview: Agent Environments scope row, retention and
  evaluator opt-out per environment, per-environment usage limits with the
  UI's Project label explained, and drop "create projects" from tracing keys
- view-traces, manage-datasets-in-application: format selection persists
  per environment; rename the dataset headings and update inbound anchors
- evaluator-spend: attached agent environments and datasets, and rename
  the override heading and anchor
- online-evaluations-composite, annotation-queues-sdk: replace tracing
  project links; project_id is the environment's Tracing Project ID
- Engine pages: one first step for Configure Engine, per-environment
  Engine configuration in byoc-migration, and the issues:create roles in
  engine-setup

@robin-agent robin-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Robin Review found 1 potential issue.

Open in Robin • View Robin trace

Comment thread src/langsmith/engine-setup.mdx Resolved
- administration-overview: replace the resource tags vs. commit tags note
  with a pointer to commit tags, say the Project list shows each
  environment's tracer_session_name, and use "per-agent-environment"
- engine-setup, engine: configuring Engine needs issues:update and the
  environment spend limit needs an Organization Admin; label the
  agent environment limit consistently
- rules: say where to change the shared thread idle time
- annotation-queues-sdk: give the Tracing path for project_id and
  thread_id
- tuned-evaluators: Workspace Admins and Editors attach tuned evaluators
- byoc-migration, manage-datasets-in-application: agent wording

@fjmorris Florence Morris (fjmorris) left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tftr robin

Comment thread src/langsmith/engine-setup.mdx Resolved
Comment thread src/langsmith/rules.mdx Resolved

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lauren Hirata Singh (@lnhsingh) ready for you to review

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits

Comment thread src/langsmith/evaluator-spend.mdx Resolved
# Conflicts:
#	src/docs.json
#	src/langsmith/audit-logs.mdx
#	src/langsmith/deploy-to-agent-environment.mdx
#	src/langsmith/workload-isolation.mdx
…ations

Apply the reviewer suggestions on the upgrade page, the agent creation
snippet, the evaluator log steps, and the feedback version floors. Replace
the audit log note about operations missing from stable self-hosted with a
0.18.0 entry in the version list.

Refs DOC-1797
…tions

Keep the short UI steps as a numbered list and move the idle time, model,
prompt, and feedback settings into subsections linked from step 6, per
review.

Refs DOC-1797
# Conflicts:
#	src/langsmith/administration-overview.mdx
#	src/langsmith/billing.mdx
#	src/langsmith/data-purging-compliance.mdx
#	src/langsmith/evaluators.mdx
#	src/langsmith/usage-and-billing.mdx
Drop "as a compatibility value" from the tracer_session_name sentence on
all six copies, link the agent variables directly in annotate-code, make
the Connect an agent dialog the link text in agent-registry, and name the
enabled value in self-host-agent-migration.

Refs DOC-1797
# Conflicts:
#	src/langsmith/administration-overview.mdx
Florence Morris (fjmorris) added a commit that referenced this pull request Nov 22, 2026
## Why

Self-hosted LangSmith v0.18 moves every workspace to agents and
environments, so the docs no longer need to describe two kinds of
workspace. Today, many pages split their content between "agent-based"
and "project-based" workspaces, offer a tracing project as an
alternative to an agent environment, or carry a "Cloud only, not
self-hosted" callout. After v0.18, all of these are wrong for every
reader.

Self-hosted v0.18.0 was released on November 20, 2026. This is the first
of four stacked PRs for DOC-1797, which merge in order: this PR and PR 2
(#7745) on November 21, PR 3 (#7757) on November 23, and PR 4 (#7784) on
November 24 or 25.

## Changes

- **Upgrade page:** rewrote the workspace migration page as "What
changes in LangSmith v0.18", for self-hosted administrators: before
upgrading, how the migration runs and how to pause it, checking whether
a workspace has migrated, how tracing projects map to agents, and what
changes in each workspace. Renamed it to `self-host-agent-migration`
(with a redirect) and moved it to Self-hosted > Manage an instance,
after "Upgrade an installation".
- **Navigation:** renamed the "Agent-based workspaces" group to "Agent
Registry", with the `agent-registry` page labeled "Overview".
- **Workspace-type split removed:** deleted the `project-based-note`,
`agent-based-note`, and `agent-availability-note` snippets. Across about
65 tracing, evaluator, and setup pages, removed "agent-based workspace"
qualifiers, project-based alternatives, and `LANGSMITH_PROJECT`
fallbacks, and replaced "tracing project, or agent environment" wording
with agent environments. No "agent-based workspace" or "project-based"
wording is left, apart from the old migration page redirect.
- **UI labels checked in the product:** the evaluator tab is **Online
Evaluators** on an agent environment, attachments are **Environments &
Datasets**, the Engine limit is **Environment limit**, the dashboard
chart source is **+ Select agent or dataset**, and the **Evaluators**
page spend widgets match an agent workspace. Steps to an environment now
consistently say "select your agent, go to **Tracing**, and select the
environment".
- **Errors already on `main`:** the create-an-agent snippet (renamed
`create-agent-for-tracing`) named an **Observability** option that does
not exist and now gives **Connect** > **Connect an existing agent**. The
alerts steps now give the real path to **Alerts** and say **+ Alert**.
The dashboards page named the chart button and tabs incorrectly, and now
places custom dashboards under **Workspace** > **Dashboards**.
- **Settings now per environment:** retention (`billing`,
`usage-and-billing`, `data-purging-compliance`, `evaluators`), resource
tags (the **Agent** tag key and the **Agent Environments** picker
label), and the tracing plugins, which always send to the agent's
**Production** environment.
- **Reference links:** `audit-logs` relabels the operations row as
"Agent environments", adds `create_tracer_session`, and lists v0.18 for
`create_agent_environment` and `mint_agent_access_token`. Its
`operations` link and the `abac` API link now point to the docs API
reference instead of the Swagger UI.
- **Smaller fixes:** `set-up-hierarchy` and `workload-isolation` anchors
renamed to `#set-up-agents` and `#isolate-by-agent`, the 403
troubleshooting paragraph removed from `log-traces-to-agent`, `webhooks`
describes `session_id` as the environment's `tracer_session_id`,
`attach-user-feedback` states the v0.18 and SDK minimums, the
`observability-quickstart` intro no longer points to compatibility
instructions that the old snippet held, and `AGENTS.md` updates the
navigation map.

## Review notes

- **Upgrade page:** the migration behavior was checked against the final
self-hosted v0.18.0 release (Helm chart `langsmith-0.18.0`, app version
`0.18.2026112100`): the roughly 10-minute start delay that restarts do
not reset, each workspace switching once its migration is verified, the
stop after 24 quiet hours and a clean full check, `authType: none`
installs not being migrated, and pausing with
`config.agentMigration.enabled`, which defaults to `true` and is
unchanged from the release candidates. The check corrected the pause
section: the 24 quiet hours count time spent paused, so after a long
pause the migration can stop soon after it resumes rather than running
another 24 hours. Robin's review then corrected the identifier rule: the
eight-digit suffix is also added when a tracing project already uses one
of the agent's future environment names, such as `checkout-staging` for
`checkout`, which matches the v0.18.0 migrator.
- **Experiments in the migration:** the Helm comment for
`config.agentMigration.enabled` says experiments migrate to agents. They
are only classified as experiments, so the page is correct.
langchain-ai/helm#1357 fixes the comment.
- **Left alone on purpose:** the default **Application** and
**Environment** resource-tag keys, which LangSmith still creates for
every workspace.
- **Out of scope, handled by the next PRs:** project-tied settings
(usage limits, thread idle time, project resource tags, per-attachment
spend limits in `evaluator-spend`), the remaining `LANGSMITH_PROJECT`
mentions, and the remaining "tracing project" prose.
- The full per-page change list and review history are on DOC-1797.

## Related

- Refs
[DOC-1797](https://linear.app/langchain/issue/DOC-1797/scrub-legacy-concepts-from-docs-after-self-hosted-update).
- Follows #7522, which removed the sidebar icons from the same group.

## Checklist

- [x] I read the [contributing
guidelines](https://docs.langchain.com/oss/contributing/documentation),
including the [language
policy](https://docs.langchain.com/oss/python/contributing/overview#language-policy)
- [x] I checked the rendered pages locally with `docs dev` or on the
preview deployment
- [x] I ran every changed code example, or none changed
- [x] `make lint_prose` passes on the changed files
- [x] `make broken-links-with-anchors` passes, or no links, pages, or
navigation changed
- [x] I updated `src/docs.json` navigation and redirects, or no page was
added, moved, or removed

Prepared with Claude Code.
Base automatically changed from fjmorris/DOC-1797-1 to main November 22, 2026 00:12
# Conflicts:
#	src/langsmith/administration-overview.mdx
#	src/langsmith/audit-logs.mdx
#	src/langsmith/engine-repositories.mdx
#	src/langsmith/engine-self-hosted-github.mdx
#	src/langsmith/engine-setup.mdx
#	src/langsmith/engine.mdx
#	src/langsmith/evaluator-spend.mdx
#	src/langsmith/online-evaluations-multi-turn.mdx
#	src/langsmith/self-host-agent-migration.mdx
#	src/langsmith/usage-and-billing.mdx

@robin-agent robin-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Robin Review found 3 potential issues.

Open in Robin • View Robin trace

Comment thread src/langsmith/tuned-evaluators.mdx Resolved
@github-actions

Copy link
Copy Markdown
Contributor

Mintlify preview branch generated: preview-fjmorris-doc-1797-2-99d9b71b

Site preview: https://langchain-5e9cc07a-preview-fjmorris-doc-1797-2-99d9b71b.mintlify.site

Important

Preview links may take a few minutes to start working while the deployment finishes.

Changed documentation pages (preview deep links):

Only the top 5 changed markdown files by diff size are listed.

@fjmorris
Florence Morris (fjmorris) merged commit 1d18652 into main Nov 22, 2026
31 checks passed
@fjmorris
Florence Morris (fjmorris) deleted the fjmorris/DOC-1797-2 branch November 22, 2026 01:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal langsmith For docs changes to LangSmith

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants