Skip to content

fix(testing): trace tracked pytest tests without LANGSMITH_TRACING - #112

Open
ARIYAN BHAKAT (arywk40-hue) wants to merge 1 commit into
langchain-ai:mainfrom
arywk40-hue:fix/pytest-tracking-without-tracing-env
Open

ARIYAN BHAKAT (arywk40-hue) wants to merge 1 commit into
langchain-ai:mainfrom
arywk40-hue:fix/pytest-tracking-without-tracing-env

Conversation

@arywk40-hue

@arywk40-hue ARIYAN BHAKAT (arywk40-hue) commented Nov 18, 2026 •

Copy link
Copy Markdown

Fixes #107

Problem

With test tracking on (the default) and LANGSMITH_TRACING unset, every t.log_inputs / t.log_outputs / t.log_feedback / t.trace_feedback call inside a @pytest.mark.langsmith test raised ValueError. Meanwhile the plugin had already created the dataset and an experiment, so each run left an empty experiment behind. The pytest docs only ask users to set LANGSMITH_TEST_SUITE / LANGSMITH_EXPERIMENT, so this setup is easy to hit.

Cause: _run_test / _arun_test open rh.trace(...) for the test, but trace only sets a current run tree when tracing is enabled. Test tracking and tracing are separate switches, so with tracking on and tracing off there was no run tree for log_* to find. The test's root run was also never posted, even though _end_run later patches it.

Separately, the --langsmith-output + LANGSMITH_TEST_TRACKING=false error message was missing spaces between its string literals (env varLANGSMITH_TEST_TRACKING, remove the'--langsmith-output').

Fix

  • _run_test and _arun_test only run when test tracking is on, so they now set enabled=True in the tracing context they already open around the test. An explicit tracing_context(enabled=...) around the test still wins: the value is only filled in when the current context leaves it unset (None).
  • Added the missing spaces in the plugin error message.

Tests

  • test_log_functions_work_with_tracking_on_and_tracing_env_unset (sync) and the async counterpart: with the tracing env vars unset, log_inputs / log_outputs work inside a tracked test, and the root run is posted to the suite's project.
  • test_explicitly_disabled_tracing_is_respected: tracing_context(enabled=False) still disables tracing.
  • test_langsmith_output_with_tracking_disabled_error_is_readable: checks the full error text.

The four new tests fail on main and pass with this change. scripts/test (3740 passed), scripts/lint and scripts/format are clean. I also ran the issue's second repro through the real plugin and the message now reads correctly. The first repro needs a LangSmith API key, so it is covered by the mocked-client unit tests rather than run against the hosted API.

Copilot AI balanced review requested due to automatic review settings November 18, 2026 09:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

With test tracking on and LANGSMITH_TRACING unset, log_* raised inside
@pytest.mark.langsmith tests and experiments were left empty, because
_run_test/_arun_test opened a trace without enabling tracing. Enable
tracing for the test's own trace unless the context explicitly sets it.

Also add the missing spaces in the --langsmith-output error message.

Fixes langchain-ai#107
@arywk40-hue
ARIYAN BHAKAT (arywk40-hue) force-pushed the fix/pytest-tracking-without-tracing-env branch from f6184b5 to 7851cb0 Compare November 18, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pytest plugin: with test tracking on, log_* raise unless LANGSMITH_TRACING=true; --langsmith-output error text is missing spaces

2 participants