Repository navigation
fix(testing): trace tracked pytest tests without LANGSMITH_TRACING - #112
Open
ARIYAN BHAKAT (arywk40-hue) wants to merge 1 commit into
Open
ARIYAN BHAKAT (arywk40-hue) wants to merge 1 commit into
ARIYAN BHAKAT (arywk40-hue) wants to merge 1 commit into
Conversation
With test tracking on and LANGSMITH_TRACING unset, log_* raised inside @pytest.mark.langsmith tests and experiments were left empty, because _run_test/_arun_test opened a trace without enabling tracing. Enable tracing for the test's own trace unless the context explicitly sets it. Also add the missing spaces in the --langsmith-output error message. Fixes langchain-ai#107
ARIYAN BHAKAT (arywk40-hue)
force-pushed
the
fix/pytest-tracking-without-tracing-env
branch
from
November 18, 2026 10:05
f6184b5 to
7851cb0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #107
Problem
With test tracking on (the default) and
LANGSMITH_TRACINGunset, everyt.log_inputs/t.log_outputs/t.log_feedback/t.trace_feedbackcall inside a@pytest.mark.langsmithtest raisedValueError. Meanwhile the plugin had already created the dataset and an experiment, so each run left an empty experiment behind. The pytest docs only ask users to setLANGSMITH_TEST_SUITE/LANGSMITH_EXPERIMENT, so this setup is easy to hit.Cause:
_run_test/_arun_testopenrh.trace(...)for the test, buttraceonly sets a current run tree when tracing is enabled. Test tracking and tracing are separate switches, so with tracking on and tracing off there was no run tree forlog_*to find. The test's root run was also never posted, even though_end_runlater patches it.Separately, the
--langsmith-output+LANGSMITH_TEST_TRACKING=falseerror message was missing spaces between its string literals (env varLANGSMITH_TEST_TRACKING,remove the'--langsmith-output').Fix
_run_testand_arun_testonly run when test tracking is on, so they now setenabled=Truein the tracing context they already open around the test. An explicittracing_context(enabled=...)around the test still wins: the value is only filled in when the current context leaves it unset (None).Tests
test_log_functions_work_with_tracking_on_and_tracing_env_unset(sync) and the async counterpart: with the tracing env vars unset,log_inputs/log_outputswork inside a tracked test, and the root run is posted to the suite's project.test_explicitly_disabled_tracing_is_respected:tracing_context(enabled=False)still disables tracing.test_langsmith_output_with_tracking_disabled_error_is_readable: checks the full error text.The four new tests fail on
mainand pass with this change.scripts/test(3740 passed),scripts/lintandscripts/formatare clean. I also ran the issue's second repro through the real plugin and the message now reads correctly. The first repro needs a LangSmith API key, so it is covered by the mocked-client unit tests rather than run against the hosted API.