Repository navigation
fix(observability-python): prevent logging recursion with coexisting agents - #637
Merged
Merged
Conversation
…agents The SDK attaches an OpenTelemetry LoggingHandler to the root logger, so it sees every log record in the process. When another logging instrumentation is also present (e.g. the New Relic Python agent) this could recurse: - NR leaves a non-serializable attribute (_nr_original_message, a function wrapper) on every LogRecord. OTel attribute validation rejects the type and emits its own WARNING, which NR re-dispatches through the root logger where it is re-captured -> unbounded feedback loop / log storm. - Any log emitted synchronously by our own export pipeline (attribute warnings, exporter errors) was re-captured by our handler. Introduce LDLoggingHandler, a hardened LoggingHandler that: - drops non-serializable record attributes before OTel validation, and - guards emit() against re-entrancy via a context-local flag. Also make root-logger registration idempotent so repeated initialization in one process (repeated set_config, gunicorn preload+worker, test suites) never stacks duplicate handlers that double-export every record. Adds regression tests covering all three behaviors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
☂️ Python Coverage
Overall Coverage
New Files
Modified Files
|
abelonogov-ld
approved these changes
Jun 19, 2026
Merged
Vadman97
pushed a commit
that referenced
this pull request
Jun 22, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>launchdarkly-observability-python: 1.2.4</summary> ## [1.2.4](launchdarkly-observability-python-1.2.3...launchdarkly-observability-python-1.2.4) (2026-06-22) ### Bug Fixes * **observability-python:** prevent logging recursion with coexisting agents ([#637](#637)) ([b66da6c](b66da6c)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Version and changelog/provenance metadata only; no source changes in the diff. > > **Overview** > Release Please bumps **launchdarkly-observability-python** from **1.2.3** to **1.2.4** in `.release-please-manifest.json`, `pyproject.toml`, `PROVENANCE.md`, and adds the **1.2.4** changelog entry. > > The release notes document bug fix **#637**: prevent **logging recursion** when the SDK runs alongside **other observability/logging agents** (no implementation changes appear in this PR diff—only packaging and release metadata). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 61bfff2. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Python SDK attaches an OpenTelemetry
LoggingHandlerto the root logger (_otel/configuration.py), so it sees every log record in the process — including records produced by other logging instrumentation and by our own export pipeline. When another logging agent is present (this was found with the New Relic Python agent, but the class of bug is general), a single application log line could spiral into an unbounded log storm / recursion.Root cause
Two compounding issues, reproduced end-to-end against an unpatched New Relic agent:
record._nr_original_message— a function wrapper — on everyLogRecord. Our handler forwarded it, OpenTelemetry's attribute validation rejected the non-serializable type and emitted its ownWARNING, NR re-dispatched that warning through the root logger, and it was re-captured here → feedback loop.emit(the attribute warning above, exporter connection errors, gRPC logs) was re-captured by our own handler.Fix
New
LDLoggingHandler(_otel/logging_handler.py), a hardenedLoggingHandlerthat is safe to install on the root logger alongside any other logging instrumentation:_get_attributesdrops attribute values that are not valid OpenTelemetry types before they reach validation — so foreign junk (NR's function wrapper, arbitrary objects) can never trigger the warning that seeds the loop.emituses a module-levelContextVarre-entrancy guard, so logs produced by the export pipeline are dropped instead of re-captured.Also makes root-logger registration idempotent (
install_on_root_logger): repeated initialization in one process (a repeatedset_config, gunicorn preload + worker, test suites) no longer stacks duplicate handlers that double-export every record.Why handle it in the SDK rather than a disable flag
instrument_logging=Falsealready exists as a coarse opt-out, but that disables LD log capture to dodge a bug. Since customers run the released NR agent (and Datadog/Sentry/etc.) that we can't patch, the SDK should be robust by default. This keepsinstrument_loggingpurely as a log-ownership/dedup switch.Validation
Reproduced the storm against an unpatched NR agent and confirmed the fix eliminates it:
Invalid type FunctionWrapperstorm linesLoggingHandlerLDLoggingHandler(this PR)New regression tests (
_otel/test_logging_handler.py, all passing) cover:emitis not re-entrant under a pipeline that logs synchronously during exportblack --checkclean.Notes / out of scope
_nr_original_messageattribute and missing re-entrancy guard are also genuinely New Relic agent bugs; those are being fixed on the NR side separately. This PR is the SDK's defense-in-depth so it's robust regardless of NR's version.app.run(debug=True)reloader runs module-level init in two processes) is fundamentally app-side — the idempotent registration here covers in-process double-init, but the cross-process reloader case is best handled by running viaflask --app app run(no reloader).🤖 Generated with Claude Code
Note
Medium Risk
Changes how every application log is exported on the root logger; behavior is defensive but could drop some custom record attributes that are not OTEL-serializable.
Overview
Replaces the stock OpenTelemetry
LoggingHandlerwithLDLoggingHandler, which is safe on the root logger when other agents (e.g. New Relic) are present.emituses aContextVarre-entrancy guard so logs from the export pipeline are not captured again;_get_attributesstrips non–OpenTelemetry-serializable record fields so foreign attributes cannot trigger OTEL validation warnings and feedback loops.Root registration now goes through
install_on_root_logger, which removes any existingLDLoggingHandlerbefore adding one so repeated in-process SDK init does not stack handlers or double-export logs.configuration.pywires these changes wheninstrument_loggingis enabled.Adds
test_logging_handler.pyfor attribute filtering, re-entrancy under synchronous export logging, and idempotent installation.Reviewed by Cursor Bugbot for commit a867d5e. Bugbot is set up for automated code reviews on this repo. Configure here.