Skip to content

fix(observability-python): prevent logging recursion with coexisting agents - #637

Merged
Vadman97 merged 1 commit into
mainfrom
fix/python-logging-recursion
Jun 22, 2026
Merged

Vadman97 merged 1 commit into
mainfrom
fix/python-logging-recursion

Conversation

@Vadman97

@Vadman97 Vadman97 commented Jun 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Python SDK attaches an OpenTelemetry LoggingHandler to the root logger (_otel/configuration.py), so it sees every log record in the process — including records produced by other logging instrumentation and by our own export pipeline. When another logging agent is present (this was found with the New Relic Python agent, but the class of bug is general), a single application log line could spiral into an unbounded log storm / recursion.

Root cause

Two compounding issues, reproduced end-to-end against an unpatched New Relic agent:

  1. Foreign record attributes. The NR agent leaves record._nr_original_message — a function wrapper — on every LogRecord. Our handler forwarded it, OpenTelemetry's attribute validation rejected the non-serializable type and emitted its own WARNING, NR re-dispatched that warning through the root logger, and it was re-captured here → feedback loop.
  2. Re-entrancy. Any log emitted synchronously while inside emit (the attribute warning above, exporter connection errors, gRPC logs) was re-captured by our own handler.

Fix

New LDLoggingHandler (_otel/logging_handler.py), a hardened LoggingHandler that is safe to install on the root logger alongside any other logging instrumentation:

  • _get_attributes drops attribute values that are not valid OpenTelemetry types before they reach validation — so foreign junk (NR's function wrapper, arbitrary objects) can never trigger the warning that seeds the loop.
  • emit uses a module-level ContextVar re-entrancy guard, so logs produced by the export pipeline are dropped instead of re-captured.

Also makes root-logger registration idempotent (install_on_root_logger): repeated initialization in one process (a repeated set_config, gunicorn preload + worker, test suites) no longer stacks duplicate handlers that double-export every record.

Why handle it in the SDK rather than a disable flag

instrument_logging=False already exists as a coarse opt-out, but that disables LD log capture to dodge a bug. Since customers run the released NR agent (and Datadog/Sentry/etc.) that we can't patch, the SDK should be robust by default. This keeps instrument_logging purely as a log-ownership/dedup switch.

Validation

Reproduced the storm against an unpatched NR agent and confirmed the fix eliminates it:

ldobserve handler Invalid type FunctionWrapper storm lines
stock LoggingHandler 43 (recursion)
re-entrancy guard only 1 (loop broken)
attribute sanitization only 0
LDLoggingHandler (this PR) 0

New regression tests (_otel/test_logging_handler.py, all passing) cover:

  • non-serializable attributes are dropped, serializable ones preserved
  • emit is not re-entrant under a pipeline that logs synchronously during export
  • root-logger registration is idempotent

black --check clean.

Notes / out of scope

  • The toxic _nr_original_message attribute and missing re-entrancy guard are also genuinely New Relic agent bugs; those are being fixed on the NR side separately. This PR is the SDK's defense-in-depth so it's robust regardless of NR's version.
  • A related Flask integration footgun (app.run(debug=True) reloader runs module-level init in two processes) is fundamentally app-side — the idempotent registration here covers in-process double-init, but the cross-process reloader case is best handled by running via flask --app app run (no reloader).

🤖 Generated with Claude Code


Note

Medium Risk
Changes how every application log is exported on the root logger; behavior is defensive but could drop some custom record attributes that are not OTEL-serializable.

Overview
Replaces the stock OpenTelemetry LoggingHandler with LDLoggingHandler, which is safe on the root logger when other agents (e.g. New Relic) are present. emit uses a ContextVar re-entrancy guard so logs from the export pipeline are not captured again; _get_attributes strips non–OpenTelemetry-serializable record fields so foreign attributes cannot trigger OTEL validation warnings and feedback loops.

Root registration now goes through install_on_root_logger, which removes any existing LDLoggingHandler before adding one so repeated in-process SDK init does not stack handlers or double-export logs. configuration.py wires these changes when instrument_logging is enabled.

Adds test_logging_handler.py for attribute filtering, re-entrancy under synchronous export logging, and idempotent installation.

Reviewed by Cursor Bugbot for commit a867d5e. Bugbot is set up for automated code reviews on this repo. Configure here.

…agents

The SDK attaches an OpenTelemetry LoggingHandler to the root logger, so it
sees every log record in the process. When another logging instrumentation
is also present (e.g. the New Relic Python agent) this could recurse:

- NR leaves a non-serializable attribute (_nr_original_message, a function
  wrapper) on every LogRecord. OTel attribute validation rejects the type and
  emits its own WARNING, which NR re-dispatches through the root logger where
  it is re-captured -> unbounded feedback loop / log storm.
- Any log emitted synchronously by our own export pipeline (attribute
  warnings, exporter errors) was re-captured by our handler.

Introduce LDLoggingHandler, a hardened LoggingHandler that:
- drops non-serializable record attributes before OTel validation, and
- guards emit() against re-entrancy via a context-local flag.

Also make root-logger registration idempotent so repeated initialization in
one process (repeated set_config, gunicorn preload+worker, test suites) never
stacks duplicate handlers that double-export every record.

Adds regression tests covering all three behaviors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Vadman97
Vadman97 requested a review from a team as a code owner June 18, 2026 19:56
@github-actions

Copy link
Copy Markdown
Contributor

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
1327 1151 87% 0% 🟢

New Files

File Coverage Status
sdk/@launchdarkly/observability-python/ldobserve/_otel/logging_handler.py 100% 🟢
sdk/@launchdarkly/observability-python/ldobserve/_otel/test_logging_handler.py 100% 🟢
TOTAL 100% 🟢

Modified Files

File Coverage Status
sdk/@launchdarkly/observability-python/ldobserve/_otel/configuration.py 52% 🟢
TOTAL 52% 🟢

updated for commit: a867d5e by action🐍

@Vadman97
Vadman97 merged commit b66da6c into main Jun 22, 2026
25 checks passed
@Vadman97
Vadman97 deleted the fix/python-logging-recursion branch June 22, 2026 15:20
Vadman97 pushed a commit that referenced this pull request Jun 22, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>launchdarkly-observability-python: 1.2.4</summary>

##
[1.2.4](launchdarkly-observability-python-1.2.3...launchdarkly-observability-python-1.2.4)
(2026-06-22)


### Bug Fixes

* **observability-python:** prevent logging recursion with coexisting
agents
([#637](#637))
([b66da6c](b66da6c))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Low Risk**
> Version and changelog/provenance metadata only; no source changes in
the diff.
> 
> **Overview**
> Release Please bumps **launchdarkly-observability-python** from
**1.2.3** to **1.2.4** in `.release-please-manifest.json`,
`pyproject.toml`, `PROVENANCE.md`, and adds the **1.2.4** changelog
entry.
> 
> The release notes document bug fix **#637**: prevent **logging
recursion** when the SDK runs alongside **other observability/logging
agents** (no implementation changes appear in this PR diff—only
packaging and release metadata).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
61bfff2. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants