Skip to content

fix(highlight-py): pass OTLP exporter timeout in seconds, not milliseconds - #797

Merged
Vadman97 merged 1 commit into
mainfrom
fix/python-otlp-timeout-unit
Oct 6, 2026
Merged

Vadman97 merged 1 commit into
mainfrom
fix/python-otlp-timeout-unit

Conversation

@Vadman97

@Vadman97 Vadman97 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem

In sdk/highlight-py/highlight_io/sdk.py, all three OTLP exporters were constructed with timeout=kwargs["schedule_delay_millis"] (= 5000). The OpenTelemetry Python gRPC exporters' timeout parameter is documented as "Backend request timeout in seconds" and is passed straight to the gRPC call — so each export RPC could block for up to 5000 seconds (~83 minutes) instead of the intended ~5.

Failure mode: if the OTLP backend is slow or hung, the single BatchSpanProcessor / BatchLogRecordProcessor worker thread blocks for over an hour per batch, the span/log queues back up and start dropping data, and H.flush() (called on every serverless invocation in integrations/serverless.py) can hang for the same duration.

Fix

Pass kwargs["schedule_delay_millis"] / 1000 (5 s) to the trace, log, and metric exporters.

Validation

  • python3 -m py_compile passes on the modified file.
  • Exporter timeout semantics verified against the pinned opentelemetry-exporter-otlp-proto-grpc 1.28.2 source.

Note

Overview
Fixes OTLP gRPC exporter timeout for traces, logs, and metrics in highlight_io/sdk.py by passing schedule_delay_millis / 1000 (~5 s) instead of the raw millisecond value (~5000 s).

That aligns with OpenTelemetry’s seconds-based exporter timeout, so slow or hung OTLP backends no longer block batch export workers and H.flush() (e.g. serverless) for up to ~83 minutes per RPC.

Reviewed by Cursor Bugbot for commit 3d32612. Bugbot is set up for automated code reviews on this repo. Configure here.

…conds

The OpenTelemetry gRPC exporters' timeout parameter is in seconds, but
the SDK passed the schedule_delay_millis value (5000) directly, so each
trace/log/metric export RPC could block for up to 5000 seconds (~83
minutes) instead of ~5. A slow or hung OTLP backend would stall the
BatchSpanProcessor/BatchLogRecordProcessor worker for over an hour per
batch — backing up the queues, dropping data, and hanging H.flush()
(called on every serverless invocation) for the same duration.

Divide by 1000 to pass seconds.
@Vadman97
Vadman97 requested a review from a team as a code owner October 5, 2026 20:06
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
628 551 88% 0% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
sdk/highlight-py/highlight_io/sdk.py 90% 🟢
TOTAL 90% 🟢

updated for commit: 3d32612 by action🐍

@Vadman97

Vadman97 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Bug: real, and the worst case is worse than the description says.

  • The pinned opentelemetry-exporter-otlp-proto-grpc==1.28.2 treats timeout as seconds ("Backend request timeout in seconds") and passes it directly to stub.Export(timeout=...). I checked this against a local gRPC server whose handler sleeps for 3s: with timeout=5.0 the export succeeds after 3s, and with timeout=1.0 it gets DEADLINE_EXCEEDED.
  • DEADLINE_EXCEEDED is on the exporter's retryable list. The exporter tries up to 6 times with 1+2+4+8+16+32 = 63s of backoff, so the worst case per batch was 6 x 5000s + 63s ≈ 8.3h, not 83 min. With this PR it is 6 x 5s + 63s ≈ 93s.
  • About H.flush() (integrations/serverless.py:28): the trace and log force_flush calls are bounded by event.wait(30s) (they return False, but the worker thread stays blocked). The metrics leg has no such bound. MeterProvider.force_flush → reader.collect → OTLPMetricExporter.export ignores timeout_millis (upstream TODO #2663), so before this PR, metrics alone could hold flush() for hours.

Fix: correct.

  • All three exporters are covered (sdk.py:240, :257, :270).
  • The SDK-side values that really are in ms, export_interval_millis and export_timeout_millis (sdk.py:281-282), plus the Batch*Processor **kwargs, are correctly left unchanged.
  • 5000 / 1000 = 5.0 is a float even though the type hint says Optional[int]. That's fine: gRPC accepts floats, and timeout or env is truthy.
  • No change to the public API. schedule_delay_millis isn't user-settable anyway, because kwargs.update(...) at sdk.py:147-153 overwrites whatever the caller passes. That's a pre-existing issue and the docstring at :145 is misleading about it.

Suggestions (optional, none blocking)

  • Tying the exporter RPC timeout to schedule_delay_millis is a coincidence of values. A named constant (e.g. _EXPORT_RPC_TIMEOUT_S = 5) would stop this from coming back.
  • Add a one-line regression test: test_sdk.py already mocks OTLPSpanExporter/OTLPLogExporter, so it can assert call_args.kwargs["timeout"] == 5.
  • Follow-up: in serverless, flush() can still take roughly 30s + 30s + 93s when the backend hangs. Consider passing an explicit timeout_millis to the three force_flush calls at sdk.py:301-304. Note this does not bound the metrics leg, because the metric exporter ignores timeout_millis.

What I ran: I installed the package in a venv and ran pytest tests/test_sdk.py tests/test_utils_dict.py: 231 passed, 1 failed. The failure is test_record_exception[True-None----None] (assert 650 == 10), and it fails the same way on origin/main, so this PR didn't cause it.

Reviewed with Claude Code

@Vadman97
Vadman97 merged commit 1f7290a into main Oct 6, 2026
18 of 27 checks passed
@Vadman97
Vadman97 deleted the fix/python-otlp-timeout-unit branch October 6, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants