Repository navigation
fix(highlight-py): pass OTLP exporter timeout in seconds, not milliseconds - #797
Merged
Merged
Conversation
…conds The OpenTelemetry gRPC exporters' timeout parameter is in seconds, but the SDK passed the schedule_delay_millis value (5000) directly, so each trace/log/metric export RPC could block for up to 5000 seconds (~83 minutes) instead of ~5. A slow or hung OTLP backend would stall the BatchSpanProcessor/BatchLogRecordProcessor worker for over an hour per batch — backing up the queues, dropping data, and hanging H.flush() (called on every serverless invocation) for the same duration. Divide by 1000 to pass seconds.
Contributor
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
Contributor
Author
|
Bug: real, and the worst case is worse than the description says.
Fix: correct.
Suggestions (optional, none blocking)
What I ran: I installed the package in a venv and ran Reviewed with Claude Code |
abelonogov-ld
approved these changes
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In
sdk/highlight-py/highlight_io/sdk.py, all three OTLP exporters were constructed withtimeout=kwargs["schedule_delay_millis"](= 5000). The OpenTelemetry Python gRPC exporters'timeoutparameter is documented as "Backend request timeout in seconds" and is passed straight to the gRPC call — so each export RPC could block for up to 5000 seconds (~83 minutes) instead of the intended ~5.Failure mode: if the OTLP backend is slow or hung, the single
BatchSpanProcessor/BatchLogRecordProcessorworker thread blocks for over an hour per batch, the span/log queues back up and start dropping data, andH.flush()(called on every serverless invocation inintegrations/serverless.py) can hang for the same duration.Fix
Pass
kwargs["schedule_delay_millis"] / 1000(5 s) to the trace, log, and metric exporters.Validation
python3 -m py_compilepasses on the modified file.opentelemetry-exporter-otlp-proto-grpc1.28.2 source.Note
Overview
Fixes OTLP gRPC exporter timeout for traces, logs, and metrics in
highlight_io/sdk.pyby passingschedule_delay_millis / 1000(~5 s) instead of the raw millisecond value (~5000 s).That aligns with OpenTelemetry’s seconds-based exporter timeout, so slow or hung OTLP backends no longer block batch export workers and
H.flush()(e.g. serverless) for up to ~83 minutes per RPC.Reviewed by Cursor Bugbot for commit 3d32612. Bugbot is set up for automated code reviews on this repo. Configure here.