Repository navigation
rotation-release-id-pollution-check: dual-report alerts to PostHog - #3489
Merged
jakebromberg merged 3 commits intoOct 18, 2026
Merged
jakebromberg merged 3 commits into
jakebromberg merged 3 commits into
Conversation
This was referenced Oct 18, 2026
Closed
…Hog with the per-rotation fingerprint (BS#3410)
…rectly at exit so the token bucket cannot drop it (BS#3410)
…t string, not a list (BS#3410) PostHog's error-tracking processor types `$exception_fingerprint` as one string, so the list form dropped every alert as InvalidProperties. The PostHog half now sends the fingerprint_for() parts joined with ":" (the per-run alert sends `rotation-release-id-pollution-check:run-degraded`); fingerprint_for() is unchanged for Sentry. The flood guard now logs `posthog_send_dropped` when it drops a send, the job-failure report moves into posthog_report_exception(), and self_test() covers the non-DRY_RUN PostHog path, the failure report, fail-open when the client raises, and an empty bucket. The env-var entry moves under its own PostHog heading.
jakebromberg
force-pushed
the
orchestrator/rotation-release-id-pollution-check-dual-report-to
branch
from
October 18, 2026 22:45
9d04ab0 to
c7cbdea
Compare
jakebromberg
deleted the
orchestrator/rotation-release-id-pollution-check-dual-report-to
branch
October 18, 2026 22:52
This was referenced Oct 19, 2026
[Epic] Sentry exit: errors and logs to PostHog, paging to CloudWatch, tracing dropped — wave 1
#3487
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3410
Part of #3393 (Sentry exit), slice B8. This is observability plumbing, so the post-launch hardening freeze (project #32) does not apply.
jobs/rotation-release-id-pollution-checknow mirrors its Sentryemit_alert(message) andmain()exception sites to PostHog error tracking through a fail-open posthog-python client keyed onPOSTHOG_ERRORS_API_KEY(optional; unset means Sentry-only, behavior unchanged). Message-only alerts are sent as a synthesized$exceptionwhose$exception_fingerprintis thefingerprint_for(kind, rotation_id)parts joined with:into one string (PostHog types the property as a single string and drops a list as InvalidProperties), e.g.rotation-release-id-pollution-check:mismatch:123, androtation-release-id-pollution-check:run-degradedfor the per-run alert, so PostHog groups the same way Sentry does, withrepo/tool/run_idproperties. Sends go through a small token bucket (burst 50, 1/s); a dropped send logsposthog_send_dropped. The client is shut down (flushed) infinally. DRY_RUN suppresses PostHog exactly where it suppresses Sentry. Sentry stays.Known limit: posthog-python's
shutdown()has no drain timeout, so a PostHog outage at exit can hold the process until the SDK's own request timeouts give up. Left as is.Tests: the payload, the string fingerprint, the bucket, no-key, DRY_RUN, the non-DRY_RUN PostHog path, the job-failure report, fail-open when the client raises, and the empty-bucket drop are pinned in
self_test(), which the jesttest:unitshell-out and the Dockerfile build run. Local:job.py --self-testand the job's jest suite pass, prettier is clean.npm run ci:testmockwas not run locally (Docker was not running); GitHub CI runs the integration suite on this PR, since the repo is public.