You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Postgres memory-authority integration tests fail intermittently in CI (insert counts, deliveries-published waits) #640
Tests in tests/integration/test_pg_authority.py fail intermittently in the "pytest integration (Postgres 16 memory authority)" and "(Postgres 18 memory authority)" jobs, on main and on pull requests that do not touch server code; a re-run passes. Three of the last fifteen CI runs on main failed this way, each on a different test:
run 43412144069: test_a_redaction_fences_the_claim_still_sending_the_old_text (eventually("the org's deliveries published") timed out) and test_writes_made_together_share_their_clickhouse_inserts (AssertionError: 20)
run 43372237566: test_a_redact_reaches_a_revision_by_its_metadata[payload_cleared] (eventually("the org's deliveries published"))
run 43362461883: test_a_write_made_unembedded_over_an_embedded_key_retires_its_vectors[memory.set-change0] (eventually(...))
On pull requests the same jobs failed with test_writes_made_together_share_their_clickhouse_inserts (AssertionError: 21 on Postgres 16 in run 43417945601, AssertionError: 20 on Postgres 18 in run 43449264257); both passed on re-run.
Does it reproduce on the latest release or main?
Yes
How to reproduce
Run the Postgres memory-authority integration jobs repeatedly (or re-run a failed one); the failures are intermittent. I have not reproduced them locally.
Expected behavior
The tests pass deterministically: the insert-count assertion does not depend on how the writes batch, and the "deliveries published" waits have a bound that holds on CI runners.
Hi @zlareb1, I'd like to look into this. I have the Postgres memory-authority setup running locally, so I'd loop tests/integration/test_pg_authority.py to reproduce the flaky insert-count and deliveries-published waits, then fix the race (or the waits) and show repeated green runs. Okay if I take it?
Insert count (AssertionError: 20–23): fixed by Batching test excludes prior publication inserts #762. The old start marker was the client clock minus 1 s, so inserts from earlier tests were counted. In 600 runs on my fork the count stayed between 6 and 12, well under the assertion.
test_a_redact_reaches_a_revision_by_its_metadata[payload_cleared] and test_a_redaction_fences_the_claim_still_sending_the_old_text: these still have the shape Postgres projection tests publish their starting rows #790 removed elsewhere: two writes to one key back to back, then one wait for all the org's deliveries. All five of their timeouts failed at that wait. Neither has failed since 18 Nov, and I couldn't make them fail on current main.
#1124 adds #790's "wait for the first publish" step to those two tests and to test_a_fenced_holder_is_waited_out_after_its_delivery_moves_on, which has the same shape. With it, 20 of 20 jobs passed on Postgres 16 and 18, and the three tests passed 800 of 800 looped runs. Current main passes the same runs, though, so this is hardening rather than a proven fix.
If you'd rather not take the hardening, close #1124 and this issue as fixed by #762 and #790.
Activity
rahulsjha commentedon Nov 21, 2026
Hi @zlareb1, I'd like to look into this. I have the Postgres memory-authority setup running locally, so I'd loop
tests/integration/test_pg_authority.pyto reproduce the flaky insert-count and deliveries-published waits, then fix the race (or the waits) and show repeated green runs. Okay if I take it?evidence: loom ClickHouse#640
rahulsjha commentedon Nov 21, 2026
I went through
main's CI history for this before writing a fix, and most of it is already gone.Over 400
ciruns onmainfrom 13 to 21 Nov:By failure:
AssertionError: 20–23): fixed by Batching test excludes prior publication inserts #762. The old start marker was the client clock minus 1 s, so inserts from earlier tests were counted. In 600 runs on my fork the count stayed between 6 and 12, well under the assertion.test_a_redact_reaches_a_revision_by_its_metadata[payload_cleared]andtest_a_redaction_fences_the_claim_still_sending_the_old_text: these still have the shape Postgres projection tests publish their starting rows #790 removed elsewhere: two writes to one key back to back, then one wait for all the org's deliveries. All five of their timeouts failed at that wait. Neither has failed since 18 Nov, and I couldn't make them fail on currentmain.#1124 adds #790's "wait for the first publish" step to those two tests and to
test_a_fenced_holder_is_waited_out_after_its_delivery_moves_on, which has the same shape. With it, 20 of 20 jobs passed on Postgres 16 and 18, and the three tests passed 800 of 800 looped runs. Currentmainpasses the same runs, though, so this is hardening rather than a proven fix.If you'd rather not take the hardening, close #1124 and this issue as fixed by #762 and #790.