Skip to content

Postgres redaction tests publish a key's first write before rewriting it - #1124

Closed
rahulsjha wants to merge 1 commit into
ClickHouse:mainfrom
rahulsjha:fix/pg-authority-flakes
Closed

rahulsjha wants to merge 1 commit into
ClickHouse:mainfrom
rahulsjha:fix/pg-authority-flakes

Conversation

@rahulsjha

@rahulsjha rahulsjha commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #640.

#640: CI history on main and repeated fork runs

What I found

Most of #640 was already fixed on main before this PR. I went through the ci runs on main from 13 to 21 November (400 runs) and sorted every failure of the two Postgres memory-authority jobs by the test that failed:

Window on main Runs Runs that failed with an #640 signature
13 Nov 07:26 to 18 Nov 04:22 UTC (before #790) 159 20 (9 insert counts, 11 "deliveries published" timeouts)
18 Nov 04:22 to 21 Nov 16:35 UTC (after #762, #781, #790, #805) 240 0

The 4 Postgres-job failures on main since 18 November are unrelated: a port bind clash in the placement drill (44718606846, both jobs), SESSION_PROTECTION_REQUIRED (44667907964) and a marked-rollup migration test (44590192615).

Per failure in the issue:

  • test_writes_made_together_share_their_clickhouse_inserts (AssertionError: 20/21/22/23). Fixed by Batching test excludes prior publication inserts #762. The old marker was the client's clock minus one second, so inserts from earlier tests were counted too (21 to 23 is more inserts than writes). Last failure on main: 16 Nov (43971062118). I left the test alone. In 300 fork runs the count was 6 to 12. 12 is the burst ceiling, 3 web workers × 4 tokens (pg_publish_interval_ms 250), so the strict < 20 holds with room to spare.
  • test_a_write_made_unembedded_over_an_embedded_key_retires_its_vectors timeouts. Fixed by Postgres projection tests publish their starting rows #790, which publishes the first write before the change. Last failure on main: 17 Nov (44196293965).
  • test_a_redact_reaches_a_revision_by_its_metadata[payload_cleared] and test_a_redaction_fences_the_claim_still_sending_the_old_text timeouts. Postgres projection tests publish their starting rows #790 did not touch these. Both still write a key twice in a row and then wait once for every delivery of the org. The tracebacks of all five timeouts (43412144069, 43372237566, 43551886431, 43561874651, 43997233336) point at that one _published call. The server log of 43412144069 shows the two memory.set calls 0.3 s apart, then nothing for 15.4 s until the wait ran out. Neither test has failed on main since 18 November, and I could not pin which change on 17 and 18 November made that happen.

The change

A key's first memory.set answers at its Postgres commit, before its flush (#433). The three tests that still write the key a second time straight after now wait for the first write's publication first, the same step #790 added to the other tests in this file:

  • test_a_redact_reaches_a_revision_by_its_metadata (both cases)
  • test_a_redaction_fences_the_claim_still_sending_the_old_text
  • test_a_fenced_holder_is_waited_out_after_its_delivery_moves_on (same shape, has not failed yet)

That is six added lines in tests/integration/test_pg_authority.py. No timeout, retry, assertion, worker setting or production code changes.

This is hardening, not a measured fix: I could not make these tests fail on current main. Reaching that state takes a second write overtaking a first one that is not yet published, and the change removes that path from these tests.

Evidence

I ran the Postgres 16 and 18 memory-authority job setup on my fork with a temporary workflow (fork branches only, not in this PR), 10 jobs per Postgres version per branch:

main (0a00565) this change
Whole test_pg_authority.py (153 tests) 20/20 jobs green 20/20 jobs green
Batching test, 15 runs per job 300/300 passed, flushes 6 to 12 300/300 passed, flushes 6 to 12
The three changed tests (5 cases), 40 runs per job 800/800 iterations green (4,000 test runs) 800/800 iterations green (4,000 test runs)

The "after" runs used a commit with the same tree as this PR's (only the commit message differs).

Both columns are green, so these runs show no regression rather than a fix. The evidence for the original failures is the run IDs from main listed above.

Not changed

While I was going through the logs I saw test_a_redacted_head_is_not_served_until_a_new_value_is_written and test_a_relabel_refused_for_a_head_written_since_its_read_fences_nothing fail with assert 0 >= 1 (last on 17 Nov). They are not part of #640, and this PR leaves them alone.

An AI agent helped with this; I checked the cause and the repeated runs myself.

A key's first memory.set answers at its Postgres commit, before its flush.
test_a_redact_reaches_a_revision_by_its_metadata,
test_a_redaction_fences_the_claim_still_sending_the_old_text and
test_a_fenced_holder_is_waited_out_after_its_delivery_moves_on each wrote
the key a second time straight after, then waited once for the org's
deliveries. The first two timed out at that wait in CI on 13-17 November
("the org's deliveries published"); the server log of run 43412144069
shows the two writes 0.3 s apart, then the 15 s wait running out.

Each test now waits for the first write's publication before the second,
as ClickHouse#790 did for the other tests in this file, so the second write never
overtakes an unpublished first one. No timeout, assertion or production
behavior changes.

Refs ClickHouse#640

Signed-off-by: rahulsjha <132360698+rahulsjha@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 21, 2026 18:26
@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 6:28pm UTC

Request Review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No findings in the three publication barriers. Each establishes the first revision before the overwrite; the redaction and fencing assertions remain intact. Integration execution was not repeated locally.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitHub reports this head as CONFLICTING with main in tests/integration/test_pg_authority.py. The current-head Actions lookup returns no runs, although the pull_request workflow trigger includes main. Please resolve the merge conflict so CI can run.

@rahulsjha

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I went to resolve the conflict, and it turns out #1114 (merged a little before I opened this) already adds the same await _published(pg_conn, org) after the first write in all three tests: test_a_redact_reaches_a_revision_by_its_metadata, test_a_redaction_fences_the_claim_still_sending_the_old_text and test_a_fenced_holder_is_waited_out_after_its_delivery_moves_on. Each of the three conflicts is only my comment line above that call.

With main merged in, this PR would only add three comments, so I'd suggest closing it rather than merging a comment-only change. #640 can be closed as fixed by #762, #790 and #1114, per the CI history in my comment there. If you'd still like the comments kept, say so and I'll push the merge.

@rahulsjha

Copy link
Copy Markdown
Contributor Author

Closing, as #1114 already has these three waits. Thanks for the review.

@rahulsjha rahulsjha closed this Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Postgres memory-authority integration tests fail intermittently in CI (insert counts, deliveries-published waits)

3 participants