Repository navigation
Conversation
A key's first memory.set answers at its Postgres commit, before its flush.
test_a_redact_reaches_a_revision_by_its_metadata,
test_a_redaction_fences_the_claim_still_sending_the_old_text and
test_a_fenced_holder_is_waited_out_after_its_delivery_moves_on each wrote
the key a second time straight after, then waited once for the org's
deliveries. The first two timed out at that wait in CI on 13-17 November
("the org's deliveries published"); the server log of run 43412144069
shows the two writes 0.3 s apart, then the 15 s wait running out.
Each test now waits for the first write's publication before the second,
as ClickHouse#790 did for the other tests in this file, so the second write never
overtakes an unpublished first one. No timeout, assertion or production
behavior changes.
Refs ClickHouse#640
Signed-off-by: rahulsjha <132360698+rahulsjha@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
zlareb1
left a comment
There was a problem hiding this comment.
No findings in the three publication barriers. Each establishes the first revision before the overwrite; the redaction and fencing assertions remain intact. Integration execution was not repeated locally.
zlareb1
left a comment
There was a problem hiding this comment.
GitHub reports this head as CONFLICTING with main in tests/integration/test_pg_authority.py. The current-head Actions lookup returns no runs, although the pull_request workflow trigger includes main. Please resolve the merge conflict so CI can run.
|
Thanks for the review. I went to resolve the conflict, and it turns out #1114 (merged a little before I opened this) already adds the same With main merged in, this PR would only add three comments, so I'd suggest closing it rather than merging a comment-only change. #640 can be closed as fixed by #762, #790 and #1114, per the CI history in my comment there. If you'd still like the comments kept, say so and I'll push the merge. |
|
Closing, as #1114 already has these three waits. Thanks for the review. |
Fixes #640.
What I found
Most of #640 was already fixed on
mainbefore this PR. I went through theciruns onmainfrom 13 to 21 November (400 runs) and sorted every failure of the two Postgres memory-authority jobs by the test that failed:mainThe 4 Postgres-job failures on
mainsince 18 November are unrelated: a port bind clash in the placement drill (44718606846, both jobs),SESSION_PROTECTION_REQUIRED(44667907964) and a marked-rollup migration test (44590192615).Per failure in the issue:
test_writes_made_together_share_their_clickhouse_inserts(AssertionError: 20/21/22/23). Fixed by Batching test excludes prior publication inserts #762. The old marker was the client's clock minus one second, so inserts from earlier tests were counted too (21 to 23 is more inserts than writes). Last failure onmain: 16 Nov (43971062118). I left the test alone. In 300 fork runs the count was 6 to 12. 12 is the burst ceiling, 3 web workers × 4 tokens (pg_publish_interval_ms250), so the strict< 20holds with room to spare.test_a_write_made_unembedded_over_an_embedded_key_retires_its_vectorstimeouts. Fixed by Postgres projection tests publish their starting rows #790, which publishes the first write before the change. Last failure onmain: 17 Nov (44196293965).test_a_redact_reaches_a_revision_by_its_metadata[payload_cleared]andtest_a_redaction_fences_the_claim_still_sending_the_old_texttimeouts. Postgres projection tests publish their starting rows #790 did not touch these. Both still write a key twice in a row and then wait once for every delivery of the org. The tracebacks of all five timeouts (43412144069,43372237566,43551886431,43561874651,43997233336) point at that one_publishedcall. The server log of43412144069shows the twomemory.setcalls 0.3 s apart, then nothing for 15.4 s until the wait ran out. Neither test has failed onmainsince 18 November, and I could not pin which change on 17 and 18 November made that happen.The change
A key's first
memory.setanswers at its Postgres commit, before its flush (#433). The three tests that still write the key a second time straight after now wait for the first write's publication first, the same step #790 added to the other tests in this file:test_a_redact_reaches_a_revision_by_its_metadata(both cases)test_a_redaction_fences_the_claim_still_sending_the_old_texttest_a_fenced_holder_is_waited_out_after_its_delivery_moves_on(same shape, has not failed yet)That is six added lines in
tests/integration/test_pg_authority.py. No timeout, retry, assertion, worker setting or production code changes.This is hardening, not a measured fix: I could not make these tests fail on current
main. Reaching that state takes a second write overtaking a first one that is not yet published, and the change removes that path from these tests.Evidence
I ran the Postgres 16 and 18 memory-authority job setup on my fork with a temporary workflow (fork branches only, not in this PR), 10 jobs per Postgres version per branch:
main(0a00565)test_pg_authority.py(153 tests)main: https://github.com/rahulsjha/loom/actions/runs/44927808740 and https://github.com/rahulsjha/loom/actions/runs/44929651820The "after" runs used a commit with the same tree as this PR's (only the commit message differs).
Both columns are green, so these runs show no regression rather than a fix. The evidence for the original failures is the run IDs from
mainlisted above.Not changed
While I was going through the logs I saw
test_a_redacted_head_is_not_served_until_a_new_value_is_writtenandtest_a_relabel_refused_for_a_head_written_since_its_read_fences_nothingfail withassert 0 >= 1(last on 17 Nov). They are not part of #640, and this PR leaves them alone.An AI agent helped with this; I checked the cause and the repeated runs myself.