Skip to content

Postgres memory-authority integration tests fail intermittently in CI (insert counts, deliveries-published waits) #640

Closed

Description

@zlareb1

Company or project name

No response

Component

Memory and search

Describe what's wrong

Tests in tests/integration/test_pg_authority.py fail intermittently in the "pytest integration (Postgres 16 memory authority)" and "(Postgres 18 memory authority)" jobs, on main and on pull requests that do not touch server code; a re-run passes. Three of the last fifteen CI runs on main failed this way, each on a different test:

  • run 43412144069: test_a_redaction_fences_the_claim_still_sending_the_old_text (eventually("the org's deliveries published") timed out) and test_writes_made_together_share_their_clickhouse_inserts (AssertionError: 20)
  • run 43372237566: test_a_redact_reaches_a_revision_by_its_metadata[payload_cleared] (eventually("the org's deliveries published"))
  • run 43362461883: test_a_write_made_unembedded_over_an_embedded_key_retires_its_vectors[memory.set-change0] (eventually(...))

On pull requests the same jobs failed with test_writes_made_together_share_their_clickhouse_inserts (AssertionError: 21 on Postgres 16 in run 43417945601, AssertionError: 20 on Postgres 18 in run 43449264257); both passed on re-run.

Does it reproduce on the latest release or main?

Yes

How to reproduce

Run the Postgres memory-authority integration jobs repeatedly (or re-run a failed one); the failures are intermittent. I have not reproduced them locally.

Expected behavior

The tests pass deterministically: the insert-count assertion does not depend on how the writes batch, and the "deliveries published" waits have a bound that holds on CI runners.

Error message and logs

tests/integration/test_pg_authority.py:404: AssertionError: 20 (and 21); AssertionError: eventually("the org's deliveries published").

Environment

GitHub Actions ubuntu-latest, the CI workflow's Postgres 16 and Postgres 18 memory-authority integration jobs.

Related issues and pull requests

No response

How this was found

An AI tool found it, and I have not reproduced it myself

Activity

  1. rahulsjha commented on Nov 21, 2026

    @rahulsjha
    Contributor

    Hi @zlareb1, I'd like to look into this. I have the Postgres memory-authority setup running locally, so I'd loop tests/integration/test_pg_authority.py to reproduce the flaky insert-count and deliveries-published waits, then fix the race (or the waits) and show repeated green runs. Okay if I take it?

  2. added a commit that references this issue on Nov 21, 2026
  3. rahulsjha commented on Nov 21, 2026

    @rahulsjha
    Contributor

    I went through main's CI history for this before writing a fix, and most of it is already gone.

    Over 400 ci runs on main from 13 to 21 Nov:

    By failure:

    • Insert count (AssertionError: 20–23): fixed by Batching test excludes prior publication inserts #762. The old start marker was the client clock minus 1 s, so inserts from earlier tests were counted. In 600 runs on my fork the count stayed between 6 and 12, well under the assertion.
    • Vector retirement timeouts: fixed by Postgres projection tests publish their starting rows #790, which waits for the first write to publish before changing the key.
    • test_a_redact_reaches_a_revision_by_its_metadata[payload_cleared] and test_a_redaction_fences_the_claim_still_sending_the_old_text: these still have the shape Postgres projection tests publish their starting rows #790 removed elsewhere: two writes to one key back to back, then one wait for all the org's deliveries. All five of their timeouts failed at that wait. Neither has failed since 18 Nov, and I couldn't make them fail on current main.

    #1124 adds #790's "wait for the first publish" step to those two tests and to test_a_fenced_holder_is_waited_out_after_its_delivery_moves_on, which has the same shape. With it, 20 of 20 jobs passed on Postgres 16 and 18, and the three tests passed 800 of 800 looped runs. Current main passes the same runs, though, so this is hardening rather than a proven fix.

    If you'd rather not take the hardening, close #1124 and this issue as fixed by #762 and #790.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions