Repository navigation
Replicated erasure tests provision their lease table - #1076
Merged
Merged
Conversation
Memory clearing takes write fences in the fixture database. Provision its KeeperMap lease table with an isolated path so the replicated mutation assertions can run. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Nov 20, 2026
Merged
zlareb1
added a commit
that referenced
this pull request
Nov 21, 2026
Related: #455, #1076, #1077 ### Changelog category - Improvement ### Changelog entry The Postgres authority stores memory heads and revision payloads without fields at their default values, reducing repeated JSON fields while preserving the rows returned to callers. ### Why Most memory fields hold their defaults. Repeating their names and values in every stored JSON row costs space. Sparse-row readers shipped in v1.2.0 and v1.3.0, so the writer can omit those fields in a later release. ### Scope and risk `commit_versions` builds the stored form before taking transaction locks and uses it for head and revision JSON. Full encoded rows still supply identity, admission, audit metadata, delivery targets and returned rows. Only fields in `FIRST_VERSION_DEFAULTS` may be omitted. Identity, content, provenance, version, hashes, required timestamps, memory type and confidence stay stored. The Cell verifier decodes both stored rows before comparing them. Its value, vector, audit, delivery and failure checks remain. Existing usage and redaction updates may store explicit defaults; readers accept either form. The shared replicated-erasure fixture repairs are already on `main`. ### Compatibility, rollout and rollback - Wire and operator steps: none. - Deploy beside builds that read sparse rows; released v1.2.0 and v1.3.0 contain those readers. - A sparse-capable released build reads existing rows after rollback. Reverting the writer requires no rewrite. - No table, migration, setting, worker or generated-file change is needed. ### Tests - All five hosted workflows pass on the published head: CI, Docker smoke, CodeQL, plugin scanner and docs. The rollup has 54 successful checks and two expected publication skips. This includes Postgres 16/18 authority, all integration shards, isolation, mixed-version, unit coverage and Cell acceptance. - The authority integration checks actual sparse head/revision storage, full/sparse point-read equality, list/search/bulk reads, revisions, cross-session defaults, usage and erasure export. - The verifier regression runs both representations through all original fault cases and assertions. Earlier focused validation produced two sparse failures before the repair and 32 passing cases after it; six related authority cases also passed. - Scoped Ruff lint/format and the public prose check pass. Local composition test attempts were refused before execution by the resource guard and remain unrun; final-head behavior is qualified by hosted CI. ### AI assistance in the code - AI-generated: Codex wrote the sparse writer and verifier regression. Independent source reviews checked the full change, published description, released-reader compatibility and current-main composition. - [x] I can explain every line of this diff and how it interacts with the rest of Loom. ### Checklist - [x] Required hosted format, lint, type, unit, coverage and integration gates pass on the published head. - [x] No generated files or migrations change. - [x] No plugin or frontend behavior changes. - [x] Architecture documentation describes sparse storage and in-place updates. - [x] Public text contains no deployment details. - [x] Commits are signed off. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
zlareb1
added a commit
that referenced
this pull request
Nov 21, 2026
Related: #1076 ### Changelog category - CI Fix or Improvement ### Changelog entry Not required. ### Why A restoration test can spend the fake key lock’s wait budget while acquiring an uncontended local lock, before reaching its restore assertions. These cases check restoration and duplicate visibility independently of CPU scheduling latency. ### Scope and risk Replicated-erasure lease provisioning and its unit regression controls are on main. This branch preserves them unchanged. It also scopes person-query observations to the authorization decision, retaining unbounded-read negative controls. No production authorization, lease or erasure rule changes. The clock change stays in one unit-test file. The existing fake lock uses a private function loop whose clock advances during selector waits. The loop, key-lock budget, lease timers and fake table share that clock. Each case cancels and drains newly pending tasks and fails if it leaked one. A control verifies that a contended key lock still refuses at its idle deadline and releases its registry entry. Every existing test body, assertion and deadline value is retained. Product behavior and shared test helpers are unchanged. ### Compatibility, rollout and rollback - Wire, schema, configuration and operator steps: none. - Rollback: revert the fixture change. No persisted data changes. ### Tests This head preserves every preceding concern blob and matches its exact three-way composition with current main. On this head, the restoration clock cases, person-query observation controls and erasure fixture controls pass. Scoped Ruff, Python formatting, committed-range whitespace and public-body prose checks pass. Full required hosted CI and a fresh independent whole review remain required for this exact head. ```sh env LOOM_ENV_FILE= .venv/bin/python -m pytest tests/unit/test_restore_unsupersede.py tests/unit/test_scim_person_query_observation.py tests/unit/test_erasure_replicated_fixture.py -q ``` ### AI assistance in the code - AI-generated: Codex wrote the fixture change and checked its clock boundaries, cancellation controls and task cleanup. - [x] I can explain every line of this diff and how it interacts with the rest of Loom. ### Checklist - [x] Existing assertions and deadline values are retained. - [x] No production code, generated interfaces, migrations or operator settings change. - [x] Commits are signed off. - [ ] Full exact-head GitHub CI passes. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #1070
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
None; integration fixture setup only.
Why
Replicated erasure tests copy the tables they exercise into a private database. Memory clearing also takes write leases in that database, but the fixture omits
_migration_lock. Both clear tests fail withUNKNOWN_TABLEbefore checking their mutation results. The same failures occur onmain.Scope and risk
Create the fixture's lease table from the bootstrap schema, using a KeeperMap path unique to the fixture database. Keep the existing assertion that all copied data tables use replicated engines. The fixture database cleanup removes the coordination table with the data tables.
Compatibility, rollout and rollback
Tests
main,pytest tests/integration/test_erasure_replicated_ch.py -qagainst a private ClickHouse with embedded Keeper reproduces two failures and three passes.pytest tests/unit/test_integration_suite_targets.py -q: 354 passed.SIGSEGVduring translation validation. The isolated translation check also passes locally.make format-check,make lint,make typecheckpass. The broader local unit suite has existing ClickHouse-local and parser limitations; CI runs the full suite.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated: Codex wrote the fixture setup and verified the reproduction and all existing assertions against a private ClickHouse.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist