Repository navigation
Conversation
… inside --wait, and doctor warns on the role it runs as The mode is taken from the first state read, set to unknown only once the router accepted attach/enforced, and set from what the durable read saw; an interrupt prints the mode line and goes on. The trigger DDL's lock bound is taken after the migration lock, from what --wait has left then. An ended mode transaction whose lock does not clear refuses, naming its backend. A busy table's holders come back with ensure_table's answer, and attach says them once per change rather than a boot's warning a try. Attach counts the role it runs as among its own; doctor does not. The dead SCIM recheck and the unused scim_ok parameter are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish cells.md and the runbook state that attach exempts the role it runs as and doctor does not. results.md gives the marked batch's per-round changes and their mean, says the delivery-pass gate is met on the mean but not established by three rounds alone, and that the memory.set p99 gate is unresolved locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS before --wait is judged spent, so --wait 0 is one try of each wait with no retry, and an idle cell attaches with it. The run's mode turns unknown just before a try sends its UPDATE and back to recording when that try rolls back. A trigger try that finds the migration lock busy names that lock's holder, in the retry line and in the refusal. The step-4 watch task is awaited after its cancel. The ATTACH_LOCK_KEY comment says who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them. Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup. Give gate batch scopes a unique prefix per run so cleanup preserves existing rows. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ancy Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise primary-visible commits, durable duplicate retries and cancellation cleanup on an isolated physical standby pair in both supported Postgres CI jobs. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…are takes the same lock The attach command holds the attach lock on a direct, marked session of the cell's Postgres for its run and resumes by the directory row and the cell's mode. It refuses before any write on the stores' identity, the session, the roles, settings and subscriptions that would let an unchecked writer through, the database encoding, durability, its own grants, the ClickHouse writers, the table list and the processes' heartbeats; posts attach/begin; creates the triggers under the migration lock; in recording scans and writes the tenancy rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable, waits out every transaction that could have read recording, rescans, places every org in batches and marks the cell serving. A refusal after the mode commit leaves the cell enforced and attaching, and a rerun resumes there. Prepare refuses a session behind a pooler or a database that is not UTF8, and takes the attach lock. Doctor runs attach's Postgres reads again and reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows inside the widened window, and PRUNE_AFTER_S names the sweep's hour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end docs/operations/cell-attach.md is the operator's procedure: the preconditions attach checks and those it relies on, the flags, the point of no return at the mode commit, the orgs attach does not place and the manual removal of an org past the length limit, and doctor's reads after serving. cells.md describes the first attach as it runs (the lock, the refusals, the trigger pass, the mode commit and its durable read, step 4's wait, rescan and final checks), with its stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py measures memory.set and a delivery-sized insert through an attached cell with the org triggers on and off; results.md records a local run. Attach compares the final root-secret fingerprint with the one its begin wrote, and the nonce cleanup's message names no single command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module instances.read_recent takes the window it reads back; attach's heartbeat module computes it from the live window and a stopping process's drain and retirement budgets. PRUNE_AFTER_S is marked as the contract it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown The refusals name every table the trigger pass would alter whose owner role attach's role is not a member of, before any write; ensure_table and _run_ddl answer applied, busy or denied, so a denial during the pass refuses at once rather than after --wait as a busy lock. --wait is wall time: each bounded wait charges its elapsed time, lock tries included, and every lock try waits at most what is left; scan time stays outside it. From the mode commit on, a stop that comes before the durable read says the commit's outcome is unknown and to rerun attach, for a refusal and for any other error alike. The prune-window check runs with the refusals, before any write; an enforced resume runs one trigger pass; the lock session opens inside the block that closes it; the durable read retries a timeout or an outage only; refusals name no command that does not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure --stray-orgs labels only a spent budget as tables left unchecked; any other refusal names its table and the read continues. A failed catalog read reports both of attach's Postgres checks as not read. The runbook and doctor describe removing a stray or over-long org by hand, and name no command that does not exist yet. cells.md labels the first attach's list with the design's step names, so step 1 to step 4 mean the same everywhere, rewrites the stuck-process failure row and the planned tests for a mode that only rises, and says --wait is wall time. The trigger-cost harness uses cell_triggers.create_statement and deletes the rows its inserts wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as The mode is taken from the first state read, set to unknown only once the router accepted attach/enforced, and set from what the durable read saw; an interrupt prints the mode line and goes on. The trigger DDL's lock bound is taken after the migration lock, from what --wait has left then. An ended mode transaction whose lock does not clear refuses, naming its backend. A busy table's holders come back with ensure_table's answer, and attach says them once per change rather than a boot's warning a try. Attach counts the role it runs as among its own; doctor does not. The dead SCIM recheck and the unused scim_ok parameter are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish cells.md and the runbook state that attach exempts the role it runs as and doctor does not. results.md gives the marked batch's per-round changes and their mean, says the delivery-pass gate is met on the mean but not established by three rounds alone, and that the memory.set p99 gate is unresolved locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS before --wait is judged spent, so --wait 0 is one try of each wait with no retry, and an idle cell attaches with it. The run's mode turns unknown just before a try sends its UPDATE and back to recording when that try rolls back. A trigger try that finds the migration lock busy names that lock's holder, in the retry line and in the refusal. The step-4 watch task is awaited after its cancel. The ATTACH_LOCK_KEY comment says who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them. Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup. Give gate batch scopes a unique prefix per run so cleanup preserves existing rows. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Count privileges a login can reach through SET ROLE or an ADMIN regrant, including noninherited group grants. Keep the owner exemptions and explicit role allow list. Repeat the writer, trigger, Postgres catalog and standby checks after the rescan and placement before marking the cell serving. Retain the heartbeat watch and its final wait floor. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check USAGE of pg_read_all_stats and pg_signal_backend because attach uses their privileges without SET ROLE. Keep MEMBER for the separate untrusted-writer reachability checks. Refuse noninherited capabilities before attach writes, and explain the inherited grant requirement in the refusal labels and runbook. Test both inherited neighbors and signaling a scratch backend. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the blank-person refusal inside enforced provisioning and redact tenancy refusal details. Match test I/O fakes and public refusal expectations to the shared authentication path, and regenerate the schema statement corpus. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Read the diagnostic owner's four-value result and assert the prepared first-cell identity. Preserve the marker and trigger checks. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the provisioning foundation while retaining update and placement control. Check the intended target before claims and reservation release, and remove an unused local tenancy observation. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the provisioning and configuration control dependencies while preserving the stopped-cell recovery protocol and its regression cases. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Bind restore and control owners before deployment and codec initialization can wait. Preserve public close on late connections and initialization failures without changing ordinary dedicated-session cleanup. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Run the shared first-cell write policy on the bound backend under the original command deadline so cancellation closes that session before cancelling its read. Keep paired-store checks and native backend-release coverage. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Bind cleanup ownership before deployment and codec checks can wait, so cancellation closes the session within the control attempt deadline. Placement uses the same acquisition path once. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the connector's exact participant tag when waiting for synchronous replication, so the drill reaches cancellation and standby promotion checks. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep stopped-cell metadata repair and its bounded owner protocol on the shipped directory cleanup and configuration refusal paths. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the in-process transport on the integration target contract so the explicit-target gate accepts the private standby drill. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the published placement deadline and marked-rollup fixture corrections with their assertions and workload intact. Print each native integration node so a cancelled run retains failing and stalled test identities. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain bounded phase and failure locations outside pytest capture so unfinished native recovery checks expose their last phase without changing their operations or limits. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain the private event loop and owner clock fixes from the merged deadline fixture. Keep restore cleanup arguments in the required formatter layout without changing the test behavior. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The identity recheck expects acknowledged cleanup without testing elapsed time. Give that test its own clock while keeping the shared fixture and deadline tests unchanged. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
2 of 7 tasks
A table alias cloned the fixed KeeperMap path and still exposed server heartbeats. Use an empty Memory table for the stopped-cell fixture while retaining the production liveness checks. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Attempt nonce cleanup without replacing an existing failure or cancellation. A cleanup fatal remains visible after a successful probe. Cover both stores with first-outcome identity checks and success-path controls. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retire the prior test pool before switching the private database and verify both connection paths use that database. Preserve the restore and erasure assertions. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the private primary's published loopback port and preserve the standby drill when Postgres jobs use separate profiles. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #735
Related: #776
Changelog category
Changelog entry
loom cell restorerepairs a stopped cell's tenancy, mode and configuration metadata from the surviving router directory before reconciliation and startup.Why
A restored backup can lose tenancy rows while the router still records their permanent placements. Reconciliation then refuses those orgs. The command restores that metadata under the cell's operator lock and a fresh directory generation claim, without deriving ownership from content, creating placements or deleting content.
Scope and risk
The command verifies paired-store identity and configuration, copies bounded directory pages, repairs cell rows and verifies triggers. It retains one owned Postgres session and rechecks the directory generation before success. The router endpoint authenticates the operator before bounded body parsing and directory access.
Connection acquisition, probes, repair and cleanup share one deadline. Interrupted commits and uncertain closure require repeating the command while the cell remains stopped. Native fatal exceptions survive cleanup. The fixture corrections preserve the original assertions and record setup, call and teardown outcomes without exception messages, SQL, credentials or row contents.
The command repairs metadata only. Its result can still require prepare, configuration recovery or attach resumption. Ordinary reconciliation remains required before serving. Missing erasure evidence, a lost directory and general identity recovery remain outside this command's guarantees. Activity checks detect observed live processes; heartbeat expiry does not prove every writer stopped.
The restore standby drill runs in the storage profile for both supported Postgres versions. Its DSN uses the loopback port Docker assigned to its own private primary, with the existing timeout, memory caps, test invocation and cleanup retained. The workflow composition preserves the separate authenticated-isolation profile.
Compatibility, rollout and rollback
loom cell restore --cell NAME [--router URL] [--wait SECONDS]and operator-authenticatedPOST /cells/v1/operator/cells/{cell}/restore-read. No public Operation, SDK field, setting or default changes.Tests
6267definished with 53 successful checks, two expected publication skips and one failed Cell check. Both Postgres native jobs passed. Cell provenance refused and the outer result was INVALID; provisional measurements do not establish an accepted verdict. That failure and its unknown cause remain recorded.AI assistance in the code
Checklist