Repository navigation
deploy: SQL-only two-phase install with server-side fan-out and per-node verification (#692 part 2) - #694
Merged
Conversation
…#692) install.sh now distributes sql.html + sql-config.json to every node of a cluster via ON CLUSTER CTAS into File(RawBLOB) tables: preflight, pre-clean, phase 1 to _asb_next_* (stage upload or --pull-url), per-node length+sha256 verify, phase 2 local promote, verify live, always-cleanup trap. Live files are only ever written from a node-local, verified copy. New flags: --artifact, --pull-url, --transport native|http, --http-port, --ddl-timeout, --verify-only. HTTP transport keeps credentials in a 0600 curl header file, never argv. uninstall.sh empties both files on every node and verifies. Shared plumbing in deploy/lib/asb-ch.sh. Unit tests run the real scripts against simulated clickhouse-client/curl shims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk
…pull, docs, CI (#692 part 2) - tests/deploy: Keeper + 2x2 ClickHouse cluster with <secret>, minimal-grant deploy users, real http_handlers.xml, redirecting file server; scenarios assert byte-exact /sql and /sql/config.json from every node (npm run test:deploy) - install/uninstall: single-node mode writes with INSERT INTO FUNCTION file (a CTAS demanded INSERT on the temp table); url() pull bounded by max_execution_time and HTTP connect/receive timeouts from --ddl-timeout; create dist/ when run from the release bundle layout - CI: path-filtered deploy-integration job (26.3 + Antalya 25.8.33) in the gate; release bundle ships deploy/ - docs: ASSET-DISTRIBUTION installer procedure, grants, limits, manual SQL appendix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk
Installer: --ddl-timeout >= 1 and distributed_ddl_output_mode=throw pinned on both transports; pre-clean and uninstall empty the _asb_next_* files; strict verification rows (row count, distinct hosts, n==1; n==0 only for uninstall's 0-byte check); zero-byte artifact refused; cleanup ignores INT/TERM, caps its DDL timeout at 30 s and always removes the credential header file; password no longer exported (clickhouse-client only); curl -q; clear error without a build tree and without --artifact; concurrent runs documented as unsupported. Tests: simulator knobs (dropped/extra/duplicate-host/n=0 rows), mutation-checked row/host/n/stage-MATCH guards, secrets/env/argv, cleanup robustness, sh shims. Integration: node killed mid-run, deploy_verify user (needs CREATE TEMPORARY TABLE), all-node temp-table/stage checks with positive controls, exact image tags, container logs on any failure. CI bundle step runs the shipped installer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk
- shellcheck SC2015 fixes (if/then in tests/deploy/run.sh and the rc capture)
- node identity is hostName():tcpPort() in verify/preflight/stage checks, so
several instances per host pass; duplicate identities fail closed
- CREATE TEMPORARY TABLE grant measured: required only by a verify-only user
(ACCESS_DENIED without it on 26.3 and Antalya 25.8); install/uninstall users
pass without it. users.xml, docs grant table and failure hint reconciled
- unit shims dump the whole environment; password value asserted absent
- cleanup is bounded: client-side curl/native timeouts; exact http cleanup
timeout assertions; behavioral SIGINT-during-cleanup test
- validate_pos: ^[1-9][0-9]{0,5}$ (no octal/overflow)
- build-tree guard checks the build entry point and package.json
- scenario (m) asserts outcomes (survivors serve B or C; re-install -> C);
docs: orphaned _asb_stage_* after a killed run and how to empty it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk
tcpPort() can fail on servers without a plain tcp_port (TLS-only). Verify and preflight now key on toString(serverUUID()), unique per server instance and independent of ports; hostName() stays as a displayed column. Sim, unit tests and docs updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk
…io (#692) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk
- docs: verified orphaned-stage listing (file(..., 'One') works on both images, lists 0-byte files); one file per killed run
- native cleanup calls wrapped in coreutils timeout when present (total bound), idle timeouts otherwise; docs reworded
- ASB_TEST_BARRIER_TIMEOUT regex ^[1-9][0-9]{0,3}$
- barrier hook: unset from child environments, loud stderr WARNING when active
- tests/deploy (m)/(m3): assert failure reason; print survivors promoted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk
This was referenced Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 2 of #692:
deploy/install.sh(anduninstall.sh) now distributesql.htmlandsql-config.jsonto every node of a multi-replica cluster using SQL only (no ssh, no per-node upload), with per-node verification. This PR contains the installer (commit 134cb9f, part A) plus the integration suite, fixes found by it, CI, release bundle, and docs (part B).Design
Two-phase so the live file is never the target of anything that can fail on a remote source. A failed
CREATE TABLE ... AS SELECTtruncates its target, and cluster-function INSERTs are rejected (Code 55), so fan-out isCREATE ... ON CLUSTER ... ENGINE=File('RawBLOB', ...) AS SELECT:DROP IF EXISTS ... ON CLUSTER ... SYNC_asb_next_*(stage: upload once to the connected node and nodes copy it; or--pull-url), verify length + SHA-256 on every nodeSpike evidence: #692 (comment) and the later design comment on #692.
Flags
--cluster,--artifact,--pull-url,--transport native|http(password never in argv),--http-port,--ddl-timeout(>= 1;distributed_ddl_output_mode=throwpinned; also bounds a--pull-urldownload viamax_execution_time+ HTTP timeouts),--verify-only;uninstall.shempties both files on every node.Tested matrix
npm run test:deploy(CI jobdeploy-integration, path-filtered, in the CI gate) onclickhouse/clickhouse-server:26.3.18.32(exact tag; keeper25.8.33.6) andaltinity/clickhouse-server:25.8.33.20001.altinityantalya; Keeper + 2 shards x 2 replicas with a cluster<secret>. Per-node byte-exact SHA of/sqland/sql/config.jsonfor: fresh install (native+stage), upgrade, http transport,--pull-urlvia a 2-hop redirect, wiped-node repair,--verify-onlyon a tampered node, a node stopped before the run (preflight rejects it), a node killed mid-run after the stage upload (installer fails, the other nodes keep the previous good version, DDL replay settles, re-install succeeds),--verify-onlyas a minimaldeploy_verifyuser, unreachable pull URL, missingREMOTE/URLgrants, single node, uninstall (0 bytes). All pass on both images;npm test,tsc --noEmit,npm run pretestand shellcheck are green.Hardening after review
Pre-clean and uninstall empty
_asb_next_*; verification requires one row per node, distinct hosts == nodes,n == 1; zero-byte artifact refused; cleanup ignores INT/TERM, caps DDL at 30 s, always removes the credential file; password reaches onlyclickhouse-client(not exported);curl -q; concurrent runs unsupported (documented, no lock).Grants (tested)
stage:
FILE,CLUSTER,REMOTEon*.*,CREATE TABLE, DROP TABLE ON default.*,SELECT ON system.clusters;--pull-urladdsURL;--verify-onlyneedsFILE,REMOTE,CREATE TEMPORARY TABLE,SELECT ON system.clusters. Tested sufficient (negative-tested:REMOTE,URL, install-as-verify-user only). Cluster needs<secret>for cross-node reads (per-replica credentials were spike-verified, not in CI) and the deploy user on every node.Fixed along the way
INSERTon the temp table; it now usesINSERT INTO FUNCTION file(FILE grant only).install.sh --artifactfrom the release bundle layout failed whendist/did not exist; the release bundle now shipsdeploy/(install.sh, uninstall.sh, lib/, http_handlers.xml).Known limits
_asb_next_*is intentionally kept as a full copy (~2 MB/node).--ddl-timeout; its queued DDL replays later against whatever source exists then, so re-run after bringing it back.--secure), a node dying between preflight and a DDL statement (simulator only). A personal~/.clickhouse-client/config.xmlcan override credentials (documented).Refs #692
🤖 Generated with Claude Code
https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk