From 7ff8477071391109756927b91d9fd5ea8660361f Mon Sep 17 00:00:00 2001 From: Boris Tyshkevich Date: Mon, 26 Oct 2026 16:10:48 +0100 Subject: [PATCH 1/2] docs: restricted-environment install runbook (#692 part 3) New docs/HOSTED-INSTALL.md (preflight decision table + SQL, tested grants, Altinity Cloud / clickhouse-operator / self-managed / HTTPS-only examples, topology and scale-out, handler verification, Option B). README, DEPLOYMENT, ASSET-DISTRIBUTION, deployment.html and the wiki link to it; deployment.html container section corrected (Caddy, not the Python runner). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk --- .wiki/Deployment-and-Security.md | 8 + .wiki/Source-Map.md | 1 + CHANGELOG.md | 7 + README.md | 4 +- docs/ASSET-DISTRIBUTION.md | 6 +- docs/DEPLOYMENT.md | 4 +- docs/HOSTED-INSTALL.md | 319 +++++++++++++++++++++++++++++++ docs/deployment.html | 18 +- 8 files changed, 356 insertions(+), 11 deletions(-) create mode 100644 docs/HOSTED-INSTALL.md diff --git a/.wiki/Deployment-and-Security.md b/.wiki/Deployment-and-Security.md index 02b4e6e6..b55caa25 100644 --- a/.wiki/Deployment-and-Security.md +++ b/.wiki/Deployment-and-Security.md @@ -20,6 +20,14 @@ including a node killed mid-run. The release bundle ships `deploy/` (install.sh, uninstall.sh, lib/) for use with `--artifact`. One installer at a time (no lock); `--ddl-timeout` must be >= 1; the password reaches only `clickhouse-client`. +The operator runbook for restricted environments (#692 part 3) is +[`docs/HOSTED-INSTALL.md`](../docs/HOSTED-INSTALL.md): preflight decision table and SQL, +tested grants as `GRANT` statements, self-managed / HTTPS-only / Altinity Cloud / clickhouse-operator +examples (the last two are **not tested end to end**), topology, scale-out and retry, handler +verification. If `http_handlers` cannot be pushed, ClickHouse-native hosting is impossible: use the Caddy +image or Helm chart below. `--cluster` takes the literal name from `system.clusters` (the `{cluster}` macro +is rejected). + ## Authentication modes - Native/Bearer OAuth for ClickHouse variants with token processors and ephemeral diff --git a/.wiki/Source-Map.md b/.wiki/Source-Map.md index 86343db5..58ab4f93 100644 --- a/.wiki/Source-Map.md +++ b/.wiki/Source-Map.md @@ -42,6 +42,7 @@ Back to [[Home]]. Related: [[Architecture]], [[Product-and-Features]]. - [`docs/ADR-0004-ui-shell.md`](../docs/ADR-0004-ui-shell.md) — the #577 Preact evaluation and its RETAIN-vanilla outcome. - [`docs/V2-UX-HANDOVER.md`](../docs/V2-UX-HANDOVER.md) — shipped-UX + committed-contract inventory for the V2 redesign (#582). - [`docs/DEPLOYMENT.md`](../docs/DEPLOYMENT.md) — deployment sequence. +- [`docs/HOSTED-INSTALL.md`](../docs/HOSTED-INSTALL.md) — restricted-environment install runbook (#692); [`docs/ASSET-DISTRIBUTION.md`](../docs/ASSET-DISTRIBUTION.md) is the mechanism. - [`docs/LOGIN-SCREEN.md`](../docs/LOGIN-SCREEN.md) — login configuration. - [`CHANGELOG.md`](../CHANGELOG.md) — released and unreleased behavior. - [`docs/ui-snapshots/CAPTURE-SPEC.md`](../docs/ui-snapshots/CAPTURE-SPEC.md) — visual baseline. diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bcc3f15..d783d1fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,13 @@ auto-generated per-PR notes; this file is the curated, human-readable history. ## [Unreleased] ### Added +- **#692 (part 3): restricted-environment install runbook, `docs/HOSTED-INSTALL.md`.** Preflight + decision table and copy-paste preflight SQL (grants, clusters, `file()` write/read/truncate probe, + interserver auth, egress), tested grants as `GRANT` statements per mode, end-to-end examples (self-managed, + HTTPS-only, Altinity Cloud, clickhouse-operator; the last two clearly marked not tested end to end), + topology/scale-out/retry procedures, handler enablement and verification, Option B as a described + alternative. README, DEPLOYMENT.md, ASSET-DISTRIBUTION.md, `docs/deployment.html` and the wiki link to it; + `docs/deployment.html` no longer describes the container image as the Python runner (it is Caddy). - **#692 (part 2): `deploy/install.sh --cluster` now works on multi-replica clusters.** SQL-only, two-phase install with server-side fan-out: preflight, pre-clean, phase 1 to `_asb_next_*` (upload once and copy, or `--pull-url`), per-node length diff --git a/README.md b/README.md index 6121d704..80ea3419 100644 --- a/README.md +++ b/README.md @@ -665,6 +665,8 @@ verify), and checks each node's length and SHA-256; re-run it after a scale-out. `deploy/uninstall.sh` empties the files. Cross-node reads need the cluster's `` (or per-replica credentials), run one installer at a time, and the deploy user needs a few grants — see [docs/ASSET-DISTRIBUTION.md](docs/ASSET-DISTRIBUTION.md#installing-with-deployinstallsh). +Restricted environments (Altinity Cloud, clickhouse-operator, no filesystem access, HTTP-only): see the +runbook [docs/HOSTED-INSTALL.md](docs/HOSTED-INSTALL.md). With **no** `--audience`, the IdP returns an **id_token** (its `aud` is the client_id) and the browser sends that as the bearer — so ClickHouse's @@ -763,7 +765,7 @@ deploy/k8s/ sample Deployment, Service, ConfigMap, Ingress example helm/ published Helm chart source tests/ vitest + happy-dom, one spec per module docs/ DEVELOPMENT.md, ARCHITECTURE.md, DEPLOYMENT.md, - ASSET-DISTRIBUTION.md, CLICKHOUSE-OAUTH.md, CLICKHOUSE-OSS-OAUTH.md + ASSET-DISTRIBUTION.md, HOSTED-INSTALL.md, CLICKHOUSE-OAUTH.md, CLICKHOUSE-OSS-OAUTH.md ``` ## Supported browsers diff --git a/docs/ASSET-DISTRIBUTION.md b/docs/ASSET-DISTRIBUTION.md index b6ba41d6..79b5b83a 100644 --- a/docs/ASSET-DISTRIBUTION.md +++ b/docs/ASSET-DISTRIBUTION.md @@ -7,7 +7,9 @@ one; this document is about the native mode.) That makes "how do the asset bytes real design question on a multi-node cluster, because **ClickHouse does not replicate the `user_files/` directory**: it is a node-local folder. -This doc explains the trade-offs and what `deploy/install.sh` ships. +This doc explains the trade-offs and what `deploy/install.sh` ships. The operator procedure for restricted +environments (preflight, tested grants, Altinity Cloud / clickhouse-operator examples, scale-out) is the +runbook [HOSTED-INSTALL.md](HOSTED-INSTALL.md); this page stays the mechanism and design reference. (A `docs/sql.html` mirror is also published to GitHub Pages on every tagged release, for an unauthenticated try-it-now demo — see `release.yml` — but that's a separate distribution channel from the cluster-serving problem @@ -262,7 +264,7 @@ before the run (the preflight rejects it), a node **killed mid-run** right after never an empty or partial file; the queued DDL replay settles; a re-install converges every node on the new one), an unreachable pull URL, missing `REMOTE` / `URL` grants, a single node, and uninstall. Images (exact tags): `clickhouse/clickhouse-server:26.3.18.32` and `altinity/clickhouse-server:25.8.33.20001.altinityantalya`, with `clickhouse/clickhouse-keeper:25.8.33.6` (CI; locally `clickhouse-keeper:25.8` works too). -**Not covered:** the real GitHub release URL, TLS (`--secure`) against a live server, and the +**Not covered:** the real GitHub release URL through the installer (a single-node `url()` download of it was probed by hand, see HOSTED-INSTALL.md), TLS (`--secure`) against a live server, and the per-replica-credentials interserver path. The script logic (strict row checks, cleanup, the password's reach) is additionally unit-tested against a simulator. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 526485d3..ef88ea9f 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -7,7 +7,9 @@ There are now two distinct deployment surfaces: 2. **Containerized runtime** — run the static SPA image in Docker or Kubernetes for local or shared use, with browser requests sent directly to ClickHouse. -This document focuses on the first mode. For source development, Docker, and +This document focuses on the first mode. If you cannot touch the server filesystem (Altinity Cloud, +clickhouse-operator, other managed clusters) or run a multi-node cluster, follow the runbook +[HOSTED-INSTALL.md](HOSTED-INSTALL.md) (preflight, tested grants, examples, scale-out). For source development, Docker, and Kubernetes/Helm instructions, see [DEVELOPMENT.md](DEVELOPMENT.md). The app is three things on the target ClickHouse: diff --git a/docs/HOSTED-INSTALL.md b/docs/HOSTED-INSTALL.md new file mode 100644 index 00000000..0cfa1a96 --- /dev/null +++ b/docs/HOSTED-INSTALL.md @@ -0,0 +1,319 @@ +# Hosted install runbook: restricted and multi-node ClickHouse® + +The operator procedure for serving the Altinity® SQL Browser **from ClickHouse itself** when you can run +SQL but cannot touch the server filesystem (Altinity Cloud, clickhouse-operator, other managed +clusters), on a single node, a replicated shard or a multi-shard cluster. + +The mechanism, its design and the installer flags live in +[ASSET-DISTRIBUTION.md](ASSET-DISTRIBUTION.md) (canonical; this page does not repeat them). The generic +install sequence is [DEPLOYMENT.md](DEPLOYMENT.md). This page answers: *can I do it here, with which +grants, and what exactly do I run?* + +Legend: **[tested]** = exercised by `tests/deploy/` (CI, real ClickHouse, 2 shards x 2 replicas). +**[probed]** = run by hand against a single-node `clickhouse-server:26.3.18.32` while writing this page. +**[untested]** = stated from documentation or design; verify in your environment. + +Tested versions: ClickHouse **26.3.18.32** and Altinity Antalya **25.8.33.20001** (full suite); **25.8 OSS** +only partly (the original #692 experiments). Other versions: expect the same, verify with +`--verify-only` and the checks below. + +## 1. Preflight + +### 1.1 Which procedure? + +Two things must reach every node: the **`http_handlers` config** and the **asset bytes** (see +ASSET-DISTRIBUTION). Answer these in order. + +| # | Question | How to check | If no | +|---|---|---|---| +| 1 | Can you push a `config.d` fragment (`http_handlers`) to every node, including future ones? | Altinity Cloud: ACM cluster setting `config.d/sql-browser.xml`. clickhouse-operator: `spec.configuration.files` (section 3d). Self-managed: `config.d/`. No SQL check exists; try it (section 5). | **ClickHouse-native hosting is impossible.** Use the container image, the Helm chart or any static host: [DEVELOPMENT.md](DEVELOPMENT.md#docker). | +| 2 | Is the `File` engine / `file()` available, and may your user use it? | Probe 1.3(c). | Standalone runtime as above. **ClickHouse Cloud** does not offer the File engine **[untested here]**: use the standalone runtime. | +| 3 | Is the cluster (or single node) defined and visible to your user? | Probe 1.3(b). | Fix the cluster name; single node needs no cluster. | +| 4 | Do cross-node reads authenticate (cluster ``, or per-replica credentials, and the deploy user on every node)? | Probe 1.3(d). Needed for `--cluster` in **both** modes. | `AUTHENTICATION_FAILED (516)`: ask for a `` in `remote_servers` (the clickhouse-operator can generate one, section 3d). Without it use single-node mode against each node you can reach, or the container image. | +| 5 | Can every node reach the artifact URL (and does `remote_url_allow_hosts` permit it)? | Probe 1.3(e). | Use the default **stage** mode (no egress, needs `REMOTE`). | +| 6 | Native protocol reachable from where you run the installer? | `clickhouse-client --host H --secure -q 'SELECT 1'`. | `--transport http` (HTTP(S) port only). | + +| Topology | Procedure | Flags | +|---|---|---| +| Single node | Single-node install. Writes with `file()`; needs `FILE` only (tested user holds the stage set). | no `--cluster` | +| One shard, several replicas | Cluster install; every replica gets its own copy. Behind a load balancer **always** pass `--cluster`, or one random replica is updated. | `--cluster ` | +| Several shards | Same; the installer fans out to every shard and replica and fails closed if any node is missing. | `--cluster ` | +| No node egress | stage mode (default). | | +| Nodes can reach the artifact | `--pull-url` is optional (saves one upload through the connected node). | `--pull-url ` | +| HTTP(S) only | `--transport http [--secure] [--http-port N]` | | + +### 1.2 What the installer needs from you, in one place + +Run as a user with the section 2 grants. **Check that a personal `~/.clickhouse-client/config.xml` is not +overriding host, user or password.** Every statement below is one the installer itself runs, except where +marked. + +### 1.3 Preflight SQL + +Run as the deploy user, on the node or endpoint you will pass as `--ch-host`. Over HTTP(S) send the same text +with `curl -sS --data-binary 'SQL' -H 'X-ClickHouse-User: U' -H 'X-ClickHouse-Key: K' https://H:8443/`. +Keep the password out of shell history (a 0600 header file, as the installer does). + +**(a) Grants** **[probed]** + +```sql +SHOW GRANTS; +``` +Compare with section 2. Grants given through a role show up under the role; use `SHOW GRANTS FOR CURRENT_USER` +**[untested]** when in doubt. + +**(b) Clusters visible to you, node counts** **[probed on a single node; the installer uses the +`count()` form per cluster]** + +```sql +SELECT cluster, count() AS nodes, countDistinct(shard_num) AS shards, max(replica_num) AS max_replica_num +FROM system.clusters GROUP BY cluster ORDER BY cluster; +``` +Pass one of these names to `--cluster`. The name must match `[A-Za-z0-9_.-]+`: a macro such as `{cluster}` is +**rejected** by the installer; use the name this query returns. Run it **on the node you connect to**: another +node can define different clusters. + +**(c) Probe write, read, truncation of a tiny file** **[probed]** (the installer's single-node write uses the same +`INSERT INTO FUNCTION file(...)` form and `engine_file_truncate_on_insert`) + +```sql +INSERT INTO FUNCTION file('_asb_probe', 'RawBLOB') SETTINGS engine_file_truncate_on_insert = 1 SELECT 'probe'; +SELECT c FROM file('_asb_probe', 'RawBLOB', 'c String'); -- expect: probe +INSERT INTO FUNCTION file('_asb_probe', 'RawBLOB') SETTINGS engine_file_truncate_on_insert = 1 SELECT ''; +SELECT count() AS n, sum(length(c)) AS bytes +FROM file('_asb_probe', 'RawBLOB', 'c String') SETTINGS engine_file_empty_if_not_exists = 1; -- expect: 0 0 +``` +`ACCESS_DENIED` means a missing `FILE` grant; any other error means `file()` is disabled or restricted on this +server (condition 2 fails). SQL cannot delete files, so the probe leaves a **0-byte `_asb_probe`** in +`user_files/`; it is never served and harmless. + +**(d) Interserver authentication and node identity** (the installer's own preflight) +**[tested on 2x2 via the installer; probed on a single node]** + +```sql +SELECT toString(serverUUID()) AS id, hostName() AS host +FROM clusterAllReplicas('', view(SELECT 1)) ORDER BY host; +``` +Expect one row per node of (b) with distinct `id`s. `AUTHENTICATION_FAILED (516)` = no cluster `` / +per-replica credentials, or the deploy user is missing on some node. Fewer rows or repeated ids fail the +installer too. + +**(e) Node egress** (only for `--pull-url`; needs the `URL` grant) **[probed]** against the real release URL +from a single node: one 2,126,368-byte download through GitHub's redirect. Not run through the installer +(`tests/deploy/` uses a local 2-hop redirect). Use your mirror URL if you have one. + +```sql +SELECT hostName() AS host, bytes FROM clusterAllReplicas('', view( + SELECT sum(length(c)) AS bytes + FROM url('https://github.com/Altinity/altinity-sql-browser/releases/latest/download/sql.html', 'RawBLOB', 'c String') + SETTINGS max_http_get_redirects = 10, http_connection_timeout = 10, max_execution_time = 60)); +``` +Every node must return the same non-zero `bytes`. (The `clusterAllReplicas` form of this probe was only run on a +single node; the plain `SELECT ... FROM url(...)` inside it is the part exercised.) + +**(f) Existing install state** **[tested: it is the installer's own check]** + +```sql +SELECT id, host, n, len, sha FROM clusterAllReplicas('', view( + SELECT toString(serverUUID()) id, hostName() host, count() n, sum(length(c)) len, lower(hex(SHA256(any(c)))) sha + FROM file('sql.html', 'RawBLOB', 'c String'))) ORDER BY host, id SETTINGS engine_file_empty_if_not_exists = 1; +``` +A node that has no file returns a row with `len = 0`. Compare `len` and `sha` with `wc -c` / `sha256sum` of the +artifact. `./deploy/install.sh ... --verify-only` does this and the preflight without changing anything, and is +the safest end-to-end rehearsal of connectivity and grants. + +## 2. Tested grants + +Create the user on **every** node (or through your replicated access storage / `ON CLUSTER` +**[untested]**), then, per mode. Every set below is exactly what the CI user holds +(`tests/deploy/cfg/users.xml`) on 26.3.18.32 and Antalya 25.8.33. `FILE` and `URL` include the source-level +`READ`/`WRITE ON FILE|URL` privileges on those versions. + +```sql +-- stage mode (default) install and uninstall +GRANT CLUSTER ON *.* TO asb_deploy; +GRANT FILE ON *.* TO asb_deploy; +GRANT REMOTE ON *.* TO asb_deploy; +GRANT CREATE TABLE, DROP TABLE ON default.* TO asb_deploy; +GRANT SELECT ON system.clusters TO asb_deploy; + +-- add for --pull-url +GRANT URL ON *.* TO asb_deploy; + +-- verify-only user (separate account, no install rights) +GRANT FILE ON *.* TO asb_verify; +GRANT REMOTE ON *.* TO asb_verify; +GRANT CREATE TEMPORARY TABLE ON *.* TO asb_verify; +GRANT SELECT ON system.clusters TO asb_verify; +``` + +Notes: sufficiency is tested, minimality only partly (see ASSET-DISTRIBUTION "Requirements"). A single-node +install needs `FILE` in practice, but only the full stage set is tested. The `default` database in `ON +default.*` must exist on every node; if your policy forbids `default`, the temp-table names are fixed to +`default._asb_t_*` **[untested elsewhere]**. Revoke or drop the user when you are done; none of this is +needed to *serve* the app. + +## 3. End-to-end examples + +Common: get the artifact. A release publishes `sql.html`; the release bundle ships `deploy/` from the first +release after #692 (the v0.7.3 bundle does not), so until then use a checkout of this repository at the tag or +`main` and pass the downloaded file as `--artifact` (no `npm`/`node` needed): + +```bash +curl -fsSL -o sql.html https://github.com/Altinity/altinity-sql-browser/releases/latest/download/sql.html +``` + +Set the password in the environment, not on the command line: `export CLICKHOUSE_PASSWORD=…` (or let the +installer prompt). `--client-id`, `--issuer`, `--audience`, `--ch-auth` are the OAuth settings from +[DEPLOYMENT.md](DEPLOYMENT.md#4-make-clickhouse-accept-the-jwt). + +### (a) Self-managed, native protocol **[tested]** + +```bash +./deploy/install.sh --ch-host ch1.example.com --ch-user asb_deploy \ + --client-id --issuer https://idp.example.com \ + --cluster prod --artifact sql.html # stage mode: no node egress needed +# verify only, any time: +./deploy/install.sh --ch-host ch1.example.com --ch-user asb_verify --client-id x --cluster prod \ + --artifact sql.html --verify-only +``` +Then add the rendered `dist/http_handlers.xml` to `config.d/` on every node and reload (section 5). + +### (b) HTTPS only (`--transport http --secure`) **[tested over HTTP; TLS not run against a live server]** + +```bash +./deploy/install.sh --ch-host ch.example.com --ch-user asb_deploy \ + --client-id --cluster prod --artifact sql.html \ + --transport http --secure # port 8443; add --http-port N if different +``` +Same statements, over `curl`; the credentials travel in a 0600 header file. Failures are checked with +`wait_end_of_query=1`: a ClickHouse failure can arrive as HTTP 200 with an `__exception__` trailer, which the +installer treats as an error. Do the same in your own scripts. + +Binary-safe single-node upload by hand (the installer's single-node path does the same through its own `curl`) +**[probed: 300 KB random bytes, SHA-256 identical on disk]**: + +```bash +curl -sS --data-binary @sql.html -H 'X-ClickHouse-User: asb_deploy' -H @pwheader.txt \ + "https://ch.example.com:8443/?wait_end_of_query=1&engine_file_truncate_on_insert=1&query=INSERT%20INTO%20FUNCTION%20file('sql.html','RawBLOB')%20FORMAT%20RawBLOB" +``` +(`pwheader.txt` contains `X-ClickHouse-Key: …`, mode 0600.) This is single node only; for a cluster use the +installer or the manual SQL in the ASSET-DISTRIBUTION appendix. + +### (c) Altinity Cloud **[not tested end to end in this repo]** + +Assumptions: ACM lets you set a cluster setting named `config.d/sql-browser.xml`, and the cluster has an +interserver `` (confirm with probe 1.3(d)); the cluster endpoint is HTTPS only. + +1. Handlers: `./deploy/install.sh --dry-run --client-id --issuer ` renders + `dist/http_handlers.xml`. In ACM add it as the cluster setting `config.d/sql-browser.xml` (about 3.7 KB, well + under ACM's ~8 KB setting-push limit) and push. This is the channel the demo clusters use; a direct edit of + the ConfigMap is reconciled away. +2. Find the cluster name: `SELECT DISTINCT cluster FROM system.clusters;` on the endpoint you connect to. Use + that literal name for `--cluster`; do **not** pass the `{cluster}` macro (rejected). +3. Bytes, through the cluster endpoint (the load balancer picks one node; fan-out then happens server-side, so + no per-pod access is needed): + ```bash + ./deploy/install.sh --ch-host ..altinity.cloud --ch-user asb_deploy \ + --client-id --issuer --cluster --artifact sql.html \ + --transport http --secure + ``` +4. Check every node: the installer prints the per-node table; re-run with `--verify-only` after any scale-out or + node replacement. Redirect URI and checks: section 5. + +If your plan forbids `config.d` pushes or SQL file writes, this does not apply: use the container image or Helm +chart. + +### (d) clickhouse-operator **[not tested end to end in this repo]** + +Handlers go in the ClickHouseInstallation. The key convention was **checked against the operator's own +examples and docs** (`docs/chi-examples/05-settings-05-files-nested.yaml`, +`docs/custom_resource_explained.md`, `docs/chi-examples/21-secure-cluster-secret-01-auto.yaml`): entries of +`spec.configuration.files` are rendered into a ConfigMap and mounted into the pod; a key with the `config.d/` +prefix (or a bare name, which goes to the common config directory) lands in `config.d`. Operator versions +differ, so confirm on yours (the effective file appears under `/etc/clickhouse-server/config.d/` in the pod). + +```yaml +spec: + configuration: + clusters: + - name: prod + secret: + auto: "True" # operator-generated interserver ; or use plaintext / secret-ref + layout: { shardsCount: 2, replicasCount: 2 } + files: + config.d/sql-browser.xml: | + … paste dist/http_handlers.xml contents (without the line) … +``` +Do **not** put `sql.html` here: the ConfigMap cap is 1 MiB and the artifact is about 2 MB. + +`secret.auto` (or a plaintext / secret-ref `secret`) is what lets `clusterAllReplicas` authenticate between +pods; the deploy user must exist on every pod (CHI `users` / `users.d` entry or replicated access storage; +**[untested]**). Then run the installer through the Service / load balancer, stage mode, HTTP(S) or native: + +```bash +./deploy/install.sh --ch-host --ch-user asb_deploy --client-id \ + --cluster prod --artifact sql.html +``` +Stage mode fans out server-side over the interserver connection, so pods need not be individually reachable +and need no egress. Whether a `files` change needs a pod restart depends on the operator and the setting: check with section 5. + +## 4. Topology procedures + +- **Fresh install:** run the install once with `--cluster` (or no `--cluster` on a single node). The run is + two-phase: every node's `_asb_next_*` copy is verified byte-exact before any live file is touched. +- **Upgrade the SPA:** run it again with the new `--artifact`. Each node ends on the old or the new version, never + an empty or partial file **[tested, including a node killed mid-run]**. +- **Scale-out or node replacement:** a new replica's `user_files/` is empty. Re-run the install (it repairs + any node); to find out first, run `--verify-only` or probe 1.3(f): the empty node shows `len = 0`. Also make sure + the new node got the `http_handlers` fragment (your config channel should do it), the `` + and the deploy user **[wipe + repair tested; a truly new pod is the same state]**. +- **A node was down during a run:** the installer fails after `--ddl-timeout` and the queued DDL replays when + the node returns, against whatever source exists then. Bring the node back, **re-run** the installer; do not + wait for the replay. +- **Retry:** every step is idempotent. On any failure fix the cause and re-run the same command. Leftover + `_asb_t_*` tables are pre-cleaned by the next run; a stale `_asb_stage_.html` after a hard kill is only + a never-served copy (how to empty it: ASSET-DISTRIBUTION "Failure and retry"). +- **Concurrency is unsupported:** one installer or uninstaller at a time per cluster (no lock). +- **Uninstall:** `deploy/uninstall.sh` empties the files on every node; remove the `http_handlers` fragment and + reload to really disable the route (a 0-byte file serves 200 with an empty body, a missing one 500). + +## 5. Enable and verify the handlers + +1. Render: `./deploy/install.sh ...` writes `dist/http_handlers.xml` with the CSP `connect-src` filled in from + your issuer's OIDC discovery (`--dry-run` renders without contacting ClickHouse). Deploy **that** file, not + the committed Google default, unless you use Google. +2. Install it through your channel (section 1.1 question 1), on **every** node, and reload. Each rule has a + unique `name=` attribute; keep it. +3. Register the OAuth redirect URI that the installer prints, e.g. `https://:8443/sql`. It must match the + origin exactly, **including the port** unless it is 443/80; use the externally exposed host/port if a load + balancer or ingress fronts ClickHouse. +4. Check, expecting `200`, the right `content-type` and (for `/sql`) the CSP: + ```bash + curl -sI https://:8443/sql | grep -iE '^(HTTP|content-type|content-security-policy|cache-control)' + curl -sI https://:8443/sql/config.json | grep -iE '^(HTTP|content-type)' + curl -s https://:8443/sql/config.json # issuer + client_id only + ``` + `/sql` is `text/html; charset=UTF-8`; `config.json` is `application/json; charset=UTF-8`. A `500` means the + handler exists but the file is missing on that node; a `404` or the ClickHouse default page means the + fragment did not load. Behind a load balancer, repeat until you have hit every node, or rely on the + per-node table from the installer (a single `/sql` smoke test proves one node at most). +5. Sign in at `/sql` and run `SELECT currentUser()`: it must return your identity, not a static user. If + sign-in works but queries are `401`, ClickHouse does not accept the JWT yet: + [CLICKHOUSE-OAUTH.md](CLICKHOUSE-OAUTH.md) (token processor) or + [CLICKHOUSE-OSS-OAUTH.md](CLICKHOUSE-OSS-OAUTH.md) (delegated verifier, `--ch-auth basic`). + +**`sql-config.json` is public.** The browser fetches it before any login; it must hold only `issuer`, +`client_id` and the other non-secret fields. A `client_secret`, if your IdP forces one, is exposed to every +visitor: lock the redirect URI and read [LOGIN-SCREEN.md](LOGIN-SCREEN.md) first. Never put a ClickHouse +password or token in it. + +## 6. Alternative: Option B (Replicated table + `predefined_query_handler`) + +Described, not built or tested. Store the bytes in a `ReplicatedMergeTree` table and serve them with a +`predefined_query_handler` running `SELECT argMax(content, updated) FROM asb_assets WHERE name = 'sql.html' +FORMAT RawBLOB`. Limits: the unauthenticated `/sql` GET runs as whatever `anonymous` resolves to, so that user +needs a read grant on the assets table (the one thing the static approach avoids); replication covers the +replicas of **one shard**, so a multi-shard cluster needs the table on each shard; the Keeper path is +cluster-specific. Its advantage is that replicas added later receive the bytes without a re-run. Details: +[ASSET-DISTRIBUTION.md](ASSET-DISTRIBUTION.md#b-store-the-asset-in-a-replicated-table--predefined_query_handler). diff --git a/docs/deployment.html b/docs/deployment.html index 23e15ad3..f6052a39 100644 --- a/docs/deployment.html +++ b/docs/deployment.html @@ -80,7 +80,8 @@

Serve it from your cluster

--cluster distributes the files to every node with server-side SQL in two phases (stage and verify, then promote and verify) and checks each node's SHA-256. The cluster needs a <secret> and the deploy user a few grants; see - ASSET-DISTRIBUTION.md.

+ ASSET-DISTRIBUTION.md. Altinity Cloud, clickhouse-operator and other restricted environments: + HOSTED-INSTALL.md.

Public file, by design. config.json is served to browsers, so prefer a PKCE public client. If your IdP requires a client_secret, @@ -99,6 +100,7 @@

OAuth in three shapes

CLICKHOUSE-OAUTH.md · OSS (delegated verifier) · ASSET-DISTRIBUTION.md · + HOSTED-INSTALL.md · LOGIN-SCREEN.md

@@ -126,12 +128,14 @@

Run it on your laptop with one command

Mode 3 · Docker local runner
-

Run the existing local app in a container

-

The container image packages the same zero-dependency Python runner used by the curl-installed local app. - It serves /sql and a generated /config.json, then reads saved - connections from a mounted clickhouse-client config directory.

- 
docker compose up --build
-SQL_BROWSER_PROBE=0 docker compose up --build   # skip startup /ping checks, keep all hosts
+

Run the static container image

+

The production image is a static Caddy server for the single-file SPA. It serves + /sql and a config.json you provide at + /sql/config.json; queries go from the browser straight to ClickHouse. This is also + the fallback when a managed provider does not let you install http_handlers (see + DEVELOPMENT.md).

+ 
docker run --rm -p 8900:8080 -v "$PWD/config.json:/config/config.json:ro" \
+  ghcr.io/altinity/altinity-sql-browser:latest   # then open http://localhost:8900/sql

The repo also ships Kubernetes examples under deploy/k8s/: a Deployment, Service, ConfigMap-backed connection file, and an Ingress example for cluster testing.

From 367488baf5df5ae7c2db3a7e3739b64824ae9352 Mon Sep 17 00:00:00 2001 From: Boris Tyshkevich Date: Mon, 26 Oct 2026 16:14:28 +0100 Subject: [PATCH 2/2] docs: address review findings on the restricted-install runbook (#692) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01P3CopPwrpzeWAJd7vaaLSk --- docs/ASSET-DISTRIBUTION.md | 2 +- docs/HOSTED-INSTALL.md | 78 ++++++++++++++++++++++++++------------ 2 files changed, 54 insertions(+), 26 deletions(-) diff --git a/docs/ASSET-DISTRIBUTION.md b/docs/ASSET-DISTRIBUTION.md index 79b5b83a..d0c9627b 100644 --- a/docs/ASSET-DISTRIBUTION.md +++ b/docs/ASSET-DISTRIBUTION.md @@ -86,7 +86,7 @@ serve it with the same `file://` static handler. build), which is too large for several channels: clickhouse-operator `spec.configuration.files` lands in a Kubernetes ConfigMap, which is capped at 1 MiB, and control planes that push settings inline (for example, ACM's - setting-push has an ~8 KB URI limit) cannot carry it either. In practice it + setting-push is reported to have a small, ~8 KB URI limit; **unverified**) cannot carry it either. In practice it has to be a file delivered by some other mechanism (image layer, volume, init container), not an inline config value. diff --git a/docs/HOSTED-INSTALL.md b/docs/HOSTED-INSTALL.md index 0cfa1a96..9216d906 100644 --- a/docs/HOSTED-INSTALL.md +++ b/docs/HOSTED-INSTALL.md @@ -35,7 +35,7 @@ ASSET-DISTRIBUTION). Answer these in order. | Topology | Procedure | Flags | |---|---|---| -| Single node | Single-node install. Writes with `file()`; needs `FILE` only (tested user holds the stage set). | no `--cluster` | +| Single node | Single-node install. Writes with `file()`. Tested with the stage grant set (section 2). It runs `INSERT INTO FUNCTION file(...)`, `SELECT ... FROM file(...)` and `DROP TABLE IF EXISTS default._asb_t_*`, so `FILE` + `DROP TABLE ON default.*` is the likely floor **[not verified]**. | no `--cluster` | | One shard, several replicas | Cluster install; every replica gets its own copy. Behind a load balancer **always** pass `--cluster`, or one random replica is updated. | `--cluster ` | | Several shards | Same; the installer fans out to every shard and replica and fails closed if any node is missing. | `--cluster ` | | No node egress | stage mode (default). | | @@ -50,9 +50,15 @@ marked. ### 1.3 Preflight SQL -Run as the deploy user, on the node or endpoint you will pass as `--ch-host`. Over HTTP(S) send the same text -with `curl -sS --data-binary 'SQL' -H 'X-ClickHouse-User: U' -H 'X-ClickHouse-Key: K' https://H:8443/`. -Keep the password out of shell history (a 0600 header file, as the installer does). +Run as the deploy user, on the node or endpoint you will pass as `--ch-host`. Send **one statement per +request** over HTTP(S); keep the credentials out of argv and shell history in a 0600 header file, as the +installer does: + +```bash +( umask 077; printf 'X-ClickHouse-User: %s\nX-ClickHouse-Key: %s\n' "$U" "$CLICKHOUSE_PASSWORD" > pwheader.txt ) +# put one statement in probe.sql, then: +curl -sS --data-binary @probe.sql -H @pwheader.txt 'https://H:8443/?wait_end_of_query=1' +``` **(a) Grants** **[probed]** @@ -74,7 +80,8 @@ Pass one of these names to `--cluster`. The name must match `[A-Za-z0-9_.-]+`: a node can define different clusters. **(c) Probe write, read, truncation of a tiny file** **[probed]** (the installer's single-node write uses the same -`INSERT INTO FUNCTION file(...)` form and `engine_file_truncate_on_insert`) +`INSERT INTO FUNCTION file(...)` form and `engine_file_truncate_on_insert`). Four statements; over HTTP(S) send +each in its own request. ```sql INSERT INTO FUNCTION file('_asb_probe', 'RawBLOB') SETTINGS engine_file_truncate_on_insert = 1 SELECT 'probe'; @@ -120,14 +127,17 @@ SELECT id, host, n, len, sha FROM clusterAllReplicas('', view( ``` A node that has no file returns a row with `len = 0`. Compare `len` and `sha` with `wc -c` / `sha256sum` of the artifact. `./deploy/install.sh ... --verify-only` does this and the preflight without changing anything, and is -the safest end-to-end rehearsal of connectivity and grants. +the safest end-to-end rehearsal of connectivity and grants. **Repeat the exact OAuth flags used at install +time** (`--client-id`, `--issuer`, `--audience`, `--ch-auth`): the expected `sql-config.json` hash is rendered +from them, so different flags report a false MISMATCH. ## 2. Tested grants Create the user on **every** node (or through your replicated access storage / `ON CLUSTER` -**[untested]**), then, per mode. Every set below is exactly what the CI user holds -(`tests/deploy/cfg/users.xml`) on 26.3.18.32 and Antalya 25.8.33. `FILE` and `URL` include the source-level -`READ`/`WRITE ON FILE|URL` privileges on those versions. +**[untested]**), then, per mode. The canonical statement of what is tested, what is only partly proven and +why is [ASSET-DISTRIBUTION.md, Requirements](ASSET-DISTRIBUTION.md#requirements); the block below is only a +copy-paste form of it (mirrors ASSET-DISTRIBUTION → Requirements; update both). It is what the CI user holds +(`tests/deploy/cfg/users.xml`) on 26.3.18.32 and Antalya 25.8.33. ```sql -- stage mode (default) install and uninstall @@ -148,7 +158,8 @@ GRANT SELECT ON system.clusters TO asb_verify; ``` Notes: sufficiency is tested, minimality only partly (see ASSET-DISTRIBUTION "Requirements"). A single-node -install needs `FILE` in practice, but only the full stage set is tested. The `default` database in `ON +install was tested only with the full stage set; it also runs `DROP TABLE IF EXISTS default._asb_t_*`, so it +needs at least `DROP TABLE ON default.*` besides `FILE` (likely floor, **not verified**). The `default` database in `ON default.*` must exist on every node; if your policy forbids `default`, the temp-table names are fixed to `default._asb_t_*` **[untested elsewhere]**. Revoke or drop the user when you are done; none of this is needed to *serve* the app. @@ -174,9 +185,12 @@ installer prompt). `--client-id`, `--issuer`, `--audience`, `--ch-auth` are the --client-id --issuer https://idp.example.com \ --cluster prod --artifact sql.html # stage mode: no node egress needed # verify only, any time: -./deploy/install.sh --ch-host ch1.example.com --ch-user asb_verify --client-id x --cluster prod \ +./deploy/install.sh --ch-host ch1.example.com --ch-user asb_verify --cluster prod \ --artifact sql.html --verify-only ``` +`` means exactly the `--client-id`, `--issuer`, `--audience`, `--ch-auth` values used at +install time: `sql-config.json`'s expected hash is rendered from them, so any difference is reported as a +mismatch. Then add the rendered `dist/http_handlers.xml` to `config.d/` on every node and reload (section 5). ### (b) HTTPS only (`--transport http --secure`) **[tested over HTTP; TLS not run against a live server]** @@ -205,21 +219,25 @@ installer or the manual SQL in the ASSET-DISTRIBUTION appendix. Assumptions: ACM lets you set a cluster setting named `config.d/sql-browser.xml`, and the cluster has an interserver `` (confirm with probe 1.3(d)); the cluster endpoint is HTTPS only. -1. Handlers: `./deploy/install.sh --dry-run --client-id --issuer ` renders - `dist/http_handlers.xml`. In ACM add it as the cluster setting `config.d/sql-browser.xml` (about 3.7 KB, well - under ACM's ~8 KB setting-push limit) and push. This is the channel the demo clusters use; a direct edit of - the ConfigMap is reconciled away. +1. Handlers: `./deploy/install.sh --dry-run --client-id --issuer --artifact sql.html` renders + `dist/http_handlers.xml` (from a checkout with a build, `--artifact` may be omitted). In ACM add it as the + cluster setting `config.d/sql-browser.xml` and push. Whether ACM limits the size of a pushed setting, and + that a direct ConfigMap edit is reconciled away, are **[unverified]**: check on your plan. 2. Find the cluster name: `SELECT DISTINCT cluster FROM system.clusters;` on the endpoint you connect to. Use that literal name for `--cluster`; do **not** pass the `{cluster}` macro (rejected). 3. Bytes, through the cluster endpoint (the load balancer picks one node; fan-out then happens server-side, so - no per-pod access is needed): + no per-pod access is needed). Behind a load balancer or Service the installer's statements may land on + different pods, so a ~2 MB `_asb_stage_*` copy can remain on a pod that was not the last one used; it is + never served, and can be emptied as described in ASSET-DISTRIBUTION + [Failure and retry](ASSET-DISTRIBUTION.md#failure-and-retry): ```bash ./deploy/install.sh --ch-host ..altinity.cloud --ch-user asb_deploy \ --client-id --issuer --cluster --artifact sql.html \ --transport http --secure ``` -4. Check every node: the installer prints the per-node table; re-run with `--verify-only` after any scale-out or - node replacement. Redirect URI and checks: section 5. +4. Check every node: the installer prints the per-node table; re-run with `--verify-only` (plus the same + OAuth flags as the install, see 1.3(f)) after any scale-out or node replacement. Redirect URI and checks: + section 5. If your plan forbids `config.d` pushes or SQL file writes, this does not apply: use the container image or Helm chart. @@ -243,8 +261,14 @@ spec: layout: { shardsCount: 2, replicasCount: 2 } files: config.d/sql-browser.xml: | - … paste dist/http_handlers.xml contents (without the line) … + + + … the elements from dist/http_handlers.xml … + + ``` +That is, paste the **whole rendered file** minus its `` line, indented under the key, with **no extra +wrapper**: the file already has the single `` root shown above. Do **not** put `sql.html` here: the ConfigMap cap is 1 MiB and the artifact is about 2 MB. `secret.auto` (or a plaintext / secret-ref `secret`) is what lets `clusterAllReplicas` authenticate between @@ -256,7 +280,9 @@ pods; the deploy user must exist on every pod (CHI `users` / `users.d` entry or --cluster prod --artifact sql.html ``` Stage mode fans out server-side over the interserver connection, so pods need not be individually reachable -and need no egress. Whether a `files` change needs a pod restart depends on the operator and the setting: check with section 5. +and need no egress. Behind a Service or load balancer the installer's statements may land on different pods, +so a ~2 MB `_asb_stage_*` copy can remain on a pod; it is never served (how to empty it: ASSET-DISTRIBUTION +[Failure and retry](ASSET-DISTRIBUTION.md#failure-and-retry)). Whether a `files` change needs a pod restart depends on the operator and the setting: check with section 5. ## 4. Topology procedures @@ -265,7 +291,7 @@ and need no egress. Whether a `files` change needs a pod restart depends on the - **Upgrade the SPA:** run it again with the new `--artifact`. Each node ends on the old or the new version, never an empty or partial file **[tested, including a node killed mid-run]**. - **Scale-out or node replacement:** a new replica's `user_files/` is empty. Re-run the install (it repairs - any node); to find out first, run `--verify-only` or probe 1.3(f): the empty node shows `len = 0`. Also make sure + any node); to find out first, run `--verify-only` (with the same OAuth flags as the install) or probe 1.3(f): the empty node shows `len = 0`. Also make sure the new node got the `http_handlers` fragment (your config channel should do it), the `` and the deploy user **[wipe + repair tested; a truly new pod is the same state]**. - **A node was down during a run:** the installer fails after `--ddl-timeout` and the queued DDL replays when @@ -281,7 +307,8 @@ and need no egress. Whether a `files` change needs a pod restart depends on the ## 5. Enable and verify the handlers 1. Render: `./deploy/install.sh ...` writes `dist/http_handlers.xml` with the CSP `connect-src` filled in from - your issuer's OIDC discovery (`--dry-run` renders without contacting ClickHouse). Deploy **that** file, not + your issuer's OIDC discovery (`--dry-run --artifact sql.html` renders without contacting ClickHouse; from a checkout with a build, + `--artifact` may be omitted). Deploy **that** file, not the committed Google default, unless you use Google. 2. Install it through your channel (section 1.1 question 1), on **every** node, and reload. Each rule has a unique `name=` attribute; keep it. @@ -290,11 +317,12 @@ and need no egress. Whether a `files` change needs a pod restart depends on the balancer or ingress fronts ClickHouse. 4. Check, expecting `200`, the right `content-type` and (for `/sql`) the CSP: ```bash - curl -sI https://:8443/sql | grep -iE '^(HTTP|content-type|content-security-policy|cache-control)' - curl -sI https://:8443/sql/config.json | grep -iE '^(HTTP|content-type)' + curl -s -D - -o /dev/null https://:8443/sql | grep -iE '^(HTTP|content-type|content-security-policy|cache-control)' + curl -s -D - -o /dev/null https://:8443/sql/config.json | grep -iE '^(HTTP|content-type)' curl -s https://:8443/sql/config.json # issuer + client_id only ``` - `/sql` is `text/html; charset=UTF-8`; `config.json` is `application/json; charset=UTF-8`. A `500` means the + Use a GET-based header dump as shown, not `curl -I`: the rules are `GET`, and a `HEAD` + request to `/sql` returned `404` on 26.3.18.32 **[probed]**, which looks like a missing fragment. `/sql` is `text/html; charset=UTF-8`; `config.json` is `application/json; charset=UTF-8`. A `500` means the handler exists but the file is missing on that node; a `404` or the ClickHouse default page means the fragment did not load. Behind a load balancer, repeat until you have hit every node, or rely on the per-node table from the installer (a single `/sql` smoke test proves one node at most).