diff --git a/.wiki/Deployment-and-Security.md b/.wiki/Deployment-and-Security.md index 7bcfe383..08c4524c 100644 --- a/.wiki/Deployment-and-Security.md +++ b/.wiki/Deployment-and-Security.md @@ -37,17 +37,18 @@ Re-check replica count (`kubectl get pods -n demo -l clickhouse.altinity.com/chi before every otel deploy — a ClickHouse Operator `chi` can add replicas independent of any SPA-deploy action. See [[Operations-Memory]]. -## Standalone nginx image + Helm chart (sql.demo.altinity.cloud) +## Standalone Caddy image + Helm chart (sql.demo.altinity.cloud) Distinct from the three ClickHouse-hosted deploys above: the SPA also ships as a -production **nginx** container image (`ghcr.io/altinity/altinity-sql-browser`, +production **Caddy** container image (`ghcr.io/altinity/altinity-sql-browser`, public, multi-arch) built by `.github/workflows/docker.yml` (`edge`+`sha-` on every push to `main`; `X.Y.Z`+`latest` on a `vX.Y.Z` tag push — the release image) and a **Helm chart** (`helm/altinity-sql-browser/`, published to `oci://ghcr.io/altinity/altinity-sql-browser/helm/altinity-sql-browser` by `release.yml` on `v*`). -The container is a standalone static server (nginx-unprivileged, uid 101, port -8080); it never proxies — the browser POSTs queries cross-origin to whatever +The container is a standalone static server (`caddy:2.8.4-alpine`, uid 101, port +8080, `deploy/caddy/Caddyfile`; serves precompressed br/zstd/gzip sidecars, no +request-time compression; CSP `connect-src` from the `CONNECT_SRC` env var); it never proxies — the browser POSTs queries cross-origin to whatever cluster the login picker selects. Live at **https://sql.demo.altinity.cloud/sql** (ns `demo`, Helm release diff --git a/CHANGELOG.md b/CHANGELOG.md index 4581ee83..ff8b59db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,35 @@ auto-generated per-PR notes; this file is the curated, human-readable history. ## [Unreleased] +### Fixed +- **#692 (part 1): deployment docs and installer output corrections.** + `deploy/install.sh` printed `https://` even without `--secure` and omitted the + HTTP port; the next-steps text now comes from a `print_next_steps` function + (also shown by `--dry-run`, covered by `tests/unit/install-script.test.js`) and + prints the right scheme and the default HTTP(S) port (8123/8443). Docs: removed + frozen artifact-size figures (`dist/sql.html` is about 2 MB raw; measure your + build), stated the real installer prerequisites (npm, node 22+, curl, + clickhouse-client; prebuilt release asset noted), clarified that + ClickHouse-native hosting needs `http_handlers` access and that the + container/Helm runtime is the alternative, corrected the option B/C caveats + (replication scope, ConfigMap 1 MiB cap), corrected the auth wording + (Bearer id_token by default, access_token with an audience, JWT as Basic + password with `ch_auth: "basic"`), and updated the wiki from nginx to Caddy. + Docs also no longer claim `install.sh --cluster` fans out to every replica: + testing (ClickHouse 25.8 OSS, 26.3 OSS, Antalya 25.8.33) showed it fails on any + multi-replica cluster (`INSERT INTO FUNCTION clusterAllReplicas` -> + `STORAGE_REQUIRES_PARAMETER`) and, since the script exits (`set -e`) before its + `DROP`, may leave a temp `_asb_*` table behind (cleanup `DROP ... ON CLUSTER` + statements are documented). `install.sh --cluster` now fails closed (exit 2, + before building) unless the `system.clusters` count query succeeds and returns + exactly 1: query failure, an unknown cluster (0), malformed output and more + than one replica each get a distinct message (the last points at the manual + procedure). Added a tested interim manual `ON CLUSTER ... ENGINE = File ... AS + SELECT FROM url()` procedure (including the `sql-config.json` literal, tested on single-node 26.3 + only) to `docs/ASSET-DISTRIBUTION.md`, structured as an idempotent + pre-clean/create/drop/verify sequence for retrying after a partial failure + (the retry flow itself is untested); the installer fix is tracked in #692. + ### Added - **#592: lock in the #586/#587 shell primitive guardrails mechanically.** `check:arch` now rejects three regrowth shapes the six-copy-pasted-overlays diff --git a/README.md b/README.md index b974c095..6ddba27f 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ variable across every tile. ## How it works -![Auth & data flow: the browser fetches the single-file SPA and its config.json from ClickHouse, signs in to your OAuth IdP with OAuth2 Authorization-Code + PKCE (id_token kept in sessionStorage), then POSTs every query to ClickHouse with an Authorization: Bearer id_token that ClickHouse validates against the IdP's JWKS via its token_processor (or a delegated verifier). There is no app-specific backend.](docs/assets/img/how-it-works.svg) +![Auth & data flow: the browser fetches the single-file SPA and its config.json from ClickHouse, signs in to your OAuth IdP with OAuth2 Authorization-Code + PKCE (id_token kept in sessionStorage), then POSTs every query to ClickHouse with an Authorization: Bearer id_token (or access_token with an audience; the JWT as the Basic password with ch_auth basic) that ClickHouse validates against the IdP's JWKS via its token_processor (or a delegated verifier). There is no app-specific backend.](docs/assets/img/how-it-works.svg) The browser never holds a static credential — each user authenticates with your IdP and ClickHouse sees their JWT. There is **no app-specific backend**: the @@ -653,9 +653,13 @@ CLICKHOUSE_PASSWORD=… ./deploy/install.sh \ [--issuer https://accounts.google.com] \ [--audience ] \ # audience-gated CH → also sends the access_token [--ch-auth basic] \ # OSS CH + ch-jwt-verify → JWT as Basic password - [--cluster ] # single-shard multi-replica only (else per-node) + [--cluster ] # BROKEN on multi-replica clusters (#692); omit ``` +`--cluster` currently fails on multi-replica clusters (tracked in #692); for a +single node run the installer without it, and for multi-node clusters follow +[the interim manual procedure](docs/ASSET-DISTRIBUTION.md#multi-node-clusters-interim-manual). + With **no** `--audience`, the IdP returns an **id_token** (its `aud` is the client_id) and the browser sends that as the bearer — so ClickHouse's `expected_audience` must be the **client_id**, not an API audience. Passing @@ -670,7 +674,7 @@ see "Security headers" below), and uploads the SPA + config into ClickHouse it as an ACM cluster setting `config.d/sql-browser.xml`) and reload ClickHouse. The SPA handler serves `/sql`; Workbench and Dashboard bookmarks differ only by the `ws`, `surface`, and optional `mode` query parameters. -2. Register the single redirect URI `https:///sql` with your OAuth IdP. +2. Register the single redirect URI `https://:8443/sql` with your OAuth IdP (the URI must match the origin exactly: include the HTTP(S) port unless it is the scheme default, 443/80, or use the port/ingress host actually exposed). 3. Make sure ClickHouse accepts the bearer JWT — either a CH `` entry validating your IdP's JWKS, or a delegated `` verifier. See [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). diff --git a/deploy/http_handlers.xml b/deploy/http_handlers.xml index 2791dfa7..861515b7 100644 --- a/deploy/http_handlers.xml +++ b/deploy/http_handlers.xml @@ -8,7 +8,9 @@ GET /sql -> the SPA (dist/sql.html in user_files) GET /sql/config.json -> the OAuth config (issuer + client_id [+ audience]) - The SPA POSTs queries to "/" with `Authorization: Bearer `; that is + The SPA POSTs queries to "/" with the IdP JWT in `Authorization: Bearer ` + (id_token by default, access_token when an audience is set; with + `ch_auth: "basic"` the JWT is the Basic password instead); that is handled by ClickHouse's built-in dynamic query handler (the rule), so no extra route is needed for query execution. ClickHouse must be configured to accept the JWT (token_processor / JWKS, or a delegated http_authentication diff --git a/deploy/install.sh b/deploy/install.sh index 12a6d5dd..19782a9b 100755 --- a/deploy/install.sh +++ b/deploy/install.sh @@ -9,6 +9,12 @@ # The password is read from the CLICKHOUSE_PASSWORD env var or prompted — never # passed on the command line (it would leak via `ps`/shell history). # +# Prerequisites: bash, node 22+ and npm (the installer builds the SPA from source), +# curl (only for OIDC discovery; it falls back to the Google connect-src without +# it), and clickhouse-client (not needed for --dry-run). To skip the build, take +# the prebuilt sql.html from the GitHub release and upload it yourself; see +# docs/ASSET-DISTRIBUTION.md. +# # Usage: # CLICKHOUSE_PASSWORD=... ./deploy/install.sh \ # --ch-host clickhouse.example.com \ @@ -17,7 +23,7 @@ # [--issuer https://accounts.google.com] \ # [--audience ] \ # audience-gated CH → also sends access_token # [--ch-auth basic] \ # OSS CH + ch-jwt-verify → JWT as Basic password -# [--cluster my_cluster] \ # single-shard multi-replica only +# [--cluster my_cluster] \ # BROKEN on multi-replica clusters (#692); omit # [--secure] \ # [--dry-run] # build + render config.json + http_handlers.xml, print, no CH contact set -euo pipefail @@ -54,15 +60,34 @@ export CLICKHOUSE_PASSWORD CH=(clickhouse-client --host "$CH_HOST" --user "$CH_USER") [[ "$SECURE" == 1 ]] && CH+=(--secure) -# user_files is node-local, and clusterAllReplicas cannot write to a multi-shard -# Distributed target, so a --cluster install only works on a single shard. +# user_files is node-local. NOTE: --cluster is currently broken (#692): the +# clusterAllReplicas INSERT below fails on any cluster with more than one +# replica (STORAGE_REQUIRES_PARAMETER). The guard fails CLOSED, before building: +# proceed only when the count query succeeds and prints exactly 1. if [[ "$DRY_RUN" != 1 && -n "$CLUSTER" ]]; then - SHARDS=$("${CH[@]}" --query "SELECT max(shard_num) FROM system.clusters WHERE cluster = '${CLUSTER}'" 2>/dev/null || true) - if [[ "$SHARDS" =~ ^[0-9]+$ ]] && (( SHARDS > 1 )); then - echo "ERROR: cluster '${CLUSTER}' has ${SHARDS} shards. clusterAllReplicas can't" >&2 - echo "write across shards, and user_files is node-local. Install per node instead" >&2 - echo "(omit --cluster; point --ch-host at each node), or serve the assets from a" >&2 - echo "replicated table — see docs/ASSET-DISTRIBUTION.md." >&2 + GUARD_ERR_FILE="$(mktemp)" + if ! NODES=$("${CH[@]}" --query "SELECT count() FROM system.clusters WHERE cluster = '${CLUSTER}'" 2>"$GUARD_ERR_FILE"); then + GUARD_ERR="$(head -n 1 "$GUARD_ERR_FILE")" + rm -f "$GUARD_ERR_FILE" + echo "ERROR: could not query system.clusters on ${CH_HOST} to check cluster" >&2 + echo "'${CLUSTER}': ${GUARD_ERR:-no error output}" >&2 + exit 2 + fi + rm -f "$GUARD_ERR_FILE" + NODES="${NODES//[[:space:]]/}" + if [[ ! "$NODES" =~ ^[0-9]+$ ]]; then + echo "ERROR: unexpected system.clusters output for cluster '${CLUSTER}':" >&2 + echo "'${NODES}' (expected a node count). Refusing to continue." >&2 + exit 2 + elif (( NODES == 0 )); then + echo "ERROR: cluster '${CLUSTER}' was not found in system.clusters on ${CH_HOST}" >&2 + echo "(0 replicas). Check the cluster name and the node you connected to." >&2 + exit 2 + elif (( NODES > 1 )); then + echo "ERROR: cluster '${CLUSTER}' has ${NODES} replicas. --cluster is currently" >&2 + echo "broken on multi-node clusters (#692); user_files is node-local. Use the" >&2 + echo "manual procedure: 'Multi-node clusters (interim, manual)' in" >&2 + echo "docs/ASSET-DISTRIBUTION.md." >&2 exit 2 fi fi @@ -119,6 +144,34 @@ HANDLERS_OUT="$ROOT/dist/http_handlers.xml" sed -E "s#(connect-src 'self')[^;]*#\1 ${CONNECT_HOSTS}#" \ "$ROOT/deploy/http_handlers.xml" > "$HANDLERS_OUT" +# Print the post-install instructions. Called after upload, and also by --dry-run +# so the URL/scheme logic is testable without a ClickHouse. The native port +# --ch-host is used with (9000/9440) differs from the HTTP(S) port the browser +# needs, so print the HTTP defaults (8123 plain / 8443 TLS) explicitly. +print_next_steps() { + local scheme="http" port="8123" host="${CH_HOST:-}" + if [[ "$SECURE" == 1 ]]; then scheme="https"; port="8443"; fi + cat < DRY RUN — no ClickHouse contact. Rendered artifacts:" @@ -126,6 +179,7 @@ if [[ "$DRY_RUN" == 1 ]]; then cat "$CONFIG_FILE" echo "--- dist/http_handlers.xml ---" cat "$HANDLERS_OUT" + print_next_steps exit 0 fi @@ -155,18 +209,6 @@ upload "$ROOT/dist/sql.html" "sql.html" echo "==> Uploading sql-config.json" upload "$CONFIG_FILE" "sql-config.json" -cat < Assets uploaded to ClickHouse user_files/. - -Final step — enable the HTTP routes. Add the rendered dist/http_handlers.xml -(its CSP connect-src is filled in for your issuer) to the server config.d/ (or -push it as an ACM cluster setting named "config.d/sql-browser.xml") and reload -ClickHouse. Then open: - - http${SECURE:+s}://$CH_HOST/sql - -Also register the OAuth redirect URI http(s)://$CH_HOST/sql with your IdP, -and make sure ClickHouse is configured to accept the bearer JWT (token_processor -+ JWKS, or a delegated http_authentication verifier). See README. -EOF +echo +echo "==> Assets uploaded to ClickHouse user_files/." +print_next_steps diff --git a/deploy/uninstall.sh b/deploy/uninstall.sh index 4f655836..bc948b06 100755 --- a/deploy/uninstall.sh +++ b/deploy/uninstall.sh @@ -5,6 +5,8 @@ # # Usage: # CLICKHOUSE_PASSWORD=... ./deploy/uninstall.sh --ch-host H [--ch-user U] [--cluster C] [--secure] +# NOTE: with --cluster this only truncates the files on the connected node: the +# CREATE/DROP run ON CLUSTER but the INSERT that empties the file is local. set -euo pipefail CH_HOST="" CH_USER="default" CLUSTER="" SECURE=0 diff --git a/docs/ASSET-DISTRIBUTION.md b/docs/ASSET-DISTRIBUTION.md index 5a8bc8f6..c196c56c 100644 --- a/docs/ASSET-DISTRIBUTION.md +++ b/docs/ASSET-DISTRIBUTION.md @@ -1,7 +1,9 @@ # Serving the assets across a ClickHouse® cluster -The Altinity® SQL Browser is served *from ClickHouse itself* — there is no -separate web server. That makes "how do the asset bytes reach every node?" a +In ClickHouse-native hosting, the Altinity® SQL Browser is served *from +ClickHouse itself* — there is no separate web server. (The containerized Caddy +runtime in [DEVELOPMENT.md](DEVELOPMENT.md) is the alternative that does use +one; this document is about the native mode.) That makes "how do the asset bytes reach every node?" a real design question on a multi-node cluster, because **ClickHouse does not replicate the `user_files/` directory**: it is a node-local folder. @@ -20,10 +22,17 @@ Two independent things must be present on every node that can answer a request: This already rides your existing config-management channel — clickhouse-operator, ACM cluster settings, a mounted ConfigMap, Ansible, etc. — which by construction reaches every node, including ones added later. - No problem here. - -2. **The ~65 KB asset bytes** (`sql.html` and `sql-config.json`). This is the - part affected by `user_files/` being node-local. + Caveat: some managed providers forbid changes to `config.d/` or + `http_handlers`. If you cannot install `http_handlers`, ClickHouse-native + hosting is impossible; use the container image, the Helm chart, or any + external static host instead (see [DEVELOPMENT.md](DEVELOPMENT.md)). + +2. **The asset bytes** (`sql.html` and `sql-config.json`). `sql.html` is a + single self-contained file of about 2 MB raw (measure your build: + `ls -l dist/sql.html`); the config is tiny. HTTP compression (gzip/br) + shrinks what goes over the wire to roughly a quarter to a third, but the + stored file, and anything that caps stored size, sees the raw bytes. This is + the part affected by `user_files/` being node-local. ## The options @@ -33,22 +42,28 @@ trade-offs. The key tension is **anonymous serving vs. native replication**: the served without credentials, which favours a static handler; native replication of the bytes favours a table, whose read path needs a user. -### A. Push to each node's `user_files` (what ships today) +### A. Push to each node's `user_files` (single node today) -`install.sh` creates a `File`-engine table and, for a cluster, uses -`INSERT INTO FUNCTION clusterAllReplicas(...)` to write the bytes into **every -replica's local `user_files/`**. The handler serves them with a `type=static` -`file://` rule. +`install.sh` (without `--cluster`) creates a `File`-engine table that writes the +bytes into the node's local `user_files/`. The handler serves them with a +`type=static` `file://` rule. **`install.sh --cluster` is currently broken** on +any cluster with more than one replica (see below); the multi-node variant of +this option is the manual procedure in +[Multi-node clusters](#multi-node-clusters-interim-manual). - **Pro:** the page is served statically, with no authentication — exactly what an unauthenticated bootstrap page needs. Simple; pure `clickhouse-client`. - **Con:** `user_files/` is node-local, so a replica **added or replaced later - starts empty** — you re-run `install.sh` after a scale-out. The install also - assumes every replica is up at the time it runs. **Multi-shard:** - `clusterAllReplicas` can't write to a target with more than one shard, so - `install.sh --cluster` only works on a single shard; on a multi-shard cluster - run the installer **per node** (omit `--cluster`, point `--ch-host` at each - node) or use option B. `install.sh` detects this and stops with guidance. + starts empty** — you re-run the install after a scale-out. The install also + assumes every replica is up at the time it runs. **`--cluster` is broken:** + `INSERT INTO FUNCTION clusterAllReplicas(...)` fails on any cluster with more + than one replica (`Code 55 STORAGE_REQUIRES_PARAMETER`: Distributed write with + more than one shard and no sharding key, regardless of `internal_replication`), + and because the script runs under `set -e`, it exits after the failed `INSERT` + and never reaches its `DROP TABLE`, so the temporary `_asb_*` table may be left + behind (see the cleanup below). `install.sh --cluster` now refuses to run + against a multi-replica cluster. Tracked in #692; use the manual procedure + below, or option B. ### B. Store the asset in a Replicated table + `predefined_query_handler` @@ -57,7 +72,9 @@ Keep the bytes in a `ReplicatedMergeTree` table and serve them with a `SELECT argMax(content, updated) FROM asb_assets WHERE name = 'sql.html' FORMAT RawBLOB`. - **Pro:** ClickHouse replication distributes the bytes to every replica - automatically, **including replicas added later**. Pure SQL, portable. + of the table's shard/replication group automatically, **including replicas + added later**. A `ReplicatedMergeTree` does not replicate to other shards, so + a multi-shard cluster needs the table (or its data) on each shard. Pure SQL, portable. - **Con:** a `predefined_query_handler` runs the query as a ClickHouse user, so the unauthenticated `/sql` GET needs a small read-only grant on the assets table for whatever anonymous resolves to. The `Replicated*` engine's @@ -72,19 +89,24 @@ serve it with the same `file://` static handler. - **Pro:** rides the exact channel that already distributes `http_handlers.xml` to all nodes, so new replicas are covered for free; still served statically (no auth, no read-grant). -- **Con:** the bytes live in config. Inlining 65 KB needs CDATA/escaping, and - some control planes cap inline config size (for example, ACM's setting-push - has an ~8 KB URI limit), so in practice the asset goes in as a *mounted file* - rather than an inline value. +- **Con:** the bytes live in config. The artifact is about 2 MB (measure your + build), which is too large for several channels: clickhouse-operator + `spec.configuration.files` lands in a Kubernetes ConfigMap, which is capped at + 1 MiB, and control planes that push settings inline (for example, ACM's + setting-push has an ~8 KB URI limit) cannot carry it either. In practice it + has to be a file delivered by some other mechanism (image layer, volume, + init container), not an inline config value. ## What this project ships -`deploy/install.sh` implements **option A**. It is the simplest mechanism, needs -nothing but `clickhouse-client`, and serves the bootstrap page anonymously. The -trade-off to be aware of: +`deploy/install.sh` implements **option A for a single node** (no `--cluster`). +It is the simplest mechanism, needs only a shell toolchain (see below), and +serves the bootstrap page anonymously. Multi-node clusters are not supported by +the installer yet (#692); see the interim procedure below. The trade-off to be +aware of: -> After scaling a cluster out (or replacing a node), re-run `install.sh` so the -> new replica's `user_files/` gets the assets. +> After scaling a cluster out (or replacing a node), re-install (or re-run the +> manual multi-node procedure) so the new replica's `user_files/` gets the assets. For deployments where that re-push is undesirable — autoscaling fleets, frequent node replacement — option B (replicated table) or option C (config-distributed @@ -92,7 +114,125 @@ file) removes it. Both are straightforward to adopt: option B is a schema + `predefined_query_handler` change to `deploy/`; option C is a change to how the file is placed on the nodes, with the existing `file://` handler unchanged. +### Installer prerequisites + +`deploy/install.sh` builds the SPA from a checkout, so it needs `npm` and +`node` (22 or newer) as well as `clickhouse-client`; `curl` is used only for +OIDC discovery (the CSP `connect-src` falls back to the Google default without +it). Each GitHub release also ships a prebuilt `sql.html` asset; if you do not +want to build, download it and upload it (plus a `sql-config.json`) to +`user_files/` yourself, then install `http_handlers.xml` as above. The installer +has no flag for this path. + +## Multi-node clusters (interim, manual) + +Until the installer is fixed (#692), pull each file onto every node with +`ON CLUSTER` DDL. Each node fetches the URL itself and writes its own +`user_files/` file. Run the sequence below **for both files** (`sql.html` and +`sql-config.json`); every step is idempotent, so on any failure you simply +start again from step 1. + +**1. Pre-clean** (removes leftovers from an earlier partial run; `IF EXISTS` +because the table may exist on only some hosts): + +```sql +DROP TABLE IF EXISTS default._asb_pull_sql_html ON CLUSTER '' SYNC; +DROP TABLE IF EXISTS default._asb_pull_sql_config_json ON CLUSTER '' SYNC; +``` + +**2. Create** (each node writes its file as a side effect of the `CREATE`): + +```sql +CREATE TABLE default._asb_pull_sql_html ON CLUSTER '' + ENGINE = File('RawBLOB', 'sql.html') + AS SELECT * FROM url('', 'RawBLOB') + SETTINGS engine_file_truncate_on_insert = 1; +``` + +For the config, use the JSON as a SQL string literal (this is what +`install.sh` renders into `sql-config.json`; adapt issuer/client id/audience): + +```sql +CREATE TABLE default._asb_pull_sql_config_json ON CLUSTER '' + ENGINE = File('RawBLOB', 'sql-config.json') + AS SELECT '{"issuer":"https://accounts.google.com","client_id":""}' AS c + SETTINGS engine_file_truncate_on_insert = 1; +``` + +The JSON must be escaped for a SQL string literal: `'` becomes `\'` and `\` +becomes `\\`. A normal config contains neither, so it can be pasted as is. The +file is written without a trailing newline (harmless to the JSON parser). + +**3. Drop the helper tables** (the files survive the `DROP`): + +```sql +DROP TABLE IF EXISTS default._asb_pull_sql_html ON CLUSTER '' SYNC; +DROP TABLE IF EXISTS default._asb_pull_sql_config_json ON CLUSTER '' SYNC; +``` + +**4. Verify on every node.** Compute the expected values locally from the +artifact you intended to deploy (`sha256sum sql.html`, or +`shasum -a 256 sql.html` on macOS; `wc -c < sql.html` for the length; likewise +for the config file, which is the exact JSON without a trailing newline). Then: + +```sql +SELECT hostName(), length(c), lower(hex(SHA256(c))) AS sha256 +FROM clusterAllReplicas('', file('sql.html', 'RawBLOB', 'c String')); + +SELECT count() FROM system.clusters WHERE cluster = ''; +``` + +(Repeat the first query with `sql-config.json`.) The deployment is successful +**only when** the first query returns exactly as many rows as the second +query's node count, and every row shows the expected length and hash, for both +files. A file missing on any node makes the verify query fail outright +(`FILE_DOESNT_EXIST`), which also counts as a failure. + +**On any failure** (an `ON CLUSTER` statement errors, some nodes cannot reach +the URL, or step 4 does not match): run step 1 again and retry from the start. +A failed `CREATE ... ON CLUSTER` can leave `_asb_pull_*` tables on the hosts +where it succeeded; a retry without the pre-clean would hit +`TABLE_ALREADY_EXISTS`, and a plain `DROP ... ON CLUSTER` (without `IF EXISTS`) +fails with `UNKNOWN_TABLE` on hosts where the table was never created. The +`IF EXISTS` pre-clean avoids both. *The `IF EXISTS` recovery sequence follows +from the ClickHouse behavior observed in testing, but the full +partial-failure-then-retry flow was not separately tested.* + +**Keep `engine_file_truncate_on_insert = 1`:** without it a re-run appends and +doubles the file. + +- **Grants** (minimal): `CLUSTER ON *.*`, `CREATE TABLE, DROP TABLE ON default.*`, + `FILE ON *.*`, `URL ON *.*`; verification also needs READ ON FILE and REMOTE. +- **Egress:** every node must reach the URL (e.g. the release asset + `https://github.com/Altinity/altinity-sql-browser/releases/download//sql.html`, + or an internal mirror), and the host must be allowed by + `remote_url_allow_hosts` if configured. GitHub release downloads redirect, so + `url()` must follow redirects, or use a mirror that serves the bytes directly. + +**Tested** on ClickHouse 25.8 OSS, 26.3 OSS and Antalya 25.8.33, on all four +nodes of a 2-shard x 2-replica cluster: byte-exact, idempotent with the +truncate setting. The `sql-config.json` literal statement was tested on a **single node of +ClickHouse 26.3 only** (byte-exact against the intended JSON, including a URL +with `?`/`&` and `\'`/`\\` escaping; idempotent on a re-run); its `ON CLUSTER` +behaviour is the same mechanism proven for `sql.html`, but was not separately +run on a multi-node cluster. **Not tested:** pulling from the real GitHub release +URL (a local HTTP server was used, so redirect handling is unverified). This +procedure is the planned basis of the fixed installer. + +### Cleaning up after a failed `install.sh --cluster` + +If you already ran the broken `--cluster` path, it may have left its temporary +tables behind on the node you connected to (and on other nodes where the +`ON CLUSTER` `CREATE` ran). Remove them: + +```sql +DROP TABLE IF EXISTS default._asb_sql_html ON CLUSTER '' SYNC; +DROP TABLE IF EXISTS default._asb_sql_config_json ON CLUSTER '' SYNC; +``` + +Dropping a `File`-engine table does not delete the file in `user_files/`. + ## Single-node installs -None of this applies to a single node: `install.sh` without `--cluster` writes +None of the above applies to a single node: `install.sh` without `--cluster` writes the assets to the one node's `user_files/` and you are done. diff --git a/docs/CLICKHOUSE-OAUTH.md b/docs/CLICKHOUSE-OAUTH.md index d4702286..66102964 100644 --- a/docs/CLICKHOUSE-OAUTH.md +++ b/docs/CLICKHOUSE-OAUTH.md @@ -191,7 +191,7 @@ defaults. See [LOGIN-SCREEN.md](LOGIN-SCREEN.md). 1. Drop the `config.d/*.xml` fragments onto every node (your config-management channel) and reload/restart ClickHouse. 2. Apply the role DDL once. -3. Register the browser's redirect URI (`https:///sql`) with your IdP. +3. Register the browser's redirect URI (e.g. `https://:8443/sql`; include the HTTP(S) port unless it is the scheme default, or use the externally exposed port/ingress host) with your IdP. 4. Smoke-test with a real token: ```bash diff --git a/docs/CLICKHOUSE-OSS-OAUTH.md b/docs/CLICKHOUSE-OSS-OAUTH.md index 844a40c3..4ffbaa9d 100644 --- a/docs/CLICKHOUSE-OSS-OAUTH.md +++ b/docs/CLICKHOUSE-OSS-OAUTH.md @@ -115,7 +115,7 @@ GRANT sql_reader TO `you@example.com`; ## 4. config.json + IdP - `config.json`: `issuer`, `client_id`, and `ch_auth: "basic"` (above). -- Register the redirect URI `https:///sql` with your IdP. +- Register the redirect URI, e.g. `https://:8443/sql` (include the HTTP(S) port unless it is the scheme default, or use the externally exposed port/ingress host), with your IdP. - Ensure the JWT actually carries the `email` claim (scopes / a mapper), since it is both the Basic username and what the verifier matches. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index ab525e74..f0feca32 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -22,14 +22,27 @@ The app is three things on the target ClickHouse: ```bash CLICKHOUSE_PASSWORD=… ./deploy/install.sh \ --ch-host clickhouse.example.com --ch-user admin \ - --client-id [--issuer …] [--audience …] [--cluster …] [--secure] + --client-id [--issuer …] [--audience …] [--ch-auth basic] \ + [--cluster …] [--secure] # --cluster: broken on multi-node clusters (#692), see below ``` -For a multi-replica cluster pass `--cluster `; the installer fans the -bytes to every replica via `clusterAllReplicas`. The password is read from the -env var or prompted — never placed on the command line. - -> **`user_files/` is node-local — re-run `install.sh` after a scale-out.** +Prerequisites: `npm` + `node` 22+ (the installer builds the SPA), `clickhouse-client`, +and `curl` (OIDC discovery only). `--audience` makes the SPA send the access +token instead of the id_token; `--ch-auth basic` sends the JWT as the HTTP Basic +password (OSS ClickHouse behind a verifier, see section 4). A GitHub release +also ships a prebuilt `sql.html` if you would rather upload it yourself. At the +end the installer prints the URL to open and the redirect URI to register; it +uses the default HTTP(S) ports 8123/8443, so adjust if yours differ. + +`--cluster ` is **currently broken** on any cluster with more than one +replica (its `clusterAllReplicas` write is rejected by ClickHouse; tracked in +#692), and the installer now refuses it before building, so use `install.sh` without `--cluster` for a single node. For +multi-node clusters see the interim manual procedure in +[ASSET-DISTRIBUTION.md](ASSET-DISTRIBUTION.md#multi-node-clusters-interim-manual). +The password is read from the env var or prompted — never placed on the command +line. + +> **`user_files/` is node-local — re-run the install (or the manual multi-node procedure) after a scale-out.** > ClickHouse does not replicate `user_files/`, so a replica added or replaced > later starts without the assets. This is a deliberate trade-off (the bootstrap > page is then served statically, with no auth). See @@ -57,12 +70,17 @@ No compression config is needed here: ClickHouse's HTTP server transparently gzips any response — including this static handler's — when the client sends `Accept-Encoding: gzip` (every browser does), independent of `` config or the `enable_http_compression` query setting. Verified live on all -three demo clusters: the ~500 KB artifact goes over the wire at ~230 KB. +three demo clusters. `dist/sql.html` is about 2 MB raw (`ls -l dist/sql.html`); +the bytes on the wire are much smaller. Measure your own build with +`curl -s -o /dev/null -w '%{size_download}\n' -H 'Accept-Encoding: gzip' https:///sql`. ## 4. Make ClickHouse accept the JWT -The SPA sends `Authorization: Bearer ` on every query. ClickHouse must -validate it. Two supported shapes: +The SPA sends the IdP token on every query. By default that is the `id_token` +as `Authorization: Bearer `; with an `audience` configured it sends the +`access_token` instead; with `"ch_auth": "basic"` the same JWT travels as the +password of `Authorization: Basic base64(user:jwt)`. ClickHouse must validate +it. Two supported shapes: - **Native token processor** (Antalya build with JWT auth): add a `` entry pointing at your IdP's JWKS with `username_claim`, @@ -75,11 +93,13 @@ validate it. Two supported shapes: `config.json` so the JWT is sent as the Basic password. Full guide with generic examples: [CLICKHOUSE-OSS-OAUTH.md](CLICKHOUSE-OSS-OAUTH.md). -Either way, the app itself is unchanged — it only sends the bearer. +Either way the SPA only ever sends the IdP-issued JWT; which header carries it +(Bearer or Basic) and which token (id_token or access_token) is set by +`config.json`. ## 5. OAuth IdP -Register the redirect URI `https:///sql` with your IdP and put the +Register the redirect URI the installer printed (e.g. `https://:8443/sql`; it must match the origin exactly, so include the HTTP(S) port unless it is the scheme default, 443/80, or use the externally exposed port/ingress host) with your IdP and put the `issuer` + `client_id` in `config.json` (the installer does this from `--issuer`/`--client-id`). Whether `config.json` also needs a `client_secret` depends on the IdP and client type — see [LOGIN-SCREEN.md](LOGIN-SCREEN.md) diff --git a/docs/deployment.html b/docs/deployment.html index e9a83d29..b92bce20 100644 --- a/docs/deployment.html +++ b/docs/deployment.html @@ -76,6 +76,8 @@

Serve it from your cluster

[--issuer https://accounts.google.com] \ [--audience <api-audience>] \ [--cluster <cluster-name>] [--secure] +

--cluster is currently broken on multi-node clusters (#692); omit it for a single node, or see the + manual multi-node procedure.

Public file, by design. config.json is served to browsers, so prefer a PKCE public client. If your IdP requires a client_secret, diff --git a/tests/unit/install-script.test.js b/tests/unit/install-script.test.js new file mode 100644 index 00000000..32f4b4fb --- /dev/null +++ b/tests/unit/install-script.test.js @@ -0,0 +1,153 @@ +// deploy/install.sh --dry-run prints the post-install next steps (#692). The +// script is copied into a scratch tree (ROOT is derived from its location) and +// run with shimmed npm/node/curl so nothing is built and nothing leaves the box. +import { describe, expect, it, beforeAll, afterAll } from 'vitest'; +import { spawnSync } from 'node:child_process'; +import { chmodSync, copyFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; + +const repo = resolve(__dirname, '../..'); +let work = ''; + +beforeAll(() => { + work = mkdtempSync(join(tmpdir(), 'asb-install-')); + mkdirSync(join(work, 'deploy')); + mkdirSync(join(work, 'dist')); + mkdirSync(join(work, 'build')); + mkdirSync(join(work, 'bin')); + copyFileSync(join(repo, 'deploy/install.sh'), join(work, 'deploy/install.sh')); + copyFileSync(join(repo, 'deploy/http_handlers.xml'), join(work, 'deploy/http_handlers.xml')); + writeFileSync(join(work, 'build/build.mjs'), ''); + writeFileSync(join(work, 'dist/sql.html'), ''); // the shimmed build produces nothing + // clickhouse-client shim: log argv and stdin size, succeed. A system.clusters + // count query answers with $ASB_NODE_COUNT (default 1; may be empty) or, with + // $ASB_FAIL set, fails with stderr. + const ch = join(work, 'bin', 'clickhouse-client'); + writeFileSync(ch, `#!/bin/sh +{ printf 'ARGS:'; for a in "$@"; do printf ' [%s]' "$(printf %s "$a" | tr '\\n' ' ')"; done; printf ' STDIN:%s\\n' "$(wc -c | tr -d ' ')"; } >> "${work}/ch.log" +case "$*" in *system.clusters*) + if [ -n "\${ASB_FAIL:-}" ]; then echo "Code: 210. DB::NetException: Connection refused" >&2; exit 210; fi + echo "\${ASB_NODE_COUNT-1}"; exit 0;; +esac +exit 0 +`); + chmodSync(ch, 0o755); + for (const [name, code] of [['npm', 0], ['node', 0], ['curl', 22]]) { + const p = join(work, 'bin', name); + writeFileSync(p, `#!/bin/sh\necho ${name} >> "${work}/build.log"\nexit ${code}\n`); + chmodSync(p, 0o755); + } +}); +afterAll(() => rmSync(work, { recursive: true, force: true })); + +function dryRun(...args) { + const r = spawnSync('bash', [join(work, 'deploy/install.sh'), '--client-id', 'cid', '--dry-run', ...args], { + env: { ...process.env, PATH: `${join(work, 'bin')}:${process.env.PATH ?? ''}` }, + encoding: 'utf8', + }); + expect(r.status, r.stderr).toBe(0); + return r.stdout; +} + +describe('install.sh next-steps output', () => { + it('prints http and the plain HTTP port without --secure', () => { + const out = dryRun('--ch-host', 'ch.example.com'); + expect(out).toContain('http://ch.example.com:8123/sql'); + expect(out).not.toContain('https://ch.example.com'); + }); + + it('prints https and the TLS HTTP port with --secure', () => { + const out = dryRun('--ch-host', 'ch.example.com', '--secure'); + expect(out).toContain('https://ch.example.com:8443/sql'); + expect(out).not.toContain('http://ch.example.com'); + }); + + it('uses a placeholder host when --ch-host is omitted in a dry run', () => { + expect(dryRun()).toContain('http://:8123/sql'); + }); +}); + +describe('install.sh real upload path', () => { + function install(...args) { + const log = join(work, 'ch.log'); + rmSync(log, { force: true }); + const r = spawnSync('bash', [join(work, 'deploy/install.sh'), '--client-id', 'cid', '--ch-host', 'ch.example.com', ...args], { + env: { ...process.env, CLICKHOUSE_PASSWORD: 'pw', PATH: `${join(work, 'bin')}:${process.env.PATH ?? ''}` }, + encoding: 'utf8', + }); + expect(r.status, r.stderr).toBe(0); + return { out: r.stdout, calls: readFileSync(log, 'utf8').trim().split('\n') }; + } + + it('uploads both assets and prints http next steps without --secure', () => { + const { out, calls } = install(); + expect(calls.some((c) => c.includes('_asb_sql_html'))).toBe(true); + expect(calls.some((c) => c.includes('_asb_sql_config_json'))).toBe(true); + expect(calls.some((c) => c.includes('[--secure]'))).toBe(false); + expect(out).toContain('==> Assets uploaded'); + expect(out).toContain('http://ch.example.com:8123/sql'); + expect(out).not.toContain('https://ch.example.com'); + }); + + it('passes --secure to every call and prints https next steps with --secure', () => { + const { out, calls } = install('--secure'); + expect(calls.some((c) => c.includes('_asb_sql_html'))).toBe(true); + expect(calls.some((c) => c.includes('_asb_sql_config_json'))).toBe(true); + expect(calls.every((c) => c.includes('[--secure]'))).toBe(true); + expect(out).toContain('https://ch.example.com:8443/sql'); + expect(out).not.toContain('http://ch.example.com'); + }); +}); + +describe('install.sh --cluster guard (#692)', () => { + function clusterInstall(count, fail = false) { + rmSync(join(work, 'ch.log'), { force: true }); + rmSync(join(work, 'build.log'), { force: true }); + const env = { ...process.env, ASB_NODE_COUNT: String(count), CLICKHOUSE_PASSWORD: 'pw', PATH: `${join(work, 'bin')}:${process.env.PATH ?? ''}` }; + if (fail) env.ASB_FAIL = '1'; + const r = spawnSync('bash', [join(work, 'deploy/install.sh'), '--client-id', 'cid', '--ch-host', 'ch.example.com', '--cluster', 'c1'], { env, encoding: 'utf8' }); + let built = ''; + try { built = readFileSync(join(work, 'build.log'), 'utf8'); } catch { /* no build ran */ } + return { r, built }; + } + + it('fails fast with exit 2 before building when the cluster has more than one replica', () => { + const { r, built } = clusterInstall(2); + expect(r.status).toBe(2); + expect(r.stderr).toContain('has 2 replicas'); + expect(r.stderr).toContain('#692'); + expect(r.stderr).toContain('docs/ASSET-DISTRIBUTION.md'); + expect(built).toBe(''); + }); + + it('fails closed when the count query fails, surfacing the first stderr line but not the password', () => { + const { r, built } = clusterInstall(1, true); + expect(r.status).toBe(2); + expect(r.stderr).toContain('could not query system.clusters'); + expect(r.stderr).toContain('Connection refused'); + expect(r.stderr).not.toContain('pw\n'); + expect(built).toBe(''); + }); + + it('fails closed when the cluster is not found (count 0)', () => { + const { r, built } = clusterInstall(0); + expect(r.status).toBe(2); + expect(r.stderr).toContain("was not found in system.clusters"); + expect(built).toBe(''); + }); + + it.each([['empty', ''], ['malformed', 'abc']])('fails closed on %s output', (_n, out) => { + const { r, built } = clusterInstall(out); + expect(r.status).toBe(2); + expect(r.stderr).toContain('unexpected system.clusters output'); + expect(built).toBe(''); + }); + + it('proceeds for a one-node cluster', () => { + const { r, built } = clusterInstall(1); + expect(r.status, r.stderr).toBe(0); + expect(built).toContain('npm'); + expect(r.stdout).toContain('==> Assets uploaded'); + }); +});