diff --git a/CHANGELOG.md b/CHANGELOG.md index d8a0de2..e6eb4e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,20 @@ expose. ClickHouse user default database. It also calls out the `REMOTE` grant for clustered preflight and the S3 IAM permissions needed by collect and delete. +- Documented the collection scope for an installation-wide cleanup. An object + layout with a `//` level tempted an operator to + collect one replica's prefix, which misses the other replicas' orphans. The + guides now say to collect the parent prefix into one inventory table and + rely on the clustered comparison against every replica's + `system.remote_data_paths` — but only when that parent prefix holds one + `S3DISKNAME`'s objects, since the orphan check matches by disk name and + would misclassify another disk's objects under the same prefix as orphaned. + +- Documented that static mode does not read the standard `AWS_ACCESS_KEY_ID` + and `AWS_SECRET_ACCESS_KEY` variables, with a mapping to the `S3GC_*` + Secret keys, including the optional session token in the copy-paste Secret + creation commands. + ## [0.7.0] - 2026-08-24 A safety release, and its headline is uncomfortable: the dangerous diff --git a/README.md b/README.md index ba96c78..0da9743 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,13 @@ Deleting an object is irreversible. `s3gc` preserves the following controls: - The Kubernetes renderer requires a digest-pinned image and keeps credentials outside the manifest. +For an installation-wide cleanup, collect the parent prefix that holds every +replica's objects and set the cluster name, so the comparison covers all +replicas. A single-replica prefix misses the other replicas' orphans. Only +collect a parent prefix that holds one `S3DISKNAME`'s objects — the orphan +check matches by disk name, so another disk's objects under the same prefix +would look orphaned. + Use a unique collection-table prefix for each bucket and prefix. Never commit credentials, rendered manifests, target-cluster details, or run output. @@ -82,6 +89,9 @@ Choose one S3 authentication mode with `S3GC_S3AUTH`: | `aws` | boto3 chain; optional `S3GC_S3PROFILE` | AWS SSO or a named workstation profile | | `iam` | MinIO workload-identity provider | EKS IRSA, EC2 instance profiles, or ECS task roles | +Static mode does not read `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY`. Pass +those values as `S3GC_S3ACCESSKEY` and `S3GC_S3SECRETKEY`. + `S3GC_S3PROFILE` selects `aws`; the tool rejects contradictory settings. An `aws` preview still needs `s3:ListBucket` for the configured bucket and prefix. diff --git a/deploy/kubernetes/README.md b/deploy/kubernetes/README.md index 4c51f34..de636e8 100644 --- a/deploy/kubernetes/README.md +++ b/deploy/kubernetes/README.md @@ -53,6 +53,21 @@ Set the values in `s3gc.env`, then check these five items before rendering: 4. Choose a unique, database-qualified `COLLECTTABLEPREFIX`, such as `s3gc.s3gc__`. Keep it after a partial delete so the replacement Job can use the deletion checkpoints. A bare prefix uses the ClickHouse user's current database, which can differ from `default` and cause a grant failure. 5. Set `USEAGE_HOURS` to 24 or more. Raise it for slow merges or long mutations; production phases cannot lower it. +For an installation-wide cleanup, set `S3PATH` to the parent prefix that +contains every replica's objects, such as `clickhouse//`, not a +single `//` prefix — but only when that parent prefix +holds objects for the one `S3DISKNAME` being cleaned up. The orphan check +matches ClickHouse's `system.remote_data_paths` by `disk_name`, so an object +that belongs to a different disk stored under the same parent prefix would +never match and would look orphaned. When disks share a parent prefix, collect +a disk-scoped prefix instead, or give each disk its own S3 path layout. +Collect the chosen prefix once into one inventory table on the pinned host. +With `CLUSTERNAME` set, dry-run and delete compare +the inventory with `system.remote_data_paths` on all replicas, so an object +that any replica still references is never a candidate. A single-replica +prefix misses the other replicas' orphans. Collect per-replica prefixes only +when you intend to, and give each one its own `COLLECTTABLEPREFIX`. + Use an immutable multi-architecture image digest in `IMAGE`. CI prints the exact value after publishing. Do not use an image tag. @@ -106,13 +121,24 @@ its own S3 client and the Secret or workload identity instead. For `S3AUTH=static`, create the Secret from a private, shell-style credential file. Keep this file outside the repository, restrict it to its owner, and -never print or commit it. Its values must be exported as follows: +never print or commit it. s3gc does not read `AWS_ACCESS_KEY_ID` or +`AWS_SECRET_ACCESS_KEY` in static mode. Store those values under the s3gc +names instead: + +| AWS variable | Secret key | +| --- | --- | +| `AWS_ACCESS_KEY_ID` | `S3GC_S3ACCESSKEY` | +| `AWS_SECRET_ACCESS_KEY` | `S3GC_S3SECRETKEY` | +| `AWS_SESSION_TOKEN` | `S3GC_S3SESSIONTOKEN` (optional) | + +The file must export these values: ```bash export S3GC_CHUSER=s3gc export S3GC_CHPASS='' export S3GC_S3ACCESSKEY='' export S3GC_S3SECRETKEY='' +# export S3GC_S3SESSIONTOKEN='' # only for temporary credentials ``` Create the Secret. This command creates it once and refuses to overwrite an @@ -123,13 +149,20 @@ set -a . /private/path/s3gc-secrets.env set +a -kubectl -n create secret generic \ - --from-literal=S3GC_CHUSER="$S3GC_CHUSER" \ - --from-literal=S3GC_CHPASS="$S3GC_CHPASS" \ - --from-literal=S3GC_S3ACCESSKEY="$S3GC_S3ACCESSKEY" \ +literals=( + --from-literal=S3GC_CHUSER="$S3GC_CHUSER" + --from-literal=S3GC_CHPASS="$S3GC_CHPASS" + --from-literal=S3GC_S3ACCESSKEY="$S3GC_S3ACCESSKEY" --from-literal=S3GC_S3SECRETKEY="$S3GC_S3SECRETKEY" +) +if [ -n "${S3GC_S3SESSIONTOKEN:-}" ]; then + literals+=(--from-literal=S3GC_S3SESSIONTOKEN="$S3GC_S3SESSIONTOKEN") +fi + +kubectl -n create secret generic \ + "${literals[@]}" -unset S3GC_CHUSER S3GC_CHPASS S3GC_S3ACCESSKEY S3GC_S3SECRETKEY +unset S3GC_CHUSER S3GC_CHPASS S3GC_S3ACCESSKEY S3GC_S3SECRETKEY S3GC_S3SESSIONTOKEN ``` The S3 principal needs `s3:ListBucket` on the scoped bucket/prefix for