Skip to content

Commit 1e0906b

Browse files
filimonovfilimonov
authored
Update users_in_keeper.md
1 parent 4a0050c commit 1e0906b

1 file changed

Lines changed: 135 additions & 117 deletions

File tree

‎content/en/altinity-kb-setup-and-maintenance/users_in_keeper.md‎

Lines changed: 135 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,27 @@ This KB explains how to make SQL RBAC changes (`CREATE USER`, `CREATE ROLE`, `GR
1212

1313
`Keeper` below means either ClickHouse Keeper or ZooKeeper.
1414

15-
## 1. Why use this instead of only `ON CLUSTER` for RBAC?
15+
Before details, the core concept is:
16+
- ClickHouse stores access entities in access storages configured by `user_directories`.
17+
- By default, following the shared-nothing concept, SQL RBAC objects are local (`local_directory`), so changes done on one node do not automatically appear on another node unless you run `... ON CLUSTER ...`.
18+
- With `user_directories.replicated`, ClickHouse stores the RBAC model in Keeper under a configured path (for example `/clickhouse/access`) and every node watches that path.
19+
- Each node keeps a local in-memory mirror of replicated access entities and updates it from Keeper watch notifications. This is why normal access checks are local-memory fast, while RBAC writes depend on Keeper availability.
20+
21+
Important mental model:
22+
- this feature replicates RBAC state (users, roles, grants, policies, profiles, quotas, masking policies);
23+
- it is not the same mechanism as distributed DDL queue execution used by `ON CLUSTER`.
24+
25+
Flow of this KB:
26+
1. Why this model helps.
27+
2. How to configure it on a new cluster.
28+
3. How to validate and operate it.
29+
4. How to migrate existing RBAC safely.
30+
5. Advanced troubleshooting and internals.
31+
32+
## 1. Choose the RBAC replication model (`ON CLUSTER` vs Keeper)
1633

1734
`ON CLUSTER` executes DDL on hosts that exist at execution time.
35+
In practice, it fans out the query through the distributed DDL queue to currently known cluster nodes.
1836
It does not automatically replay old RBAC DDL for replicas/shards added later.
1937

2038
Keeper-backed RBAC solves that:
@@ -39,110 +57,7 @@ Cons:
3957
- Very large RBAC sets (thousands of users/roles or very complex grants) can increase Keeper/watch pressure.
4058
- If Keeper is unavailable during server startup and replicated RBAC storage is configured, startup can fail, so DBA login is unavailable until startup succeeds.
4159

42-
## 2. Backup and migration first (important)
43-
44-
Before switching to Keeper-backed RBAC, treat this as a migration.
45-
46-
Key facts:
47-
- Changing `user_directories` storage or changing `zookeeper_path` does **not** move existing SQL RBAC objects automatically.
48-
- If path changes, old users/roles are not deleted, but become effectively hidden from the new storage path.
49-
- `zookeeper_path` cannot be changed at runtime via SQL.
50-
51-
Recommended migration sequence:
52-
1. Back up RBAC objects.
53-
2. Apply the new `user_directories` config on all nodes.
54-
3. Restart/reload config as required by your environment.
55-
4. Restore/recreate RBAC objects to the target storage.
56-
5. Validate on all nodes.
57-
58-
### 2.1 Migration with pure SQL (no backup tool)
59-
60-
This path is useful when:
61-
- RBAC DDL is already versioned in your repo, or
62-
- you want to dump/replay access entities using SQL only.
63-
64-
Recommended SQL-only flow:
65-
1. On source, check where entities are stored (local vs replicated):
66-
67-
```sql
68-
SELECT name, storage FROM system.users ORDER BY name;
69-
SELECT name, storage FROM system.roles ORDER BY name;
70-
SELECT name, storage FROM system.settings_profiles ORDER BY name;
71-
SELECT name, storage FROM system.quotas ORDER BY name;
72-
SELECT name, storage FROM system.row_policies ORDER BY name;
73-
SELECT name, storage FROM system.masking_policies ORDER BY name;
74-
```
75-
76-
2. Export RBAC DDL from source:
77-
- simplest full dump:
78-
79-
```sql
80-
SHOW ACCESS;
81-
```
82-
83-
Save output as SQL (for example `rbac_dump.sql`) in your repo/artifacts.
84-
85-
You can also export individual objects with `SHOW CREATE USER/ROLE/...` when needed.
86-
87-
3. Switch config to replicated `user_directories` on target cluster and restart/reload.
88-
4. Replay exported SQL on one node (without `ON CLUSTER` in replicated mode).
89-
5. Validate from another node (`SHOW CREATE USER ...`, `SHOW GRANTS FOR ...`).
90-
91-
### 2.2 Migration with `clickhouse-backup` (external tool)
92-
93-
```bash
94-
# backup local RBAC users/roles/etc.
95-
clickhouse-backup create --rbac --rbac-only users_bkp_20260304
96-
97-
# restore (on node configured with replicated user directory)
98-
clickhouse-backup restore --rbac-only users_bkp_20260304
99-
```
100-
101-
Important:
102-
- this applies to SQL/RBAC users (created with `CREATE USER ...`, `CREATE ROLE ...`, etc.);
103-
- if your users are in `users.xml`, those are config-based (`--configs`) and this is not an automatic local->replicated RBAC conversion.
104-
105-
### 2.3 Migration with embedded ClickHouse SQL `BACKUP/RESTORE`
106-
107-
```sql
108-
BACKUP
109-
TABLE system.users,
110-
TABLE system.roles,
111-
TABLE system.row_policies,
112-
TABLE system.quotas,
113-
TABLE system.settings_profiles,
114-
TABLE system.masking_policies
115-
TO <backup_destination>;
116-
117-
-- after switching config
118-
RESTORE
119-
TABLE system.users,
120-
TABLE system.roles,
121-
TABLE system.row_policies,
122-
TABLE system.quotas,
123-
TABLE system.settings_profiles,
124-
TABLE system.masking_policies
125-
FROM <backup_destination>;
126-
```
127-
128-
`allow_backup` behavior for embedded SQL backup/restore:
129-
- Storage-level flag in `user_directories` (`<replicated>`, `<local_directory>`, `<users_xml>`) controls whether that storage participates in backup/restore.
130-
- Entity-level setting `allow_backup` (for users/roles/settings profiles) can exclude specific RBAC objects from backup.
131-
132-
Defaults in ClickHouse code:
133-
- `users_xml`: `allow_backup = false` by default.
134-
- `local_directory`: `allow_backup = true` by default.
135-
- `replicated`: `allow_backup = true` by default.
136-
137-
Operational implication:
138-
- If you disable `allow_backup` for replicated storage, embedded `BACKUP TABLE system.users ...` may skip those entities (or fail if no backup-allowed access storage remains).
139-
140-
About `clickhouse-backup --rbac/--rbac-only`:
141-
- It is an external tool, not ClickHouse embedded backup by itself.
142-
- If `clickhouse-backup` is configured with `use_embedded_backup_restore: true`, it delegates to SQL `BACKUP/RESTORE` and follows embedded rules.
143-
- Otherwise it uses its own workflow; do not assume full equivalence with embedded `allow_backup` semantics.
144-
145-
## 3. Minimal server configuration
60+
## 2. Configure Keeper-backed RBAC on a new cluster
14661

14762
`user_directories` is the ClickHouse server configuration section that defines:
14863
- where access entities are read from (`users.xml`, local SQL access files, Keeper, LDAP, etc.),
@@ -170,7 +85,7 @@ Why `replace="replace"` matters:
17085
- defaults include `local_directory`, so SQL RBAC may still be written locally;
17186
- this can cause mixed behavior (some entities in Keeper, some in local files).
17287

173-
### 3.1 `user_directories` behavior, defaults, and coexistence
88+
### 2.1 Understand `user_directories`: defaults, precedence, coexistence
17489

17590
What can be configured in `user_directories`:
17691
- `users_xml` (read-only config users),
@@ -200,7 +115,7 @@ CREATE USER my_user IDENTIFIED BY '***' IN replicated;
200115
This is supported, but for access control we usually do **not** recommend mixing storages intentionally.
201116
For sensitive access rights, a single source of truth (typically `replicated`) is safer and easier to operate.
202117

203-
## 4. Altinity operator (CHI) example
118+
## 3. Altinity Operator (CHI) configuration example
204119

205120
```yaml
206121
apiVersion: clickhouse.altinity.com/v1
@@ -223,7 +138,7 @@ spec:
223138
</clickhouse>
224139
```
225140
226-
## 5. Quick validation checklist
141+
## 4. Validate the setup quickly
227142
228143
Check active storages and precedence:
229144
@@ -254,11 +169,11 @@ FROM system.zookeeper
254169
WHERE path = '/clickhouse/access';
255170
```
256171

257-
## 6. Relationship with `ON CLUSTER` (important)
172+
## 5. Handle existing `ON CLUSTER` RBAC scripts safely
258173

259174
There are two independent propagation mechanisms:
260175
- Replicated access storage: Keeper-based replication of RBAC entities.
261-
- `ON CLUSTER`: distributed DDL queue execution.
176+
- `ON CLUSTER`: query fan-out through the distributed DDL queue (also Keeper/ZooKeeper-dependent).
262177

263178
When replicated access storage is enabled, combining both can be redundant or problematic.
264179

@@ -294,7 +209,110 @@ Also decide your strictness for invalid replicated entities:
294209
- `true`: fail fast on invalid entity payload in Keeper.
295210
- `false`: log and skip invalid entity.
296211

297-
## 7. Common support issues (generalized)
212+
## 6. Migrate existing clusters/users
213+
214+
Before switching to Keeper-backed RBAC, treat this as a storage migration.
215+
216+
Key facts before migration:
217+
- Changing `user_directories` storage or changing `zookeeper_path` does **not** move existing SQL RBAC objects automatically.
218+
- If path changes, old users/roles are not deleted, but become effectively hidden from the new storage path.
219+
- `zookeeper_path` cannot be changed at runtime via SQL.
220+
221+
Recommended high-level steps:
222+
1. Export/backup RBAC.
223+
2. Apply the new `user_directories` config on all nodes.
224+
3. Restart/reload as needed.
225+
4. Restore/replay RBAC.
226+
5. Validate from multiple nodes.
227+
228+
### 6.1 SQL-only migration (export/import RBAC DDL)
229+
230+
This path is useful when:
231+
- RBAC DDL is already versioned in your repo, or
232+
- you want to dump/replay access entities using SQL only.
233+
234+
Recommended SQL-only flow:
235+
1. On source, check where entities are stored (local vs replicated):
236+
237+
```sql
238+
SELECT name, storage FROM system.users ORDER BY name;
239+
SELECT name, storage FROM system.roles ORDER BY name;
240+
SELECT name, storage FROM system.settings_profiles ORDER BY name;
241+
SELECT name, storage FROM system.quotas ORDER BY name;
242+
SELECT name, storage FROM system.row_policies ORDER BY name;
243+
SELECT name, storage FROM system.masking_policies ORDER BY name;
244+
```
245+
246+
2. Export RBAC DDL from source:
247+
- simplest full dump:
248+
249+
```sql
250+
SHOW ACCESS;
251+
```
252+
253+
Save output as SQL (for example `rbac_dump.sql`) in your repo/artifacts.
254+
255+
You can also export individual objects with `SHOW CREATE USER/ROLE/...` when needed.
256+
257+
3. Switch config to replicated `user_directories` on target cluster and restart/reload.
258+
4. Replay exported SQL on one node (without `ON CLUSTER` in replicated mode).
259+
5. Validate from another node (`SHOW CREATE USER ...`, `SHOW GRANTS FOR ...`).
260+
261+
### 6.2 Migration with `clickhouse-backup` (`--rbac-only`)
262+
263+
```bash
264+
# backup local RBAC users/roles/etc.
265+
clickhouse-backup create --rbac --rbac-only users_bkp_20260304
266+
267+
# restore (on node configured with replicated user directory)
268+
clickhouse-backup restore --rbac-only users_bkp_20260304
269+
```
270+
271+
Important:
272+
- this applies to SQL/RBAC users (created with `CREATE USER ...`, `CREATE ROLE ...`, etc.);
273+
- if your users are in `users.xml`, those are config-based (`--configs`) and this is not an automatic local->replicated RBAC conversion.
274+
275+
### 6.3 Migration with embedded SQL `BACKUP/RESTORE`
276+
277+
```sql
278+
BACKUP
279+
TABLE system.users,
280+
TABLE system.roles,
281+
TABLE system.row_policies,
282+
TABLE system.quotas,
283+
TABLE system.settings_profiles,
284+
TABLE system.masking_policies
285+
TO <backup_destination>;
286+
287+
-- after switching config
288+
RESTORE
289+
TABLE system.users,
290+
TABLE system.roles,
291+
TABLE system.row_policies,
292+
TABLE system.quotas,
293+
TABLE system.settings_profiles,
294+
TABLE system.masking_policies
295+
FROM <backup_destination>;
296+
```
297+
298+
`allow_backup` behavior for embedded SQL backup/restore:
299+
- Storage-level flag in `user_directories` (`<replicated>`, `<local_directory>`, `<users_xml>`) controls whether that storage participates in backup/restore.
300+
- Entity-level setting `allow_backup` (for users/roles/settings profiles) can exclude specific RBAC objects from backup.
301+
302+
Defaults in ClickHouse code:
303+
- `users_xml`: `allow_backup = false` by default.
304+
- `local_directory`: `allow_backup = true` by default.
305+
- `replicated`: `allow_backup = true` by default.
306+
307+
Operational implication:
308+
- If you disable `allow_backup` for replicated storage, embedded `BACKUP TABLE system.users ...` may skip those entities (or fail if no backup-allowed access storage remains).
309+
310+
About `clickhouse-backup --rbac/--rbac-only`:
311+
- It is an external tool, not ClickHouse embedded backup by itself.
312+
- If `clickhouse-backup` is configured with `use_embedded_backup_restore: true`, it delegates to SQL `BACKUP/RESTORE` and follows embedded rules.
313+
- Otherwise it uses its own workflow; do not assume full equivalence with embedded `allow_backup` semantics.
314+
315+
## 7. Troubleshooting: common support issues
298316

299317
| Symptom | Typical root cause | What to do |
300318
|---|---|---|
@@ -312,7 +330,7 @@ Also decide your strictness for invalid replicated entities:
312330
| Short window where user seems present/absent via load balancer | Propagation + node routing timing | Validate directly on each node; avoid assuming LB view is instantly consistent |
313331
| Server fails after aggressive `user_directories` replacement | Required base users/profiles missing in config | Keep `users_xml` (or equivalent base definitions) intact |
314332

315-
## 8. Operational guardrails
333+
## 8. Operational guardrails for production
316334

317335
- Keep the same `user_directories` config on all nodes.
318336
- Keep `zookeeper_path` unique per cluster/tenant.
@@ -321,7 +339,7 @@ Also decide your strictness for invalid replicated entities:
321339
- Treat Keeper health as part of access-management SLO.
322340
- Plan RBAC backup/restore before changing storage path or cluster topology.
323341

324-
## 9. Debugging and observability
342+
## 9. Observability and debugging signals
325343

326344
Keeper connectivity:
327345

@@ -367,7 +385,7 @@ Force access reload:
367385
SYSTEM RELOAD USERS;
368386
```
369387

370-
## 10. Keeper structure (advanced troubleshooting)
388+
## 10. Keeper path structure and semantics (advanced)
371389

372390
If `zookeeper_path=/clickhouse/access`:
373391

@@ -401,14 +419,14 @@ Advanced note:
401419
- primary cache: `MemoryAccessStorage` inside replicated access storage;
402420
- higher-level caches in `AccessControl` (`RoleCache`, `RowPolicyCache`, `QuotaCache`, `SettingsProfilesCache`) are updated/invalidated via access change notifications.
403421

404-
## 11. Low-level behavior that explains real incidents
422+
## 11. Low-level internals behind real incidents
405423

406424
- Read path is memory-backed (`MemoryAccessStorage` mirror), not direct Keeper reads per query.
407425
- Write path requires Keeper availability; if Keeper is down, RBAC writes fail while some reads can continue from loaded state.
408426
- Insert target is selected by storage order and writeability in `MultipleAccessStorage`; this is why leftover `local_directory` can hijack SQL user creation.
409427
- `ignore_on_cluster_for_replicated_access_entities_queries` is implemented as AST rewrite that removes `ON CLUSTER` for access queries when replicated access storage is enabled.
410428

411-
## 12. History highlights
429+
## 12. Version and history highlights
412430

413431
| Date | Change | Why it matters |
414432
|---|---|---|
@@ -418,7 +436,7 @@ Advanced note:
418436
| 2025-06-03 | Keeper replication logic extracted to `ZooKeeperReplicator` (`39eb90b73ef`, PR #81245) | Cleaner architecture, shared replication core |
419437
| 2026-01-24 | Optional strict mode on invalid replicated entities (`3d654b79853`) | Lets operators fail fast on corrupted Keeper payloads |
420438

421-
## 13. References for engineers
439+
## 13. Code references for deep dives
422440

423441
- `src/Access/AccessControl.cpp`
424442
- `src/Access/MultipleAccessStorage.cpp`

0 commit comments

Comments
 (0)