Skip to content

Commit 5c945b5

Browse files
Revise resource allocation details for ClickHouse merges
Updated the section on resource allocation for merging processes in ClickHouse, including clarifications on settings and their implications. Adjusted formatting and removed redundant information for better readability.
1 parent 49c228c commit 5c945b5

1 file changed

Lines changed: 9 additions & 17 deletions

File tree

content/en/altinity-kb-setup-and-maintenance/altinity-kb-aggressive_merges.md

Lines changed: 9 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -9,23 +9,17 @@ description: >
99

1010
Q: Is there any way I can dedicate more resources to the merging process when running ClickHouse® on pretty beefy machines (like 36 cores, 1TB of RAM, and large NVMe disks)?
1111

12-
## When this article applies
12+
A: Such things are done by increasing the level of parallelism:
1313

14-
This article is for bulk-ingest or merge-backlog workloads on oversized hardware, where you want ClickHouse® to spend more CPU and I/O budget on background merges. It is not a generic tuning template for clusters that must sustain low-latency reads and writes on the same nodes.
15-
16-
Those recommendations are NOT generic. For systems with real-time insert and select pressure happening together with merges, those adjustments can be too aggressive. If you have different setups with different usage patterns, avoid using the same aggressive settings template for all of them.
17-
18-
Mostly such things are done by changing the level of parallelism:
19-
20-
1.  `background_pool_size` - how many threads will actually be doing merges and mutations. If you can push most server resources toward merges, for example in a controlled backlog-clearing window with little foreground traffic, you can raise it aggressively. If you use replicated tables, review `max_replicated_merges_in_queue` together with it.
14+
1.  `background_pool_size` - how many threads will actually be doing merges and mutations. If you can push most server resources toward merges, for example, in a controlled backlog-clearing window with little foreground traffic, you can raise it aggressively. If you use replicated tables, review `max_replicated_merges_in_queue` together with it.
2115

2216
2.  `background_merges_mutations_concurrency_ratio` - how many merges and mutations may be assigned relative to `background_pool_size`. Sometimes the default (`2`) may work against you by favoring more smaller tasks, which is useful for continuous real-time inserts but less useful when you want a backlog-clearing merge window. In that case, trying `1` is reasonable.
2317

24-
3. `number_of_free_entries_in_pool_to_lower_max_size_of_merge` (merge_tree setting) should be changed together with background_pool_size (50-90% of that). "When there is less than a specified number of free entries in the pool (or replicated queue), start to lower the maximum size of the merge to process (or to put in the queue). This is to allow small merges to process - not filling the pool with long-running merges." To make it really aggressive try 90-95% of background_pool_size, for ex. 34 (so you will have 34 huge merges and 2 small ones).
18+
3. `number_of_free_entries_in_pool_to_lower_max_size_of_merge` (merge_tree setting) should be changed together with background_pool_size (50-90% of that). "When there is less than a specified number of free entries in the pool (or replicated queue), start to lower the maximum size of the merge to process (or to put in the queue). This is to allow small merges to process - not filling the pool with long-running merges." To make it really aggressive, try 90-95% of background_pool_size, for ex. 34 (so you will have 34 huge merges and 2 small ones).
2519

2620
## Runtime vs restart semantics
2721

28-
Treat this template as a deliberate capacity-mode change, not casual live tuning. According to current ClickHouse docs, `background_pool_size` and `background_merges_mutations_concurrency_ratio` can be increased at runtime, but lowering them requires a restart.
22+
`background_pool_size` and `background_merges_mutations_concurrency_ratio` can be increased at runtime, but lowering them requires a restart.
2923

3024
## Merge scheduling tradeoffs
3125

@@ -36,8 +30,6 @@ Treat this template as a deliberate capacity-mode change, not casual live tuning
3630

3731
## Other settings to consider
3832

39-
Additionally, you can:
40-
4133
- control how large target parts may become via `max_bytes_to_merge_at_max_space_in_pool` if the backlog is dominated by many medium parts instead of tiny fragments.
4234
- review `min_merge_bytes_to_use_direct_io` if you suspect page-cache churn during very large merges. Direct I/O is workload-dependent, so benchmark it instead of assuming it is always better or worse.
4335
- on replicated tables with slow merges and a fast network, consider `execute_merges_on_single_replica_time_threshold` so one replica performs the merge and others can fetch the merged part instead of repeating the same work.
@@ -65,7 +57,7 @@ If the relevant pool task counters stay near saturation while backlog does not i
6557
- the cluster is already constrained by disk bandwidth rather than merge thread count;
6658
- the workload is dominated by mutations, where `number_of_free_entries_in_pool_to_execute_mutation` may need separate treatment.
6759

68-
## Current recommended server config
60+
## Server config example
6961

7062
```
7163
cat /etc/clickhouse-server/config.d/aggresive_merges.xml
@@ -91,10 +83,10 @@ Only use the `default` profile layout if you are intentionally keeping an older
9183
cat /etc/clickhouse-server/users.d/aggresive_merges.xml
9284
<clickhouse>
9385
<profiles>
94-
<default>
95-
<background_pool_size>36</background_pool_size>
96-
<background_merges_mutations_concurrency_ratio>1</background_merges_mutations_concurrency_ratio>
97-
</default>
86+
<default>
87+
<background_pool_size>36</background_pool_size>
88+
<background_merges_mutations_concurrency_ratio>1</background_merges_mutations_concurrency_ratio>
89+
</default>
9890
</profiles>
9991
</clickhouse>
10092
```

0 commit comments

Comments
 (0)