diff --git a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/_index.md b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/_index.md index 5f82f3de57..1caed0930d 100644 --- a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/_index.md @@ -1,15 +1,19 @@ --- -title: Learn how to Tune PostgreSQL +title: Tune PostgreSQL performance on Arm-based platforms +description: Learn how to tune PostgreSQL configuration, Linux memory settings, and storage options to improve database performance on Arm-based platforms. -minutes_to_complete: 30 +minutes_to_complete: 150 -who_is_this_for: This is an advanced topic for software developers and DevOps professionals interested in optimizing PostgreSQL performance. +who_is_this_for: This Learning Path is for database administrators (DBAs) and software developers who want to optimize PostgreSQL performance on Arm-based platforms. learning_objectives: - - Tune PostgreSQL to increase performance + - Configure PostgreSQL settings that affect connection handling, memory use, write-ahead logging, query planning, and concurrency. + - Enable huge pages for PostgreSQL and size them for the shared memory area. + - Evaluate storage, kernel, compiler, and library choices that can affect PostgreSQL performance. prerequisites: - - Bare-metal or cloud [installation of PostgreSQL](/learning-paths/servers-and-cloud-computing/postgresql/) + - On-prem or cloud [installation of PostgreSQL](/learning-paths/servers-and-cloud-computing/postgresql/) + - A repeatable PostgreSQL workload or benchmark that you can run before and after tuning author: Julio Suarez @@ -42,12 +46,24 @@ tools_software_languages: further_reading: - resource: title: PostgreSQL documentation - link: https://www.postgresql.org/ + link: https://www.postgresql.org/docs/current/ type: documentation - resource: - title: "PostgreSQL on ARM: default page size matters" - link: https://dev.to/aws-heroes/postgresql-on-arm-default-page-size-matters-2n7a - type: blog + title: PostgreSQL resource consumption settings + link: https://www.postgresql.org/docs/current/runtime-config-resource.html + type: documentation + - resource: + title: PostgreSQL write-ahead log settings + link: https://www.postgresql.org/docs/current/runtime-config-wal.html + type: documentation + - resource: + title: Linux HugeTLBpage documentation + link: https://docs.kernel.org/admin-guide/mm/hugetlbpage.html + type: documentation + - resource: + title: Migrating applications to Arm servers + link: /learning-paths/servers-and-cloud-computing/migration/ + type: learning-path ### FIXED, DO NOT MODIFY # ================================================================================ @@ -55,4 +71,3 @@ weight: 1 # _index.md always has weight of 1 to order corr layout: "learningpathall" # All files under learning paths have this same wrapper learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. --- - diff --git a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/before_and_after.md b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/before_and_after.md index 1508253a07..e2145f330c 100644 --- a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/before_and_after.md +++ b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/before_and_after.md @@ -1,17 +1,34 @@ --- -title: "Before and after tuning PostgreSQL" +title: "Measure the impact of tuning PostgreSQL" +description: Learn how to approach PostgreSQL performance tuning as a measurement-driven process and compare baseline results with tuned results. weight: 2 layout: "learningpathall" --- -## About database performance tuning +## About performance tuning -Deployment configurations and the profile of SQL requests made by clients will differ based on the use case. This means there is no one size fits all set of tuning parameters for `PostgreSQL`. Use the information in this learning path as guidance to help you tune `PostgreSQL` for your use case. Make sure to test any changes made to tuning parameters. +Performance tuning is most useful when you treat it as a measurement process, not a fixed checklist. You can tune by changing one parameter at a time, running a designed experiment, comparing profiles, or using automation and AI-assisted tools to explore a larger configuration space. -## Importance of tuning +There isn't a universal set of tuning parameters that works best for every PostgreSQL deployment. The right settings depend on the workload, data size, memory capacity, storage performance, network behavior, PostgreSQL version, system architecture, operating system, and other application-specific factors. -Application tuning allows you to gain performance without scaling your deployment up (bigger machines) or out (more machines). You have the option to use the gained performance or trade it for cost savings by reducing the total compute resources provisioned. Below is a graph that shows the difference performance tuning on `PostgreSQL` can make. +Whatever method you use, keep the measurements repeatable. Record the system configuration, workload, software versions, and tuning parameters so you can identify which changes improved performance and which changes had little effect. -![Before and after Tuning](beforeandafter.png) +## Why tune PostgreSQL -Requirements vary based on the use case. In the example shown above, the AWS c7g.8xlarge instance could be down sized to a c7g.4xlarge to gain cost savings. +PostgreSQL performance can be limited by memory use, storage I/O, connection handling, write-ahead logging, concurrency, or synchronization overhead. By tuning PostgreSQL, you can use the available compute, memory, and storage resources more efficiently. + +Improved performance can give you higher throughput, lower latency, or better cost efficiency. A tuned configuration can increase capacity on the same system or help you meet the same performance target with fewer compute resources. + +## Example benchmark result + +The following example shows PostgreSQL throughput with HammerDB TPROC-C before and after tuning on an Arm Neoverse V3 system. The throughput metric, NOPM, stands for New Orders Per Minute. The bars show normalized NOPM, with the out-of-box configuration set to `1.00`. + +![Bar chart comparing PostgreSQL HammerDB TPROC-C normalized NOPM throughput before and after tuning on an Arm Neoverse V3 system. At 128 clients, the out-of-box configuration is 1.00 and the tuned configuration is 4.43.#center](beforeandafter.png "PostgreSQL HammerDB TPROC-C throughput before and after tuning") + +This benchmark result is an example, not a guaranteed improvement for every workload. Your results depend on the PostgreSQL version, database size, storage device, memory capacity, client concurrency, and SQL statements used in the test. Record those details with your own results so you can interpret and reproduce the comparison. + +## What you've learned and what's next + +You've learned why PostgreSQL tuning benefits from a measurement-driven approach and reviewed an example of throughput improvement after tuning. + +Next, you'll examine the system-level settings that can affect PostgreSQL performance. diff --git a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/beforeandafter.png b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/beforeandafter.png index 6d22bb7fe8..db01b90d9e 100644 Binary files a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/beforeandafter.png and b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/beforeandafter.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/kernel_comp_lib.md b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/kernel_comp_lib.md index 21c6db977c..6167c2e874 100644 --- a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/kernel_comp_lib.md +++ b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/kernel_comp_lib.md @@ -1,89 +1,105 @@ --- -# User change -title: "System, Kernel, compiler, and Libraries" +title: Tune storage, kernel, compiler, and library settings for performance +description: Learn how storage, kernel page size, Linux huge pages, compiler settings, and OpenSSL choices can affect PostgreSQL performance on Arm-based systems. -weight: 3 # 1 is first, 2 is second, etc. - -# Do not modify these elements +weight: 3 layout: "learningpathall" --- -## Storage technology and file system format - -The underlying storage technology and the file system format can impact performance significantly. In general, locally attached SSD storage will perform best. However, network based storage systems can perform well. As always, performance is dependent on the request profile coming from clients. You should spend some time studying and experimenting with different storage technologies and configuration options. +## Tune the system around PostgreSQL -Aside from the storage technology, it is also worth testing different file system formats with `PostgreSQL`. The `xfs` file system is a good starting point. The `ext4` file system is another good alternative. +PostgreSQL configuration is only one part of performance tuning. Operating system settings, storage behavior, kernel memory management, compiler choices, and library versions can also affect throughput and latency. Check these system-level areas before or alongside PostgreSQL-specific tuning. -## Kernel configuration +### Storage technology and file systems -`PostgreSQL` can benefit from adjustments to kernel parameters. Below is a list of kernel related settings that can have a positive impact on performance. +Storage technology and file system choices can affect performance. Locally attached SSD storage often provides the lowest latency, but network-based storage can also perform well. Test the storage technologies and configuration options available in your environment. -### Linux-PAM limits +The file system used with PostgreSQL can also affect performance. The `xfs` file system is a useful starting point, and `ext4` is another common option. For production systems, use storage volumes dedicated to the database instead of sharing them with the operating system or other applications. -Linux-PAM limits can be changed in the `/etc/security/limits.conf` file, or by using the `ulimit` command. +### Kernel resource limits -If you want more information about how to display and modify parameters check the documentation of the `ulimit` command. +PostgreSQL uses operating system resources such as file descriptors, processes, and locked memory. Review the limits for the account that runs PostgreSQL, especially when you increase connection count or shared memory use. -To display all limits: -```bash -ulimit -a -``` - -An explanation about why these Linux-PAM settings can help performance is found in the [PostgreSQL documentation](https://www.postgresql.org/docs/current/kernel-resources.html). - -Some of the key settings that can affect performance are: +The limits most likely to affect a PostgreSQL deployment are: - Maximum number of open file descriptors (`ulimit -n`) - Maximum number of processes (`ulimit -u`) - Maximum data segment size (`ulimit -d`) - Maximum locked-in-memory address space (`ulimit -l`) +To display the current limits, run: -### Linux virtual memory subsystem - -Making changes to the Linux Virtual Memory subsystem can also improve performance. +```bash +ulimit -a +``` -These settings can be changed in the `/etc/sysctl.conf` file, or by using the `sysctl` command. +For more information, see the PostgreSQL documentation about [kernel resource limits](https://www.postgresql.org/docs/current/kernel-resources.html). -If you want more information about how to display and modify virtual memory parameters check the documentation of the `sysctl` command. +### Kernel memory configuration -Documentation on each of these parameters can be found in the [admin-guide for sysctl in the Linux source code](https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/sysctl/vm.rst). +You can change Linux virtual memory settings by using the `sysctl` command or by adding configuration files under `/etc/sysctl.d/`. -To list all kernel parameters available: +To list the available sysctl parameters, run: ```bash sudo sysctl -a ``` -### Overcommit memory +For details about virtual memory parameters, see the Linux [sysctl documentation](https://docs.kernel.org/admin-guide/sysctl/vm.html). -The overcommit policy is set via the sysctl `vm.overcommit_memory' setting. +#### Overcommit memory -The recommended setting for `vm.overcommit_memory` is 2 according to the [PostgreSQL documentation](https://www.postgresql.org/docs/15/kernel-resources.html). +The `vm.overcommit_memory` setting controls Linux memory-overcommit behavior. PostgreSQL recommends a value of [`2`](https://www.postgresql.org/docs/current/kernel-resources.html) to reduce the risk that the kernel terminates the PostgreSQL server when memory is scarce. With a value of `2`, Linux uses strict overcommit accounting and denies memory allocations that would exceed its commit limit. -To set the overcommit_memory parameter to 2 temporarily, run the following command: +To set the value temporarily, run: -```console +```bash sudo sysctl -w vm.overcommit_memory=2 ``` -To make the change permanent: + +To make the setting persistent, create a file under `/etc/sysctl.d/` and reload the settings: ```bash -sudo sh -c 'echo "vm.overcommit_memory=2" >> /etc/sysctl.conf' +echo "vm.overcommit_memory=2" | sudo tee /etc/sysctl.d/99-postgresql-memory.conf +sudo sysctl --system ``` -This tells Linux to never over commit memory. Setting `vm.overcommit_memory` to 2 is to avoid a situation where the kernel might terminate the `PostgreSQL` postmaster when memory is scarce. + +#### Consider 64 KB kernel pages on Arm + +Some Arm Linux distributions provide kernels built with `64 KB` base pages instead of `4 KB` base pages. This is a kernel selection or build-time choice, not a `sysctl` setting that you can change on a running system. + +A `64 KB` base page can improve some memory-intensive workloads because each base page maps more memory. This can reduce Memory Management Unit (MMU) translation overhead, reduce page-table walk depth, and relieve Translation Lookaside Buffer (TLB) pressure, including instruction-side TLB pressure in the CPU front end. + +The tradeoff is memory efficiency. Larger base pages can increase internal fragmentation for workloads with many small or sparsely touched mappings, which can reduce the effective memory available to PostgreSQL and the operating system. + +The benefit is workload dependent, so compare a `4 KB` kernel and a `64 KB` kernel with the same PostgreSQL and storage configuration. + +Linux uses several page-table levels. A Page Middle Directory (PMD) entry can map one contiguous block of memory directly, without individual page-table entries for the base pages inside that block. The amount of memory mapped by a PMD entry depends on the base page size and page-table geometry. + +On Arm, this means a PMD-level huge page commonly maps `2 MiB` with `4 KB` base pages, or `512 MiB` with `64 KB` base pages: + +| Kernel base page size | PMD-level huge page size | +|-----------------------|--------------------------| +| `4 KB` | `2 MiB` | +| `64 KB` | `512 MiB` | + +Other huge page sizes can be available, but use the value reported by `/proc/meminfo` when configuring `vm.nr_hugepages`. + +For more information, see the Linux kernel documentation for [Memory Layout on AArch64 Linux](https://docs.kernel.org/next/arm64/memory.html) and [HugeTLBpage on ARM64](https://docs.kernel.org/next/arm64/hugetlbpage.html). ### Huge memory pages -`PostgreSQL` benefits from using huge memory pages. Huge pages reduce how often virtual memory pages are mapped to physical memory. +PostgreSQL can benefit from explicit huge memory pages because they use smaller page tables and can reduce CPU time spent on memory management. They have a similar performance goal to `64 KB` base pages: map more memory per translation entry and reduce page-table walks and TLB pressure. -To see the current memory page configuration, run the following command on the host: +The tradeoff is also similar. Huge pages reserve memory in larger chunks, so over-allocating them can reduce memory available for other uses. The performance impact depends on your workload and system configuration. + +To see the current huge page configuration, run: ```bash -cat /proc/meminfo | grep ^Huge +grep '^Huge' /proc/meminfo ``` -The output should be similar to: +The output is similar to: ```output HugePages_Total: 0 @@ -94,61 +110,72 @@ Hugepagesize: 2048 kB Hugetlb: 0 kB ``` -Huge pages are not being used if `HugePages_Total` is 0 (this is the default). +The `Hugepagesize` value depends on the kernel base page size and platform configuration. Use the value reported by `/proc/meminfo` when calculating `vm.nr_hugepages`. -Also note that `Hugepagesize` is 2MB which is the typical default for huge pages on Linux. +The `vm.nr_hugepages` parameter sets the number of huge pages that the kernel makes available to applications. The total memory reserved for huge pages is `vm.nr_hugepages` multiplied by `Hugepagesize`. -You can modify the huge page values. +For example, if you want `1 GiB` of huge page space and `Hugepagesize` is `2 MiB`, set `vm.nr_hugepages` to `512`: -The setting that enables huge pages is shown below: +```bash +sudo sysctl -w vm.nr_hugepages=512 +``` -```output -vm.nr_hugepages +To make the setting persistent: + +```bash +echo "vm.nr_hugepages=512" | sudo tee /etc/sysctl.d/99-postgresql-hugepages.conf +sudo sysctl --system ``` -This parameter sets the number of huge pages you want the kernel to make available to applications. +#### Configure PostgreSQL to use huge pages -The total amount of memory that will be used for huge pages will be this number (defaulted to 0) times the `Hugepagesize`. +The PostgreSQL [`huge_pages`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-HUGE-PAGES) setting controls whether PostgreSQL requests huge pages for its main shared memory area. The default, `try`, attempts to use them and falls back to regular pages if allocation fails. -As an example, if you want a total of 1GB of huge page space, then you should set `vm.nr_hugepages` to 500 (500x2MB=1GB). +Set `huge_pages=on` when you want PostgreSQL to fail at startup instead of silently using regular pages: -```bash -sudo sysctl -w vm.nr_hugepages=500 +```ini +huge_pages = on ``` -To make the change permanent: +Start with `shared_buffers`, which is usually the largest part of PostgreSQL shared memory. Allocate enough huge pages to cover that value plus a small margin for other shared-memory allocations. The next section explains how to select a `shared_buffers` value. -```bash -sudo sh -c 'echo "vm.nr_hugepages=500" >> /etc/sysctl.conf' -``` +{{% notice Important %}} +After restarting PostgreSQL, confirm the result with `SHOW huge_pages_status;` and check the PostgreSQL server log. With `huge_pages=try`, PostgreSQL can fall back to regular pages when the huge page pool is too small. With `huge_pages=on`, it does not start until enough huge pages are available. +{{% /notice %}} + +PostgreSQL explicitly configured huge pages are different from Transparent Huge Pages (THP). PostgreSQL documentation discourages THP because it has caused performance degradation on some Linux versions. -### Selecting the number of huge pages to use +For more information about Linux huge page configuration, see the [Linux HugeTLBpage documentation](https://docs.kernel.org/admin-guide/mm/hugetlbpage.html). -You should set `vm.nr_hugepages` to a value that gives a total huge page space slightly bigger than the `PostgreSQL` shared buffer size (discussed later). +### Page-cache writeback settings -Make it slightly larger than the shared buffer because `PostgreSQL` will use additional memory for things like connection management. +The Linux page cache affects PostgreSQL write behavior. The `vm.dirty_background_ratio` and `vm.dirty_ratio` settings control when the kernel begins background writeback and when application writes can be delayed while flushing catches up. -More information on the different parameters that affect the configuration of huge pages can be found in the [admin-guide for hugetlbpage in the Linux source code](https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/mm/hugetlbpage.rst). +The following values are a useful starting point for a write-heavy workload. They are not required for every PostgreSQL deployment: + +```ini +vm.dirty_background_ratio = 5 +vm.dirty_ratio = 20 +``` -### Page cache parameters +`vm.dirty_background_ratio` controls the percentage of page cache that must be dirty before the kernel starts background writeback. Lowering it can spread storage writes over time and reduce long write stalls for some write-heavy workloads. -`PostgreSQL` writes data to files like any Linux process does. The behavior of the page cache can affect performance. There are two sysctl that parameters control how often the kernel flushes the page cache data to disk. +`vm.dirty_ratio` controls the threshold at which processes that generate writes can be delayed while the kernel catches up. Keep it higher than `vm.dirty_background_ratio`. A higher value gives background writeback more time to catch up with bursty writes, but it also allows more dirty data to accumulate and can create longer stalls when the kernel finally needs to flush it. -- `vm.dirty_background_ratio=5` -- `vm.dirty_ratio=20` +These settings depend on memory capacity, storage latency, and write behavior, so change them only when measurement shows that writeback is a bottleneck. -The `vm.dirty_background_ratio` sets the percentage of the page cache that needs to be dirty in order for a flush to disk to start in the background. +### Compiler considerations -Setting this value to lower than the default (typically 10) helps write heavy workloads. This is because by lowering this threshold, you are spreading writes to storage over time. This reduces the probability of saturating storage performance. Setting this value to 5 can improve performance. +If you build PostgreSQL from source, the compiler version and optimization flags can affect performance. Use a recent version of GCC, and consider flags such as `-mcpu` and `-flto` for additional optimization. These flags are explained in the [Migrating C/C++ applications](/learning-paths/servers-and-cloud-computing/migration/c/) section of the [Migrating applications to Arm servers](/learning-paths/servers-and-cloud-computing/migration/) Learning Path. -The `vm.dirty_ratio` sets the percentage of the page cache that needs to be dirty in order for threads that are writing to storage to be paused to allow flushing to catch up. +### OpenSSL considerations -This should be set higher than `vm.dirty_background_ratio`. The OS default is typically around 20 which is usually good. In some cases it may be beneficial to set this value to be higher, it gives the background flusher (controlled by `vm.dirty_background_ratio`) more time to catch up if disk writes are very bursty. In general, you can leave this to the OS default, but it may be worth experimenting with higher values on specific workload profiles. +PostgreSQL relies on [OpenSSL](https://www.openssl.org/) for cryptographic operations. The OpenSSL version used with PostgreSQL, and the compiler version and switches used to build it, can affect performance. The default OpenSSL version provided by your Linux distribution is typically sufficient. -## Compiler Considerations +If you build PostgreSQL from source, testing a newer OpenSSL release might help workloads that spend significant time in cryptographic operations. -The easiest way to gain performance is to use the latest version of GCC. Aside from that, the flags `-mcpu` and `-flto` can be used to potentially gain additional performance. Usage of these flags is explained in the [Migrating C/C++ applications](/learning-paths/servers-and-cloud-computing/migration/c/) section of the [Migrating applications to Arm servers](/learning-paths/servers-and-cloud-computing/migration/) learning path. +## What you've learned and what's next -## OpenSSL Considerations +You've explored system-level choices related to storage, memory pages, kernel settings, compiler flags, and OpenSSL that can affect PostgreSQL performance on Arm. -PostgreSQL relies on [OpenSSL](https://www.openssl.org/) for cryptographic operations. Thus, the version of OpenSSL used with PostgreSQL (and the GCC version and switches used to compile it) can impact performance. Typically using the Linux distribution default version of OpenSSL is sufficient so long as the Linux distribution used isn't very old. That said, it is possible to use newer versions of OpenSSL which could yield performance improvements. We leave this as an exercise to the reader. +Next, you'll learn about PostgreSQL parameters that you can tune for performance. diff --git a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/testing.md b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/testing.md index ed6ab20ee2..137e1ad3ce 100644 --- a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/testing.md +++ b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/testing.md @@ -1,6 +1,7 @@ --- # User change -title: "Testing PostgreSQL Tunings" +title: Test PostgreSQL tuning with HammerDB +description: Learn how to use HammerDB TPROC-C as a repeatable PostgreSQL workload when evaluating tuning changes. weight: 5 # 1 is first, 2 is second, etc. @@ -8,23 +9,25 @@ weight: 5 # 1 is first, 2 is second, etc. layout: "learningpathall" --- -## Testing PostgreSQL Optimizations +## Measure PostgreSQL tuning changes -Skip this section if you already have a performance test methodology for your `PostgreSQL` deployment. +Use this section if you need a repeatable way to test PostgreSQL changes. If you already have a performance test methodology for your PostgreSQL deployment, use that instead. -This section presents a method for testing `PostgreSQL` using HammerDB TPROC-C. This is useful information if you do not already have an established test methodology. To understand the impact of tuning on specific use cases and deployments, it is recommended that you develop a performance test strategy that reflects your use cases. +HammerDB TPROC-C is one option for comparing a baseline with a tuned configuration. It does not replace a performance test that reflects your application's SQL statements, data size, client concurrency, and latency requirements. ## Before you begin -You will need a physical machine or cloud node with [`PostgreSQL`](https://www.postgresql.org/) installed and configured. +You need a physical system or cloud instance with [PostgreSQL](https://www.postgresql.org/) installed and configured, plus enough storage and memory for the test database. ## About HammerDB -[HammerDB](https://www.hammerdb.com/) is a database benchmarking tool. It can test `PostgreSQL`, MySQL, MariaDB, Db2, & SQL Server. It offers two benchmark types; TPROC-C and TPROC-H. TPROC-C models the TPC-C benchmark, while TPROC-H models the TPC-H benchmark. We will only discuss TPROC-C which is an [Online Transactional Processing](https://www.hammerdb.com/docs/ch03s01.html) style of workload. TPROC-C simulates a company that processes customer orders and manages the warehouses which contain the products the company sells. +[HammerDB](https://www.hammerdb.com/) is a database performance test tool that supports PostgreSQL, MySQL, MariaDB, Db2, and SQL Server. It provides TPROC-C and TPROC-H workloads. TPROC-C models an [online transaction processing](https://www.hammerdb.com/docs/ch03s01.html) workload based on TPC-C, while TPROC-H models a decision-support workload based on TPC-H. -You can use this topic as a primer on how to run HammerDB. +TPROC-C and TPROC-H results are not official TPC-C or TPC-H benchmark results. If you need a TPC-C or TPC-H result, run the corresponding official TPC benchmark and follow its rules. -To learn more about HammerDB refer to the [documentation](https://www.hammerdb.com/document.html). +This Learning Path uses TPROC-C, which simulates a company processing customer orders and managing warehouses containing the products it sells. + +For more information, see the [HammerDB documentation](https://www.hammerdb.com/document.html). ## Installing HammerDB @@ -32,24 +35,24 @@ Follow the [installation instructions](https://www.hammerdb.com/docs/ch01.html) ## Running tests using the GUI -You can use TCL scripts to run HammerDB in an automated fashion. +You can use TCL scripts to run HammerDB automatically. -It is also possible to run a HammerDB test using the GUI. +You can also run a HammerDB test with the GUI. If you are interested in running HammerDB tests through a GUI, read the [Quick Start](https://www.hammerdb.com/docs/ch02.html) guide. ## Running tests using TCL scripts -The [CLI and command documentation](https://www.hammerdb.com/docs/ch09s08.html) contains information on how to setup and run TCL script based tests. +The [CLI and command documentation](https://www.hammerdb.com/docs/ch09s08.html) explains how to set up and run TCL-script-based tests. The example in the documentation is for MySQL. The example below is for testing `PostgreSQL`. -The script creates a test database with 1000 warehouses. +The script creates a test database with 1000 warehouses. -The user count is set to 128 to help populate the database faster. Use stored procedures (pg_storedprocs) instead of functions (default) because this is [recommended for PostgrSQL version v11 and above](https://www.hammerdb.com/docs/ch04s03.html#d0e1734). +The user count is set to 128 to populate the database faster. It uses stored procedures (`pg_storedprocs`) instead of functions because HammerDB recommends this option for PostgreSQL version 11 and later. -1. Use a text editor to save the code below in a file named `test.tcl` +1. Use a text editor to save the code below in a file named `build.tcl` Replace `postgresql_host_ip` with your IP address. @@ -69,14 +72,14 @@ buildschema 2. Execute the script: ```console -hammerdbcli auto test.tcl +hammerdbcli auto build.tcl ``` -The next script will run 6 iterations of the the TPROC-C test. +The next script runs six TPROC-C test iterations. -It will run the test with 8, 16, 32, 64, 128, and 256 users. Each test iteration will ramp for 3 minutes, then run the actual test for 15 minutes. +It runs with 8, 16, 32, 64, 128, and 256 users. Each iteration ramps for three minutes and then runs for 15 minutes. -3. Use a text editor to save the code below in a file named `test2.tcl` +3. Use a text editor to save the code below in a file named `test.tcl` Replace `postgresql_host_ip` with your IP address. @@ -116,16 +119,21 @@ puts "Tests complete" 4. Execute the script: ```console -hammerdbcli auto test2.tcl +hammerdbcli auto test.tcl ``` -The above scripts can be used as a starting point for testing. You can adjust the scripts to create scenarios that are closer to your use case. +Use these scripts as a starting point. Adjust the data size, user count, ramp time, duration, and SQL execution mode to create a workload closer to your use case. ## Running tests against a pool of PostgreSQL nodes -You can run tests against a `PostgreSQL` cluster. +You can run tests against a PostgreSQL cluster. + +The cluster can consist of a primary read/write node and one or more read-only standby nodes. + +See the [HammerDB connection pool documentation](https://www.hammerdb.com/docs/ch04s06.html#d0e2280) for cluster testing options. -The cluster could consist of a primary (RW) node and two secondary (RO) nodes. +## What you've learned -Look into using the connect pool feature of HammerDB in the [documentation](https://www.hammerdb.com/docs/ch04s06.html#d0e2280). +You've learned how to use HammerDB TPROC-C as a repeatable PostgreSQL test workload. +Use the same workload, system configuration, and measurement process before and after each tuning change so you can attribute the result to the change you made. diff --git a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/tuning.md b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/tuning.md index 817c7d5804..8865385efb 100644 --- a/content/learning-paths/servers-and-cloud-computing/postgresql_tune/tuning.md +++ b/content/learning-paths/servers-and-cloud-computing/postgresql_tune/tuning.md @@ -1,120 +1,138 @@ --- -# User change -title: "Tuning PostgreSQL" +title: Tune PostgreSQL configuration for performance +description: Learn how to tune PostgreSQL parameters for connections, memory, write-ahead logging, query planning, I/O, and parallel execution. -weight: 4 # 1 is first, 2 is second, etc. - -# Do not modify these elements +weight: 4 layout: "learningpathall" --- -## PostgreSQL configuration +## Choose PostgreSQL parameters for tuning -There are different ways to set configuration parameters for `PostgreSQL`. +You can set PostgreSQL parameters in `postgresql.conf`, with `ALTER SYSTEM`, or for a database, role, or session when the parameter supports that scope. For persistent tuning, use a configuration method that can be reviewed, version controlled, and applied consistently when PostgreSQL restarts. See the PostgreSQL [configuration settings documentation](https://www.postgresql.org/docs/current/config-setting.html) for details. -This is discussed in the [Setting Parameters documentation](https://www.postgresql.org/docs/current/config-setting.html). +{{% notice Note %}} +Leave most PostgreSQL settings at their defaults and change them only when a workload requirement, test result, profile, or observed bottleneck supports the change. +{{% /notice %}} -The configurations below can be directly pasted into a `PostgreSQL` configuration file. +### Connections -### Connections and prepared transactions +The [`max_connections`](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-MAX-CONNECTIONS) parameter sets the maximum number of concurrent connections PostgreSQL accepts. It does not directly improve query performance. -```output -max_connections = 1000 # Default 100 -max_prepared_transactions = 1000 # Default 0 +```ini +# Example high connection limit. Size this for your expected peak connection count. +max_connections = 1000 ``` -`max_connections` doesn't impact performance of queries directly, but if a high client connection count is expected or required, it's a good idea to raise this in order to not reject request from clients. If connections start getting reject, this will affect overall performance. - -Keep in mind that more client connections means more resources will be consumed (especially memory). Setting this to something higher is completely dependent on use case and requirements. +Each PostgreSQL connection uses memory and operating system resources. Size this value for expected peak demand, and review the kernel resource limits when you increase it. When an application needs many client connections, use a connection pooler such as [PgBouncer](https://www.pgbouncer.org/) to reuse a smaller number of PostgreSQL server connections. This can reduce memory use and process overhead, but it does not make individual queries faster. -`max_prepared_transactions` is 0 by default. +### Memory configuration -This means that stored procedures and functions cannot be used out of the box. It must be enabled by setting `max_prepared_transactions` to a value greater than 0. If this is set to a number larger than 0, a good number to start with would be at least as large as `max_connections`. In a test or development environment, it doesn't hurt to set it to an even larger value(10000) to avoid errors. +#### Shared buffers -Using procedures and functions can greatly improve performance. +The [`shared_buffers`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-SHARED-BUFFERS) parameter controls the shared memory cache that PostgreSQL uses for table and index data. -### Memory related configuration +For a dedicated database server with at least `1 GB` of memory, set `shared_buffers` to `25%` to `40%` of system memory as a starting range. PostgreSQL also relies on the operating system page cache, so values above `40%` are unlikely to help. Larger `shared_buffers` settings often need a corresponding increase in `max_wal_size`. -```output -huge_pages = on # default is try -shared_buffers = <25%-40% system memory> # Default is 128MB -work_mem = 32MB # default is 4MB -maintenance_work_mem = 2GB # Default is 64MB +```ini +# Example for a dedicated server with 128 GiB of memory. +shared_buffers = 32GB ``` -Turning on `huge_pages` is not required because the default is `try`. +#### Query and maintenance memory -However, you can explicitly set it to `on` because errors will be produced if huge pages are not enabled in Linux. +The [`work_mem`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-WORK-MEM) parameter is the base memory limit for each sort or hash operation before PostgreSQL writes temporary files. A single complex query can use multiple operations, and many sessions can run them concurrently. Treat increases as a measured change, not a global performance setting. -`shared_buffers` is one of the most important configuration parameters that can be set. It determines how much memory can be used to store indexes and table data. It's a cache that improves read/write latency and relieves pressure on storage. The [PostgreSQL documentation](https://www.postgresql.org/docs/15/runtime-config-resource.html) suggests this be set to 25% - 40% of total system memory. +```ini +# Example values. Test with realistic query concurrency. +work_mem = 32MB +maintenance_work_mem = 2GB +``` -`work_mem` is memory used when queries are being processed. Raising this significantly from the default value can help performance. +The [`maintenance_work_mem`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-MAINTENANCE-WORK-MEM) parameter controls memory for maintenance operations such as `VACUUM` and `CREATE INDEX`. A larger value can improve maintenance performance, especially for databases with frequent updates or deletes, but account for concurrent autovacuum workers. Set [`autovacuum_work_mem`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-AUTOVACUUM-WORK-MEM) separately if you need to limit each worker. -`maintenance_work_mem` is memory used for operations like VACUUM. In a scenario where data is removed often, raising this can help performance. +#### Huge pages -### Processing and process count +The [`huge_pages`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-HUGE-PAGES) setting requests huge pages for PostgreSQL shared memory. The default value, `try`, falls back to regular pages if allocation fails. Set it to `on` when you want PostgreSQL to fail at startup instead of using regular pages. -```output -deadlock_timeout = 10s # Default is 1s -max_worker_processes = # Default is 8 +```ini +huge_pages = on ``` -`deadlock_timeout` sets a polling interval for checking locks. The [documentation](https://www.postgresql.org/docs/15/runtime-config-locks.html) states that this check is expensive from a CPU cycles standpoint, and that the default of 1s is probably the smallest that should be used. Consider raising this timeout much higher to save some CPU cycles. +Huge pages must also be configured at the Linux kernel level. For setup instructions and sizing guidance, see the [system, kernel, compiler, and library settings](/learning-paths/servers-and-cloud-computing/postgresql_tune/kernel_comp_lib/) section of this Learning Path. + +### Write-ahead log and durability configuration -`max_worker_processes` is a key parameter for performance. It's the number of total background processes allowed. A good starting point is to set this to the number of cores present on the PostgreSQL node. +PostgreSQL uses the write-ahead log (WAL) to protect committed data and support crash recovery. The following settings affect checkpoint behavior and transaction durability. -### Write Ahead Log (WAL) configuration +#### Checkpoint and WAL capacity -```output -synchronous_commit = off # Default is on -max_wal_size = 20GB # Default is 1GB -min_wal_size = 1GB # Default is 80MB +The [`max_wal_size`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-MAX-WAL-SIZE) parameter is a soft limit on WAL size during automatic checkpoints. Increasing it can reduce checkpoint frequency and smooth write I/O for write-heavy workloads. + +Under sustained high load, WAL can fill quickly enough to trigger frequent checkpoints and checkpoint warnings. Increasing `max_wal_size` gives PostgreSQL more WAL space before a checkpoint is required. The following value is an example, not a universal recommendation: + +```ini +# Example for a write-heavy workload. Test against storage capacity and recovery requirements. +min_wal_size = 1GB +max_wal_size = 20GB ``` -If `synchronous_commit` is on (default), it tells the WAL processor to wait until more of the log is applied before reporting success to clients. Turning this off means that the PostgreSQL instance will report success to clients sooner. This will result in a performance improvement. It is safe to turn this off in most cases, but keep in mind that it will increase the risk of losing transactions if there is a crash. However, it will not increase the risk of data corruption. +The [`min_wal_size`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-MIN-WAL-SIZE) parameter keeps a minimum amount of WAL available for recycling. Increasing it can help absorb bursts of WAL activity and reduce the need to create new WAL files during a spike. -In high load scenarios, check pointing can happen very often. In fact, in testing with HammerDB, there may be so much check pointing that PostgreSQL reports warnings. One way to reduce how often check pointing occurs is to increase the `max_wal_size` of the WAL log. Setting it to 20GB can make the excessive check pointing warnings go away. `min_wal_size` can also be increased to help absorb spikes in WAL log usage under high load. +Larger WAL settings use more storage and can increase crash recovery time. PostgreSQL logs a checkpoint warning when WAL-triggered checkpoints occur more often than `checkpoint_warning`, which can indicate that `max_wal_size` is too small for the workload. -### Planner/Optimizer configuration +#### Transaction durability settings -The optimizer (also called planner) is responsible for taking statistics about the execution of previous queries, and using that information to figure out what is the fastest way to process new queries. Some of these statistics include shared buffer hit/miss rate, execution time of sequential scans, and execution time of index scans. Below are some parameters that affect the optimizer. +Use the following setting when transaction durability is required. It is the PostgreSQL default: -```output -effective_cache_size = <80% of system memory> # Default is 4GB -random_page_cost = 1.1 # Default is 4.0 +```ini +# Keep the default when transaction durability is required. +synchronous_commit = on ``` -One key piece of information that a `PostgreSQL` instance will not have access to is the size of the OS page cache. `effective_cache_size` provides a way to inform `PostgreSQL` of the page cache size. Assuming the host is dedicated to running `PostgreSQL`, a good starting value is to set this to about 80% of total system memory. The value of this parameter should roughly be the shared buffer size and the OS page cache size combined. Use a tool like `free` while `PostgreSQL` is running to understand how much memory is being used for the OS page cache. This can help further refine the value from the suggested 80%. Also note that this parameter does not affect memory allocations. +The [`synchronous_commit`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) parameter controls whether PostgreSQL waits for WAL records to reach durable storage before reporting commit success to the client. -**How does `effective_cache_size` affect the optimizer and help performance?** +When `synchronous_commit` is `on`, PostgreSQL prioritizes transaction durability over peak write throughput. Setting `synchronous_commit=off` lets PostgreSQL acknowledge commits before the WAL data is flushed to durable storage. This can improve parallel transaction execution and write throughput, especially for write-heavy workloads, but it increases the time window where acknowledged transactions exist only in memory or operating system cache. -When data is loaded into the PostgreSQL shared buffer, the same data may also be present in the page cache. It is also possible that data that isn't in the shared buffer is present in the page cache. This second case creates a scenario where tuning `effective_cache_size` can help improve performance. +{{% notice Warning %}} +Changing `synchronous_commit` to `off` trades away transaction durability, which is part of ACID compliance. A power failure, operating system crash, or unexpected PostgreSQL exit can lose recently acknowledged transactions, though this setting does not cause database corruption. -Sometimes `PostgreSQL` needs to read data that is not in the shared buffer, but it is in the page cache. From the perspective of `PostgreSQL`, there will be a shared buffer miss when it tries to read the data. When this happens, the `PostgreSQL` instance will assume that reading this data will be slow because it will come from disk. It assumes the data will come from disk because `PostgreSQL` has no way to know if the data is in the page cache. However, if it turns out that the data is present in the page cache, the data will be read faster than if it was read from disk. +Change this setting only when your application and recovery process can tolerate potential transaction loss. +{{% /notice %}} -If the page cache is large, it is far more likely that the data will in fact be in the page cache. `effective_cache_size` gives us a way to tell `PostgreSQL` that there is a lot of system memory used for the page cache, and thus, even if there is a shared buffer miss, it's very possible it will be "saved" by the page cache. The bigger `effective_cache_size` is set, the more likely `PostgreSQL` is to favor doing something like trying to read an index that is not present in the shared buffer, over doing a sequential scan of a table that is in the shared buffer. Even with the overhead of moving the index to the shared buffer from the page cache, the index scan will likely be faster than a sequential scan from the shared buffer. On average, this should improve performance. +### Query planner configuration -`random_page_cost` has a similar effect as `effective_cache_size`. +The PostgreSQL query planner selects an execution plan using table and index statistics, along with cost estimates for operations such as sequential scans and index scans. Use `EXPLAIN` and `EXPLAIN ANALYZE` with representative queries to understand a plan before changing planner-related settings. -`random_page_cost` tells the optimizer how much of a relative cost there is to accessing data from storage. The default of 4.0 is fairly conservative and is more appropriate for HDD based storage or when the shared buffer hit rate is below 90%. If the underlying storage technology is SSD, then it's best to reduce this number. Also, if the shared buffer hit rate is very high (90%+), it is also a good idea to reduce this number. The [documentation](https://www.postgresql.org/docs/15/runtime-config-query.html#GUC-RANDOM-PAGE-COST) suggests 1.1 for these cases. +#### Effective cache size -**How does `random_page_cost` affect the optimizer and help performance?** +The [`effective_cache_size`](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-EFFECTIVE-CACHE-SIZE) parameter tells the planner how much memory it can expect to be available for caching PostgreSQL data. Set it to an estimate of the total cache available to PostgreSQL, including `shared_buffers` and the operating system page cache. It does not allocate memory or reserve page cache. -The effect to the planner/optimizer is similar to that of `effective_cache_size`. Basically, a lower `random_page_cost` tells the optimizer to favor doing something like reading an index from disk over doing a sequential table scan from the shared buffer. Also, keep in mind that when `PostgreSQL` tries to access the disk, it might actually be accessing from the page cache which is faster. +For a dedicated PostgreSQL server, `80%` of total memory is a useful starting estimate. Use `free` while PostgreSQL is running to understand page-cache use, then refine the value for the memory available to PostgreSQL and the expected number of concurrent queries. -### Concurrency configuration +When the page cache is large, data that misses `shared_buffers` is more likely to be read from memory instead of storage. A larger `effective_cache_size` makes index scans look less expensive to the planner, while a smaller value makes sequential scans more likely. This can help the planner choose an index scan when the index is likely to be cached, rather than assuming the access requires a slow storage read. -Increasing parallelism uses available resources more efficiently. It's always a good idea to look at parameters related to parallel execution. +#### Random page cost -```output -max_parallel_workers = # Default is 8 -max_parallel_workers_per_gather = 4 # Default is 2 -max_parallel_maintenance_workers = 4 # Default is 2 -effective_io_concurrency = 300 # Default is 1 -``` +The [`random_page_cost`](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-RANDOM-PAGE-COST) parameter estimates the relative cost of a non-sequential page read. The default is `4.0`. Lowering it relative to `seq_page_cost` makes index scans more likely; raising it makes them less likely. + +The default can be a better fit for storage with a high random-read penalty, such as hard disks. On SSD-backed systems, or when a large part of the database is likely to be cached, a lower value such as `1.1` can be worth testing. Storage latency, cache behavior, and network latency all affect the right value, so compare the resulting plans and execution times with representative queries instead of applying a fixed value to every system. + +`random_page_cost` and `effective_cache_size` both influence how the planner values index scans, but from different directions. `effective_cache_size` estimates how likely the data is to be cached. `random_page_cost` estimates the penalty when PostgreSQL has to access a page that is not already in cache. + +### Parallel execution and I/O configuration + +PostgreSQL uses worker processes for parallel queries, maintenance, and extensions. The [`max_worker_processes`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-MAX-WORKER-PROCESSES) setting is the overall limit. The [`max_parallel_workers`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-MAX-PARALLEL-WORKERS) setting limits workers available to parallel operations, and must not exceed `max_worker_processes`. + +Set `max_worker_processes` and `max_parallel_workers` high enough to allow useful parallel work, but leave CPU capacity for connection handling, background processes, and other applications. A starting point for a dedicated test system is the number of available logical CPUs. PostgreSQL can launch fewer workers than the configured limits when capacity is unavailable. + +Use [`max_parallel_workers_per_gather`](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-MAX-PARALLEL-WORKERS-PER-GATHER) and [`max_parallel_maintenance_workers`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-MAX-PARALLEL-MAINTENANCE-WORKERS) to limit workers used by an individual query or maintenance operation. A value of `4` is a reasonable test starting point for each on a sufficiently large system. Increase the values only when suitable queries or maintenance operations have unused CPU capacity. More workers can increase coordination overhead or reduce performance when CPU, memory bandwidth, or storage is already saturated. + +The [`effective_io_concurrency`](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-EFFECTIVE-IO-CONCURRENCY) parameter controls how many storage I/O operations a PostgreSQL session expects to issue in parallel. A value in the low hundreds, such as `300`, can be a useful starting point for high-IOPS SSD or cloud storage. Higher values can help high-latency storage with enough I/O capacity, but unnecessarily high values can increase I/O latency. Its default has changed across PostgreSQL versions, so check the value on the version you deploy before overriding it. + +## What you've learned -`max_parallel_workers` selects how many parallel operations can occur. A good starting point for this parameter is to set this to match the number of cores in the system. +You've explored PostgreSQL parameters that affect connection handling, memory use, WAL behavior, query planning, and parallel execution. -Doubling `max_parallel_workers_per_gather` and `max_parallel_maintenance_workers` to 4 seems to provide the most benefit. +Use the guidance in this Learning Path to design measured tuning experiments for your PostgreSQL workload on Arm. -`effective_io_concurrency` affects how many parallel IO requests you can send to storage. Modern storage technologies tend to allow a large number of IOPS. Thus, setting this higher is advised. Also note, this parameter only affects bitmap heap scans. A bitmap heap scan is an "in between" method for processing a query. That is, while Index scans (and Index only scans) are typically the fastest way to access data, and sequential scans are typically the slowest way to access data. A bitmap heap scan is in between these extremes. +Next, use the HammerDB TPROC-C workflow to measure the effect of your tuning changes.