Skip to content

feat(postgres): expose PostgreSQL tunables as SHELLHUB_POSTGRES_* variables - #6892

Open
otavio wants to merge 1 commit into
perf/identity-resolution-statementsfrom
perf/postgres-tunables
Open

feat(postgres): expose PostgreSQL tunables as SHELLHUB_POSTGRES_* variables#6892
otavio wants to merge 1 commit into
perf/identity-resolution-statementsfrom
perf/postgres-tunables

Conversation

@otavio

@otavio otavio commented Aug 12, 2026

Copy link
Copy Markdown
Member

Top of the perf stack — based on perf/identity-resolution-statements (#6890), itself on
#6888#6886#6883. Merge those first; this then targets master on its own.

Every ShellHub deployment inherits the same untuned postgres:18.0: 128 MB of
shared_buffers regardless of the machine underneath
. On the largest managed instance that is
a 84.76% buffer cache hit ratio against a 5.9 GB database on a 15.6 GB host.

This PR does not tune anything. It ships the seam, with every default equal to the value the
deployment already runs with, so the rendered command line is unchanged. The values are set
per host — for the managed fleet, in shellhub-io/shellhub-managed-deploy#400, which derives them
from /proc/meminfo.

Why env vars and not a second Compose file

The obvious alternative is for a deployment to override command: in its own overlay. That
breaks quietly: Compose replaces command: rather than merging it, so an override has to
restate io_method, wal_compression and shared_preload_libraries — and silently drops
whatever this list grows next. #6883 and #6886 each added a flag here in the last week, so that
is not hypothetical.

bin/docker-compose already chains COMPOSE_ENV_FILES with .env.override last, so
interpolation is a seam both community users and managed hosts already have.

:- rather than - throughout: an override that sets a key to nothing must fall back, not
expand to -c shared_buffers=, which postgres refuses to start on.

What is exposed

default note
shared_buffers, max_wal_size, min_wal_size, max_connections 128MB, 1GB, 80MB, 100 written by initdb; this file is now their source
effective_cache_size, work_mem, maintenance_work_mem 4GB, 4MB, 64MB PG defaults
checkpoint_timeout, random_page_cost 5min, 4 PG defaults
effective_io_concurrency 16 PG18 raised this from 1; the default is deliberately not the old value
wal_compression lz4 as set by #6883, not PG's off

checkpoint_completion_target is not exposed: it has defaulted to 0.9 since PG14, so
setting it is a no-op. Verified against the running image rather than assumed.

Two Docker-side knobs come along because they bound the same thing:

  • shm_size — parallel-query segments are allocated from /dev/shm, which is 64 MB by
    default. shared_buffers is not (shared_memory_type is mmap), so this tracks work_mem
    and parallelism instead.
  • mem_limit — stays at today's unlimited (0, which Compose normalises away entirely), but
    is worth setting once shared_buffers grows on a host with no swap, where an OOM is a hard
    kill.

Testing

  • ./bin/docker-compose config renders and shlex-splits the folded scalar into correct argv, at
    defaults and under override.
  • An empty override falls back rather than emitting a bare -c shared_buffers=.
  • A throwaway postgres:18.0 booted with a fully tuned command line reports all twelve settings
    with source = command line — confirming they override the initdb-written postgresql.conf
    and pg_stat_statements still loads alongside.
  • Rendered with the exact environment a 15.6 GB managed host produces, end to end.

Refs shellhub-io/team#198.

@otavio
otavio requested a review from a team as a code owner August 12, 2026 19:17
@otavio
otavio force-pushed the perf/postgres-tunables branch from 5700314 to 307e51e Compare August 12, 2026 19:36
…iables

PostgreSQL runs on stock defaults on every deployment -- 128 MB of shared_buffers
regardless of the machine underneath. On the largest managed instance that is a
84.76% buffer cache hit ratio against a 5.9 GB database on a 15.6 GB host.

Sizing that per host needs a seam, and overriding `command:` from a second Compose
file is the wrong one: Compose replaces `command:` rather than merging it, so an
override has to restate io_method, wal_compression and shared_preload_libraries,
and silently drops whatever this list grows next.

Parameterise the flags instead. Every default is the value the deployment already
runs with, so the rendered command line is unchanged and this carries no behaviour
of its own: .env documents the knobs, and .env.override -- which bin/docker-compose
already loads last -- is where a host sets them.

Two Docker-side knobs come along because they bound the same thing. shm_size is
where parallel-query segments are allocated from, not shared_buffers, so it has to
track work_mem. mem_limit stays at today's unlimited, but is worth setting once
shared_buffers grows on a host with no swap.

Refs: shellhub-io/team#198
@otavio
otavio force-pushed the perf/identity-resolution-statements branch from ac00c50 to a5652ac Compare August 12, 2026 21:46
@otavio
otavio requested a review from a team as a code owner August 12, 2026 21:46
@otavio
otavio force-pushed the perf/postgres-tunables branch from 307e51e to 9e9d9b0 Compare August 12, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant