Repository navigation
Support distributed REPACK CONCURRENTLY #8772
Description
Current behavior
Citus supports ordinary REPACK for non-partitioned distributed tables. The command is propagated to every shard placement, and regression coverage verifies that shard heaps are rewritten while rows, placements, shard metadata, colocation, and distributed query behavior remain intact.
Citus intentionally rejects REPACK (CONCURRENTLY) for distributed tables. PostgreSQL implements the concurrent form using logical decoding, background workers, replication slots, and execution outside a transaction block; it cannot be propagated through the existing per-shard DDL path.
This is a standalone capability request, not a PostgreSQL 19 integration requirement. It may remain unimplemented indefinitely unless product priority and a safe distributed execution model are established.
Design prerequisites
- Placement-level execution and transaction/restart boundaries.
- Per-node
max_repack_replication_slotsand background-worker admission control. wal_level=logicalpreflight across all participating nodes.- Interaction with rebalancing, shard moves/splits, and existing logical replication.
- Partial-success reporting, cleanup, idempotent retry, and operator recovery.
- Behavior for replicated shards, reference tables, Citus local tables, and partitioned distributed tables.
Relationship to existing work
- Ordinary distributed
REPACKis already supported and is therefore not a prerequisite issue to create. - PG19: reject distributed REPACK CONCURRENTLY before exclusive lock wait #8756 / PG19: use ShareUpdateExclusiveLock for REPACK CONCURRENTLY #8805 fixed lock ordering so unsupported distributed concurrent requests fail clearly and ordinary local-table
REPACK (CONCURRENTLY)works in a Citus-enabled database.
Do not create an implementation PR until the execution and failure model is approved.
Activity
ihalatci commentedon Aug 15, 2026
Scoping note: distributed
REPACK CONCURRENTLYI traced this against PostgreSQL 19beta2 (
~/.pgenv/src/postgresql-19beta2) and Citus atpg19-support, and verified the load-bearing claims on a live single-node cluster ratherthan from the docs. Recommendation: defer. There is a hard upstream blocker that no
amount of Citus-side engineering removes, and the fallback designs all trade the current
fail-closed behaviour for something worse.
The decisive blocker
repack.c:307:Citus ships shard DDL through
worker_apply_shard_ddl_commandinside a distributedtransaction. That is a transaction block on the worker, so
REPACK (CONCURRENTLY)on ashard errors out by construction — it is not a matter of Citus choosing not to
propagate it.
cluster.c:129-134already records this.Any implementation therefore has to run each shard's REPACK outside the coordinated
transaction, on its own connection, with no 2PC. That is not a variation on the existing
shard-DDL path; it is a second, parallel execution model with its own failure semantics.
Corrections to premises in the issue
wal_level = logicalis not required.check_concurrent_repack_requirements(
repack.c:907+) gates only onwal_level < WAL_LEVEL_REPLICA, i.e. the defaultreplicais sufficient.
EnsureLogicalDecodingEnabled()(repack_worker.c:224) turns on logicaldecoding on demand. Confirmed live: a core-only
REPACK (CONCURRENTLY)on a cluster withwal_level=replicacreated slotpg_repack_76251and completed. So there is no cluster-widewal_levelprecondition to check or document.The slot budget is per-backend, not per-shard. The slot is named
pg_repack_<PID>(
repack_worker.c:221) and isRS_TEMPORARY; one REPACK per backend means N shards on onenode do not consume N slots from a single statement. The pressure is still real, just
differently shaped:
max_repack_replication_slotsdefaults to 5(
src/backend/replication/slot.c:163,postgresql.conf.sample:359) and has contextPGC_POSTMASTER(src/backend/utils/misc/guc_parameters.dat:2108) — it cannot be raisedwithout a restart — while
citus.max_adaptive_executor_pool_sizedefaults to 16.Concurrent user REPACKs, not fan-out width, are what exhaust it.
Repack slots come from a dedicated reserved slice, not the general pool.
ReplicationSlotCreate(slot.c:440-441) partitions the shared slot array:so REPACK allocates only from
[max_replication_slots, max_replication_slots + max_repack_replication_slots)and everything else only from[0, max_replication_slots).Two consequences. REPACK cannot starve the shard rebalancer or shard splits of slots,
and vice versa — so slot contention between the two is not a concern. But the slice is a
hard per-node ceiling of 5 concurrent
REPACK CONCURRENTLY, not a soft limit that canbe borrowed against: raising
max_replication_slotsdoes not move it, only thePGC_POSTMASTERmax_repack_replication_slotsdoes, and that needs a restart. It can alsobe set to
0, disabling the feature on a node outright. (An operator who hits the ceilingis at least pointed at the right knob:
slot.c:461-462emitserrhint("Free one or increase \"%s\"", repack ? "max_repack_replication_slots" : ...).)This is the load-bearing capacity argument, and it is a firmer one than contention would
have been: any fan-out design must gate on a fixed, dedicated resource it cannot borrow
against — 5 per node, against a
citus.max_adaptive_executor_pool_sizedefault of 16 —rather than compete for an elastic shared pool. Admission control is therefore mandatory,
not an optimisation.
What is actually required, if this is ever built
asynchronously and independently. A mid-run failure leaves a table repacked unevenly.
There is no rollback story, so this needs an explicit contract (resumable? best-effort?
reported how?) before any code.
PGC_POSTMASTERlimit of 5, with a clear error rather than a mid-fan-out failure.identity that is neither
NOTHINGnorFULL, and a valid identity index ornon-deferrable PK. Plenty of distributed tables fail these, and the failure would surface
per shard, late.
splits already use logical replication slots on the same nodes. As noted above they do
not contend for the same slot budget, so this is not a capacity problem — but REPACK and a
concurrent shard move do both need
AccessExclusiveLockon the same relation at theirfinal swap, so this still needs a mutual-exclusion story with the rebalancer.
Verdict
Defer. Item 1 alone means a new execution model, and items 2 and 5 are design questions
about correctness guarantees, not implementation detail. The current behaviour — reject with
CONCURRENTLY option is currently unsupported for distributed tables— is fail-closed andcorrect, and is much better than a version that succeeds on some shards.
Worth noting for whoever picks this up:
repack.c:3234hasINJECTION_POINT("repack-concurrently-before-lock", NULL), which would make thelock-ordering parts testable deterministically.
Separately, while tracing this I confirmed #8756:
PreprocessClusterStmttookAccessExclusiveLockunconditionally, before theCONCURRENTLYcheck, which both made therejection wait on an exclusive lock and silently escalated the lock on non-distributed
tables inside a Citus-enabled database. That is a distinct, small fix and is tracked there.
Edited to fix two errors in the original version of this comment: (1)
PGC_POSTMASTERandthe default of 5 were attributed to
src/include/replication/slot.h, which only carries theexterndeclaration — the context lives inguc_parameters.datand the default inslot.c; (2) I claimed REPACK competes with the rebalancer for the same slot budget. It doesnot —
slot.c:440-441gives REPACK a dedicated slice. Subsequently also softened a claimthat the dedicated slice leaves operators without guidance: core's own
errhint(
slot.c:461-462) names the correct GUC. The capacity argument now rests on the hardper-node ceiling of 5 rather than on contention, which is a stronger footing. The verdict is
unchanged throughout.
ihalatci commentedon Aug 15, 2026
Correction to this issue body (author self-correction)
Two of the premises I wrote here do not survive checking against the PG19 beta2 source. Recording rather than editing, so the error stays visible.
1. The
wal_level = logicalpremise is DISPROVEN. I statedREPACK CONCURRENTLYrequireswal_level = logical. It does not - seerepack.c:907+and:914. This removes what I presented as the main deployment obstacle.2. Slot contention is DISPROVEN as I framed it. I implied concurrent repacks would collide over replication slots. They do not collide in the way I described (
slot.c:440-441,:461-462).3. What actually survives, and it is still blocking. The real constraint is a fixed, dedicated, per-node ceiling of 5 slots, changeable only by restart (
guc_parameters.dat:2108,PGC_POSTMASTER). Citus defaultscitus.max_adaptive_executor_pool_sizeto 16. A distributed REPACK CONCURRENTLY fanning out to more nodes than the ceiling allows will fail partway, so admission control is mandatory, not optional. That is a stronger and more specific argument than the one I originally made.4. Separate blocker, unchanged:
PreventInTransactionBlockatrepack.c:307.Net effect: the recommendation to defer stands, but for one concrete reason (fixed per-node slot ceiling plus mandatory admission control) rather than the three I originally listed, two of which were wrong.
ihalatci commentedon Aug 19, 2026
Status: no local implementation/PR; recommendation is defer. Needs: architecture for nontransactional placement execution, rebalancer exclusion, fixed per-node REPACK slot/background-worker admission control, partial-failure/retry semantics, then design approval before code. The smaller rejection-timing bug is tracked separately in #8756.
ihalatci commentedon Sep 28, 2026
Beta 4 reassessment: REPACK CONCURRENTLY remains available, but this issue stays deferred design work rather than an active integration PR. Correct support still requires a nontransactional placement execution model, admission control for the fixed per-node REPACK slot budget, rebalancer exclusion, and explicit partial-failure/retry semantics. #8756/#8805 is the only near-term REPACK guardrail.
Update PostgreSQL 19 CI to RC1 (#8929)
[-]PG19: support distributed REPACK CONCURRENTLY[/-][+]Support distributed REPACK CONCURRENTLY[/+]