Restructure MongoDB V3 compacting

[rkistner]

The goals and design are described in v3-compaction-design.md.

The main changes are:

1. On V3 storage, initial replication now triggers a new "chunk-merge compact", instead of a checksum pre-calculation. This has the same benefits as checksum pre-calculation, but additionally merges small chunks into larger chunks. Unlike a full compact, this does not de-duplicate operations, making it much cheaper to run.
2. The fields and indexes on the bucket-state collections are modified (breaking change for V3 storage; no migration path; V3 storage is still considered unstable).
3. Any compact run only checks buckets that may need compacting, making it fast to run on much shorter intervals.
4. Multiple compact jobs may be run concurrently, without duplicating significant work.
5. All buckets would eventually get a full compact, preventing historical data from persisting indefinitely if the bucket is not modified further.

This does not affect compacting on MongoDB storage V1 or Postgres storage. This does not affect compacting of parameter indexes.

The per-bucket full compact implementation is mostly unchanged, apart from stats/state calculations.

Compact action

This adds a new --incremental-only option, which can be used for compact jobs running on a short interval. This has skips compacting streams using V1 storage. This is useful for cases that may still have replication stream using both V1 and V3 storage, or where the job does not know upfront what storage version is used.

For V3 compaction, the compact job now also covers PROCESSING streams.

Work performed

Compaction	What it does	bucket_data work	Expected performance
Chunk-merge	Combines adjacent small chunks without changing their operations	One metadata scan per batch. Each actual merge reads full operations, then verifies and replaces its inputs.	Usually limits work to new chunks added since the last run.
Full	Reads full bucket, turns superseded row operations into smaller MOVE operations, and combines adjacent results	One payload scan per batch. Each changed group is verified and replaced. One final metadata query totals the untouched tail.	Reads and processes every eligible operation.
Full with CLEAR	Performs full compaction, then replaces a leading run of MOVE, REMOVE, and CLEAR operations with one CLEAR operation	Full-compaction work, plus batched metadata reads and replacements for complete leading documents, then one boundary read and replacement.	Adds work at the start of the bucket.
Nothing below the limit	Finds that the operation limit excludes every document	One lightweight scan that returns no documents; nothing is written.	Minimal cost. The bucket is simply rescheduled.

Tracking the bucket stats becomes quite complex in some cases, and adds some implementation complexity. I tried to limit the complexity, but couldn't avoid it in all cases. Specifically, to avoid re-reading chunks to calculate stats, we need to combine in-memory stats with data we processed, with other stats from the db.

Future work

1. Make some parameters configurable, such as minimum period between chunk-merge compacts or full compacts.
2. Add a mode for running a persistent compact job that polls for new work, in addition to the current mode of periodic jobs.
3. Automatically run a compact job concurrently with initial replication, instead of only afterwards, to reduce the overall time required for initial replication.

AI Usage

Manually designed and documented the approach. Used Codex gpt-5.6 to implement, especially the MongoDB queries. Reviewed manually and with Claude Opus 5.

[changeset-bot]

🦋 Changeset detected

Latest commit: b5a3418

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@powersync/service-module-mongodb-storage	Minor
@powersync/service-core	Minor
@powersync/service-module-postgres-storage	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-postgres	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mysql	Patch
@powersync/service-schema	Minor
@powersync/service-module-mssql	Patch
@powersync/service-image	Minor
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2c76691643

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e56441ab21

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cb7dc4bf65

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ed22d41b0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".