S3: Stability and performance fixes

[rkistner]

Bugfix

An error during the initial snapshot can cause an uncaught rejection that kills the process. This is because the initialSnapshotDone promise is unusued in that case. The fix just silences the uncaught rejection by doing .catch(() => {}). This affected MongoDB replication using MongoDB V3 storage.

Performance Improvements

These were performance issues that I picked up during an initial round of performance testing. Without these fixes, the performance using S3 was significantly slower than V1 storage, both for replication and syncing. With these fixes, the performance is on par or faster in my tests.

Concurrent Uploads

The compact process already does concurrent uploads to S3, but the replication process still processed uploads sequentially. This refactors to also do concurrent uploads when replicating.

Inline threshold size

This increases the threshold from 1KB -> 16KB. During performance testing, I found that having 1000x 1KB S3 objects in one batch just adds too much overhead. 16KB should work better.

The trade-off here is increased source db load and potential memory usage. For example, when we query for 1000 chunks/documents for a sync request, this can be up to 16MB of data now.

Redo sync batching (getBucketDataBatch)

In V1 storage, sync batching roughly works as follows:

1. Query for 1000x operations, also limited by MongoDB to 64MB.
2. Split by bucket, and send each batch to the client.
3. Query for the next 1000x operations.

In V3 storage, we don't limit by operation count on the db level. So instead we had:

1. Query for 1000x chunks.
2. Cut the chunks to a max of 16MB. This may discard some data that we queried from MongoDB, but that overhead is generally low. Note that this is a lower ceiling than the 64MB we have with V1, which is a good thing.
3. Fetch all the referenced S3 files.
4. Merge operations from all fetched buckets / s3 chunks.
5. Re-chunk operations and yield to the client, stopping after 1000 operations.
6. Repeat, to query for the next batch.

Step 5 above effectively matched the old behavior from the client's point-of-view. The issue is that it caused a lot of rework:

1. We may fetch files from S3 and then not use them at all.
2. We may fetch a file from S3 and send only a small part of that to the client.
3. We'd do unnecessary work by merging and re-chunking.

This rewrites the batching process to effectively remove step 4 and 5. Instead, we send all data that we have loaded already, to prevent that rework.

This is still limited to 16MB, so the upper limit on the amount of data we keep in memory is still lower than the 64MB in V1 storage.

We could reduce memory usage for some cases further by:

1. Fetching the S3 files on demand as we're sending them to the client, rather than upfront. This will need some care to not make everything completely sequential, which can add a lot of latency.
2. Reducing the 16MB limit further. Note that it would still be a "soft" limit, since there could be 16MB in a single operation.

There were some tests that relied on the assumption of an exact limit of 1000 per batch. This rewrites some of those tests to not rely on that assumption anymore.

This also returns exactly one bucket definition group per batch now. Once one group is complete, we effectively start from scratch with a new database query, so there is no gain in continuing the current batch. This does change the return type: we may now yield a global { hasMore: true } flag, indicating the client should fetch another batch, even if every individual chunk had { has_more: false }.

AI Usage

Manually guided the optimizations, using Codex gpt-5.6 to implement them and rewrite the tests. Manually simplified and optimized getBucketDataBatch.

[changeset-bot]

🦋 Changeset detected

Latest commit: b4288e8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-schema	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-core	Patch
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[stevensJourney]

The changes make sense, look reasonable and are implemented cleanly.

Codex pointed out that we're missing a Changeset entry to @powersync/service-core - but I think we already have other Changeset entries on main for that package, so it should not cause any publishing issues I believe.