A receiver for bucket notifications, so a change in a customer bucket the service does not front enqueues a change set of exactly the delivered keys instead of waiting for the daily reconcile #1226

Read from main at 877e8a28 on 2026-10-23, as phase 3.2 of obin-ai/handbook design/bucket-datasources/roadmap.md (PR obin-ai/handbook#364). Filed with the owner's go-ahead; not to start before #1211's retained manifest (#1213) is merged — without it an event can only trigger a full re-send, which the daily reconcile already does.

What we are trying to do

A change in a customer's own bucket — one the composed object-storage service does not front — reaches its collection without waiting for the daily reconcile and without a person pressing "Sync now".

What the code says

AddSource refuses a webhook secret for the object-store provider (datasources.go:930-933, "Object storage is pulled, not pushed"); the provider's live_delivery is NONE and supports_webhook false. The only push receivers are GitHub's: the per-source webhook and the App webhook (datasource_webhook.go, datasource_github_app_push.go). For a bucket the service fronts, codefly-dev/service-object-storage#77 point 2 republishes the backing store's notifications on Watch; that does not cover a bucket written by the customer's own systems behind their own endpoint.

What to build

• A receiver for bucket notifications — S3 EventBridge/SQS, MinIO webhook, GCS Pub/Sub — on the host's existing inbound-webhook path (pkg/datasource/handler.go, the StaticSecretResolver → store-resolver seam), verified by the provider's signature or by a per-source shared secret stored as a webhook envelope like GitHub's.
• A delivery names keys, not content: the receiver resolves the source by the (endpoint, bucket) it is registered for, intersects the keys with the source's prefix and patterns (feat(accounts,saas-ui): a bucket source takes include and exclude key patterns, applied before any fetch #1216), and enqueues a change set of exactly those keys against the retained manifest (feat(accounts): the object-storage connector diffs the current listing against one the host retains #1213) — never a full re-send.
• supports_webhook true for the provider; live_delivery reports SOURCE_WEBHOOK when a secret is stored; AddSource/UpdateSource accept the secret for the provider; the kit shows the endpoint to register at the store, as it does for a GitHub source.
• The reconcile stays as the safety net; with a diffing connector its interval can drop toward GitHub's 30 minutes (datasourceReconcileInterval).

Tests before review

A signed delivery for a registered bucket enqueues one change set naming exactly the delivered keys that match the source's scope; an unsigned or wrong-bucket delivery is a quiet 404 as for GitHub; a delivery naming a key outside the prefix or excluded by pattern enqueues nothing; a duplicate delivery is one job (effect identity); the reconcile after a delivery finds nothing to do.

🤖 Generated with Claude Code