feat(deletion): add durable whole-community deletion

[kalvinnchau]

Summary

Adds a durable, operator-controlled V1 for deleting an entire Buzz community without deleting another tenant's data.

The workflow is exposed through buzz-admin deletions:

• sweep records independent fleet storage-taxonomy observations
• submit, list, inspect, and approve manage a deletion request
• unblock resumes a fail-closed request after an operator records remediation identity and reason
• run and drain execute bounded work

Requests advance through a PostgreSQL-backed state machine and stop at retention_pending after logical deletion has been independently verified across PostgreSQL, object storage, and Redis.

This PR ships the engine and CLI, not a continuously running worker or Kubernetes packaging. For V1, a cluster/VM administrator invokes /usr/local/bin/buzz-admin from the existing relay image, for example with kubectl exec or an equivalent container/VM exec path.

What whole-community V1 removes

For the target community, V1 removes:

• rows from the allowlisted community-scoped PostgreSQL catalog, including members, profiles, authored events and bodies, DMs, reactions, mentions, memberships, tokens, workflows, moderation, audit, feedback, and rate-limit state
• media sidecars and upload-attribution records under _meta/<community>/ and _uploads/<community>/
• Git repository pointers under repos/<community>/
• Redis keys under buzz:<community>:*

The community row survives as a permanent tombstone, and deletion control-plane records remain as evidence of the request, approval, execution, and result.

Safety model

Deletion is not a broad DELETE CASCADE followed by optimistic cleanup. The destructive boundaries are durable and fail closed.

1. Inventory and approval

• submit resolves the target and freezes the schema plus summary-only storage inventory.
• Approval is bound to the exact request, community, and frozen inventory digest.
• Unsupported manifest versions, malformed keys inside the target's owned prefixes, live scoped-table/write-fence coverage drift, frozen-inventory mismatch, and approval mismatch block execution rather than guessing. Migration and catalog revision numbers are not authorization gates; the executor validates the live safety shape instead.
• Storage inventory is server-side prefix scoped to exactly:
  • _meta/<community>/
  • _uploads/<community>/
  • repos/<community>/
• The deletion path never lists the whole shared bucket and has no arbitrary per-community object cap. Its listing work is proportional to the target community's bindings, not total fleet storage.
• Fleet-wide taxonomy sweeps remain independent observability. They report unknown writer shapes but do not gate deletion submission, fencing, or destructive progress. Maintainers must add deletion taxonomy coverage whenever a new community-owned object-key class is introduced; writer-coverage tests bind the current media and Git writers to that contract.

2. Quiesce, fence, and destructive freeze

• Writes continue through submission, inventory, and approval. They stop when execution moves the target into quiescing and then establishes the durable fence.
• Already-admitted external effects finish under heartbeated serving-write leases; the exact admitted lease may renew while the community is quiescing, but new lease acquisition is rejected. The executor drains admitted leases before destructive work.
• Invite minting after quiescing begins fails as typed AccessDenied (HTTP 503 at the relay boundary) before an invite can be persisted.
• Database triggers enforce the community write fence across the complete catalog of community-scoped tables. Startup/readiness and destructive execution validate that catalog so a newly added but unfenced table cannot silently escape.
• Named isolation assumption — fresh write snapshot. Every writer transaction that can reach a community-fenced relation must use PostgreSQL READ COMMITTED; each guarded write therefore observes a statement snapshot no older than acquisition of the community deletion lock. REPEATABLE READ and SERIALIZABLE can retain a pre-fence snapshot and are unsupported for writers. The writer pool refuses non-READ COMMITTED sessions at connection setup, and both SQL fence functions reject an explicit per-transaction isolation override with SQLSTATE 25000. Configuration-delivered bad isolation can surface through SQLx as a pool-acquire timeout because every after_connect attempt is rejected; the precise community writes require READ COMMITTED isolation reason remains observable when the SQL guard is reached. Read-only replica transactions are outside this assumption.
• Holding the shared advisory lock until the guarded write executes is a separate liveness condition: under READ COMMITTED, releasing it early does not permit resurrection because the trigger rechecks the fence, but it can turn a fleet sweep into a statement-wide SQLSTATE 55000 abort.
• After the fence closes writers, storage is re-enumerated into chunked side-table rows. Per-prefix counts and digests bind those concrete keys to the destructive manifest.
• Manifest chunk insertion, update, and deletion are protected after freeze. This closes the race where an unbound key could otherwise appear after the manifest was committed.

3. Checkpointed destruction

• Target-owned object bindings are deleted from the frozen destructive manifest in bounded batches with durable progress.
• The concrete key list lives in chunked side-table rows rather than one request-row JSON value. It supports large communities, resumable execution, and terminal cleanup.
• Missing objects are accepted as idempotent crash-window outcomes; malformed ownership, changed evidence, and unexplained target-prefix drift fail closed.
• PostgreSQL purging remains scoped by community_id, including the guarded NIP-RS hard-delete path discovered with real Desktop kind 30078 read-state data.
• Redis cleanup explicitly scans and UNLINKs only buzz:<community_id>:*. Natural expiry is insufficient because some keys, including tunnel generation counters used as fencing state, are deliberately persistent.

4. Independent verification

• PostgreSQL logical absence is checked after purge.
• The three target-owned storage prefixes are freshly inventoried again and must be empty.
• Redis requires two complete empty namespace scans.
• Only after all three stores pass does the request advance through logically_verified to retention_pending.

What V1 deliberately does not erase

Shared content-addressed storage

Per-community deletion removes bindings, metadata, attribution records, and Git pointers. It does not physically delete fleet-shared CAS bytes that another community may still reference:

• media blobs and thumbnails
• Git manifests, packs, and indexes (manifests/, packs/, and idx/)

Safe reclamation requires a separate fleet-wide reachability and retention GC. Unknown keys elsewhere in the shared bucket do not block one community's deletion; malformed or unrecognized keys inside that community's three owned prefixes still fail closed.

External retained copies

The online logical-deletion proof does not erase object versions/replicas, database backups/WAL, CDN copies, provider retention copies, or observability exports. Those require their own retention and purge controls.

Member-only erasure

This PR erases a whole community. It does not implement the different operation "erase one npub while preserving the community."

Removing membership or accepting NIP-09 is not member erasure. A member-only workflow would need to find and selectively remove or redact authored event content and pubkeys, profile data, DMs, reactions, mentions, memberships/roles, tokens, workflows/subscriptions, upload attribution, moderation/audit history, repository attribution, and identity embedded in tags or JSON. It would also need explicit rules for ownership transfer, surviving replies and thread metadata, audit-chain integrity, immutable Git history, and shared-CAS reachability. That requires a pubkey-level fence and selective graph rewrite; it is a separate deletion product, not a safe extension of this whole-tenant worker.

In scope

• migration 0029_community_deletion.sql: requests, approvals, leases, manifest chunks, checkpoints, tombstones, and the universal write-fence catalog
• durable executor leases, generations, heartbeats, retry/block state, and resumable stage transitions
• operator-driven sweep, submit, list, inspect, approve, unblock, run, and drain commands
• serving-path fences for database writes and external effects across event ingest, media, Git, workflow, push, invites, mesh/tunnel, and related paths
• target-prefix-only storage inventory, summary manifests, post-fence destructive chunks, and bounded batch deletion
• exact community Redis namespace purge and two-pass absence verification
• cross-community isolation, crash/resume, manifest-integrity, writer-taxonomy, and schema/migration regressions
• desired-state schema/schema.sql support without requiring a SQLx migration ledger

Deferred / not covered

• dedicated Helm/chart worker Deployment, service account, secrets, probes, resources, and network policy
• autonomous buzz-admin deletions worker poll loop and worker-only health server
• least-privilege separation among migration, relay-serving, and destructive execution roles
• fleet-wide shared-CAS physical GC
• backup/provider/CDN/observability retention completion
• member-only erasure
• provider-native conditional-delete improvements
• a general force-continue escape hatch; permanent safety failures remain fail closed unless an operator remediates the cause and records an audited unblock

The removed continuous-worker implementation remains deferred; no remote follow-up branch is claimed by this PR.

Validation

Current PR head and repository state

Current pushed head: 359d8402ee15f049768f54156f67b953c7a7e2ed, rebased onto cc9a2f783375e51a6e8d1f2f9d01d5f7e22813d1 (origin/main at push time). The complete PR diff is now 47 files, 9,834 additions, and 517 deletions.

The bespoke source-scanner stack was removed to keep this PR scoped to community deletion. Tyler/team requested the underlying fenced-write safety behavior, not ast-grep, crates/buzz-db/tests/community_fenced_writes.rs, its 27 fixtures, or the new scripts/lints/community_*.yml rules. Those scanner-specific files, dependencies, Hermit links, and runner wiring are absent from the current tree. The production database write fence, startup/destructive live-catalog validation, and deletion behavior remain.

Source validation on this exact SHA passed:

• cargo fmt --all -- --check
• bash -n scripts/run-tests.sh
• cargo nextest run -p buzz-db --all-targets: 102 passed, 173 skipped, 0 failed
• cargo nextest run -p buzz-deletion --all-targets: 10 passed, 9 skipped, 0 failed
• cargo nextest run -p buzz-admin --all-targets: 1 passed, 0 failed
• affected-package/all-target Clippy with warnings denied
• lockfile consistency
• Helm 3.16.4 lint and all 44 chart unit tests
• Helm region controls using that fixture: default BUZZ_S3_REGION=us-east-1, explicit eu-west-2 override, and blank-region schema rejection

The prior Kubernetes battery below was run against 928992237358a3294621ac0280830b77155abc04. It remains useful evidence for the patch-equivalent production deletion implementation, but it is not claimed as exact-SHA evidence for current head 359d8402ee15f049768f54156f67b953c7a7e2ed; the current cleanup removes only scanner/test/tooling infrastructure. CI restarted for the new head after the rebase and is pending. Human review remains CHANGES_REQUESTED.

Prior-head live Kubernetes deletion and safety gates

The full program used one immutable image, real PostgreSQL, Redis, MinIO, and a three-relay Kubernetes release:

• source: 928992237358a3294621ac0280830b77155abc04 (prior head)
• image: buzz-e2e:sha-928992237358
• immutable image digest: sha256:a1a204f4618ac22d9e210be5e5290645a15d79831ae30b0e44379357c8e4a895
• evidence root: /tmp/buzz-e2e/20260807T033025Z-928992237358-full-gates/
• evidence-manifest digest: 82875c5bc9bea7370b796a7aef3457b3a1c8306c84c59e0f7388bbb5ad30e865

Passed gates at that prior head:

• Chart/operator region: default us-east-1, explicit nondefault propagation, blank-region schema rejection, live in-pod environment, and an in-pod taxonomy sweep over 18 objects with zero unknown.
• Fenced writers and lifecycle: open-write/fence ordering; 100-attempt anti-starvation; invite, push matcher, and exhausted-reaper bystander isolation; non-READ-COMMITTED rejection; manifest/tombstone contracts; eight-failure stage block and audited unblock.
• Destructive lifecycle: submit → approve → run → retention_pending; PostgreSQL tombstone and Redis/S3 verification true; zero retries/errors; terminal reruns rejected with exit 5.
• Fresh 10,001-object crash boundary: exactly two chunks (10,000 + 1). The executor deleted chunk 0 from MinIO while its PostgreSQL stamp was row-lock-blocked, was killed with SIGKILL, left one object and both stamps absent, then resumed the same request under generation 2 to zero objects and terminal state.
• Independent dead-owner recovery: a dedicated executor claimed generation 1, blocked before effects, and was killed through containerd with SIGKILL (no TERM cleanup). The request remained owned and unreclaimable before lease expiry; a successor claimed generation 2 after 60 seconds and completed with two attempts and zero retries.
• Three-pod socket isolation: ordinary NIP-42 and joined huddle-audio target witnesses on every replica received exact 1008 / community deleted; healthy-tenant witnesses on those pods remained live; deleted-host reconnect returned HTTP 404.
• Health/provenance: all replicas independently returned ready and retained the exact image digest before/after destructive runs and an audio-enabled rolling restart; PostgreSQL, Redis, and MinIO were healthy at close.

Instrument corrections were retained as evidence rather than counted as product failures: a foreground PostgreSQL forward caused an initial PoolTimedOut; Kubernetes pod deletion exercised graceful TERM rather than dead-owner recovery; shell-background socket witnesses died with their parent; and the first image build hit the corporate TLS proxy. Detached forwarding/witnesses, containerd SIGKILL, and the configured internal CA/Artifactory mirror produced the discriminating runs without weakening product security.

Prior-head cleanup

For the prior-head Kubernetes run, the Helm release was removed, namespace absence was verified, run-owned Screen sessions were absent, and that source worktree remained clean. The evidence manifest was independently recomputed and every indexed artifact passed shasum -a 256 -c. The current 359d8402 source worktree is also clean after the scanner-only cleanup and push.

[wesbillman]

Review submitted by Carl on Wes's behalf. I traced the deletion lifecycle, DB trigger and lease model, storage manifest freeze/delete path, Redis purge/verification, and serving-side call sites, with independent adversarial passes from Mongo and Princess Donut. CI is green, but these findings block the feature's advertised safety/operability contract.

[wesbillman]

[P0] Bump this to 28 and pin it to the actual migration set. This PR moved community deletion to 0028_community_deletion.sql, but SQLx-backed deployments report max(version) = 28. validate_destructive_migration_version requires exact equality, so validate_catalog() rejects every submit/inventory and execution attempt with expected 27, got 28. Desired-state deployments hide this because they have no _sqlx_migrations ledger. Please also add a regression that compares this constant to the migrator's concrete max version rather than testing only values derived from the constant.

[wesbillman]

[P1] Keep already-admitted leases renewable while quiescing, or otherwise preserve the exclusion proof for long effects. begin_quiescing promises that admitted external effects finish and the fence drains them, but this predicate makes the next 10-second heartbeat fail as soon as state becomes quiescing; ServingWriteGuard then cancels the protected operation and its drop path releases the lease. Media/video and Git object-store writes can legitimately exceed that interval, and cancellation of the client future does not prove the remote S3 mutation did not commit after the lease was released. That allows destructive listing/freeze to overtake an admitted remote write—the exact race the lease is meant to close. New acquisition is already blocked by deletion_state = 'active'; renewal should remain allowed for the exact existing lease/fence generation through quiescing, with a regression covering a long admitted external write across quiesce.

[kalvinnchau]

Fixed in 4e0726e5a. Renewal now permits only the exact existing, unexpired lease while the community is active or quiescing; owner, lease generation, fence generation, expiry, tombstone, and community fence-generation checks remain intact, while new acquisition is still restricted to active. Added PostgreSQL coverage for renewal/drain behavior and a long-running ServingWriteGuard heartbeat regression across quiesce. Both pass against isolated PostgreSQL at PR HEAD 7c755960d.

[wesbillman]

[P2] This does not map the actual fence failure. mint_relay_invite performs an ordinary insert; the new relay_invites trigger raises SQLSTATE object_not_in_prerequisite_state, which arrives as DbError::Sqlx, not DeletionSafety. The fallback therefore returns HTTP 500 during quiescing/fencing. Guard the invite transaction so the boundary produces a typed access/fence error (and map it to the intended restriction status), or explicitly translate this SQLSTATE. Please cover mint-after-quiesce at the API boundary.

[kalvinnchau]

Fixed in 7c755960d. mint_relay_invite now takes the community advisory lock and checks the write fence inside the same transaction before inserting, returning typed DbError::AccessDenied instead of leaking the trigger's SQLx error. The API maps that typed fence to HTTP 503. Added PostgreSQL coverage proving mint-after-quiesce returns AccessDenied and persists no row, plus an HTTP boundary regression proving the typed error maps to 503. Both pass at PR HEAD 7c755960d.

[kalvinnchau]

All three original blocking review findings are addressed on current head 928992237358a3294621ac0280830b77155abc04: migration-revision coupling was removed in favor of live structural validation; exact admitted serving leases remain renewable through quiescing while new acquisition is blocked; and invite minting now takes the lifecycle guard and maps its typed AccessDenied result to HTTP 503 with database/API regressions. The latest repair also restores the calibrated fenced-writer class guard, fixes Helm-deployed deletion operator S3-region operability, and corrects the PR isolation/evidence documentation. Existing CI is running on this head, and the independent exact-SHA Kubernetes battery is in progress. @wesbillman please re-review when those checks are terminal.