From e376d7bcc9c1ed2226463fb9fa49e35ae7ef615f Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:01:57 -0400 Subject: [PATCH 01/58] docs(design): true up implementation status against the codebase The module map marked the implemented crypto core (primitives, keys, encryption, provenance, verify_asset, backup) as planned while a dozen docs described deferred networked/MLS surfaces in the present tense. Align every status claim with what actually ships: - module-map: drop stale (planned) markers; add the missing crypto::authority, lifecycle, validation, and cbor rows; re-home hardware-key adapters from capsule-sdk to capsule-core + the swift/kotlin harnesses; roster rows for capsule-core-ffi and capsule-vision plus a non-cargo package appendix; mark the E2E surface as contract-only until the first networked slice lands - principles: define the Implemented / Planned / Blocked status convention all docs now follow - keys: document the implemented write-authority seam (AlbumAuthority, ReferenceAuthority, SignedEpochLedger) that verify_asset actually consumes, with OpenMlsAuthority as the blocked drop-in - mls: add the upstream-blocker status note (0x004D has no IANA codepoint and no RustCrypto OpenMLS backend, openmls#1940) that MLS-dependent docs now link instead of implying live MLS - federation, peering, authentication, validation, organization, versioning, mls-resilience, device-enrollment, moderation: fix present-tense implemented-in claims for planned surfaces - filesystem/client: mark the cache-eviction sweep implemented (issue #23); import/pipeline: mark the streaming mode and free-space probe planned and note the legacy executor still writes the unsigned sidecar --- .../src/content/docs/design/authentication.md | 2 +- .../content/docs/design/cryptography/index.md | 2 +- .../content/docs/design/cryptography/keys.md | 22 ++++++- .../content/docs/design/cryptography/mls.md | 4 +- .../content/docs/design/device-enrollment.md | 2 +- .../src/content/docs/design/federation.md | 2 +- .../content/docs/design/filesystem/client.md | 2 + .../content/docs/design/import/pipeline.md | 8 ++- .../src/content/docs/design/mls-resilience.md | 2 +- .../src/content/docs/design/moderation.md | 2 +- .../src/content/docs/design/module-map.md | 58 +++++++++++-------- .../src/content/docs/design/organization.md | 4 +- .../src/content/docs/design/peering.md | 2 +- .../src/content/docs/design/principles.md | 10 ++++ .../docs/design/threat-model/validation.md | 2 +- .../src/content/docs/design/versioning.md | 2 +- 16 files changed, 86 insertions(+), 40 deletions(-) diff --git a/capsule-docs/src/content/docs/design/authentication.md b/capsule-docs/src/content/docs/design/authentication.md index 5ba4dbcb..c68df011 100644 --- a/capsule-docs/src/content/docs/design/authentication.md +++ b/capsule-docs/src/content/docs/design/authentication.md @@ -5,7 +5,7 @@ description: Identity, account portability, session and access tokens Authentication binds a user identity to their master key, which is the root of every encryption and decryption operation in Capsule. The server can prove "this request is from a session it issued" but cannot prove "this user is who they say they are" — the master key, owned client-side, is the actual identity root. Everything below works to keep that binding intact through the lifetime of a session and across server moves. -Implemented in `capsule-api-auth`: OIDC handler (`oidc`), session ledger (`session`), claim validation (`claims`), per-device records (`devices`). The session token format and the OIDC discovery surface below are the contracts other components — including federated peers — depend on. +Implemented in `capsule-api-auth`: OIDC handler (`oidc`), session ledger (`session`), claim validation (`claims`). The per-device enrollment records (`devices`) are planned with [Device Enrollment](/design/device-enrollment/). The session token format and the OIDC discovery surface below are the contracts other components — including federated peers — depend on. ## Design Principles diff --git a/capsule-docs/src/content/docs/design/cryptography/index.md b/capsule-docs/src/content/docs/design/cryptography/index.md index ef00ccb4..df01cc2f 100644 --- a/capsule-docs/src/content/docs/design/cryptography/index.md +++ b/capsule-docs/src/content/docs/design/cryptography/index.md @@ -28,6 +28,6 @@ Capsule's E2E security stacks four layers, each owned by its own sub-doc: ## Implementation Posture - **Centralized.** All cryptographic primitives, key handling, and `verify_asset` live in `capsule-core::crypto`. There is no per-platform divergence in what gets verified or how — only in where keys are physically held. -- **Audited libraries only.** libcrux (formally verified), RustCrypto, ed25519-dalek, x25519-dalek, OpenMLS. Capsule is never the first serious user of a primitive's implementation. +- **Audited libraries only.** RustCrypto, ed25519-dalek, x25519-dalek today; OpenMLS (over libcrux, formally verified) is the *planned* group layer — currently [blocked upstream](/design/cryptography/mls/). Capsule is never the first serious user of a primitive's implementation. - **Memory hygiene.** Decrypted bytes and key material are zeroed on drop; secure-allocation is used where the platform supports it, to prevent swap leaks. - **Trust the server for storage, never for authorization.** The server holds opaque ciphertext and key-free index facts. Every authorization is verified against MLS-distributed material; a server's assertion of access is never sufficient. diff --git a/capsule-docs/src/content/docs/design/cryptography/keys.md b/capsule-docs/src/content/docs/design/cryptography/keys.md index f2464536..c67fd5b2 100644 --- a/capsule-docs/src/content/docs/design/cryptography/keys.md +++ b/capsule-docs/src/content/docs/design/cryptography/keys.md @@ -3,7 +3,7 @@ title: Key Management description: Capsule's key hierarchy, device coordination, and write authorization --- -Capsule's keys form a single hierarchy with one backed-up root. The hierarchy is implemented in `capsule-core::crypto::keys`; hardware-bound storage adapters (Secure Enclave, StrongBox/Keystore, TPM) live in per-platform glue under `capsule-sdk::hardware-keys`. +Capsule's keys form a single hierarchy with one backed-up root. The hierarchy is implemented in `capsule-core::crypto::keys`. Every signing site consumes the device signing key through the `Signer` seam there, and hardware-backed keys implement the `HardwareSigner` foreign trait (exported under `capsule-core`'s `ffi` feature), so software and hardware keys are interchangeable. Reference adapters ship for software (`crypto::keys::software`, CI-smoked) and TPM 2.0 (`crypto::keys::tpm`, feature-gated); Secure Enclave and StrongBox adapters live in the `capsule-core-swift` / `capsule-core-kotlin` harness packages. End-to-end hardware composition awaits the planned **P-256 hybrid-DSK variant** — secure elements expose ECDSA-P256, not Ed25519 (see [Device Enrollment](/design/device-enrollment/)). - The **account master key** is the only key that is escrowed/backed up. It does not encrypt assets directly. Its job is to (1) wrap the per-device identity private keys and (2) anchor the encrypted backup that escrows album keys. - **Device keys** are hardware-bound, non-exportable, and therefore disposable — a device is re-bootstrapped from the master key rather than recovered. @@ -90,11 +90,11 @@ A device signature on an [asset manifest](/design/cryptography/provenance/#asset This is **the** contract every consumer of `capsule-core::crypto` depends on. It is invoked from import, sync, federation, peering, and backup-restore — anywhere an asset enters the local trusted set. - **Epoch-bound write proof.** Every asset manifest carries, in addition to the device DSK signature, a signature under the album's **per-epoch write-tier signing key**. Only writers at that epoch hold that key. The manifest's `amk_version` identifies the epoch. -- **Authorization authority is MLS history, not the server.** The client verifies the write-tier signature against the write-tier public key it learned for that epoch *from MLS* — the album's MLS commit chain (admin-signed) is the sole authority on who could write when. A server-asserted authorization is never sufficient. +- **Authorization authority is admin-signed history, not the server.** The client verifies the write-tier signature against the write-tier public key it learned for that epoch from the album's **write authority** — an admin-signed attestation chain, consumed through the [`AlbumAuthority` interface](#write-authority-interface) below. The design-target authority is the album's MLS commit chain; a server-asserted authorization is never sufficient. - **Epoch ceiling is MLS-attested, not server-asserted.** The monotonic `amk_version` ceiling a client enforces is derived from the album's admin-signed [MLS commit chain](/design/cryptography/mls/#membership-operations) — the same authority that admits writers — never from the server's stored counter. A brand-new client learns the current epoch from the MLS group state delivered in its [Welcome](/design/cryptography/mls/#history-delivery-for-new-joiners), then rejects any manifest whose `amk_version` exceeds that MLS-attested epoch, so a server can neither fabricate a future epoch nor rewind to an old one and have a client honor it. The server's own no-key monotonicity check (invariant 18 in [Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants)) is a structural backstop, not the authority — it stops a *client* from skipping epochs, while MLS stops the *server* from fabricating them. - **What this accepts vs. rejects.** An asset signed by a writer who was *later* removed is still acknowledged — it was valid when written, and nothing after removal un-seeds it. An asset signed at an epoch where the signer lacked write capability is **rejected**: an attacker (or a buggy/colluding server) cannot produce a valid write-tier signature for an epoch they were not a writer in. - **Backdating buys nothing.** Ordering and authorization ride the provenance hash-chain and the `amk_version` epoch, never the self-asserted `timestamp` (see the timestamp note below). The "pre-sign backdated assets, then upload them after removal" attack therefore fails on the *epoch*, not the clock: a manifest must carry a `write_sig` for the epoch it claims, and a removed writer holds no write-tier key for any epoch past their removal. Anything they upload afterward either names an old epoch the chain has already advanced beyond (rejected by the monotonic `amk_version` + chain-head checks) or a new epoch they cannot sign for. -- **Single verification chokepoint.** All of this lives in one `verify_asset(manifest, ciphertext, mls_state)` function in `capsule-core::crypto`. It is the only path by which a client acknowledges an asset, and per [contract-driven development](/design/principles/) it ships with exhaustive negative test cases: reader-signed, removed-writer, wrong-epoch, forged certificate chain, replayed manifest. +- **Single verification chokepoint.** All of this lives in one `verify_asset(manifest, ciphertext, authority)` function in `capsule-core::crypto`, where `authority` is the album's [`AlbumAuthority`](#write-authority-interface). It is the only path by which a client acknowledges an asset, and per [contract-driven development](/design/principles/) it ships with exhaustive negative test cases: reader-signed, removed-writer, wrong-epoch, forged certificate chain, replayed manifest. - **Crypto validity, not durability.** `verify_asset` proves an asset is cryptographically valid and authorized; it says nothing about whether the *server* still durably holds the bytes. Confirming durable, indexed, retrievable server storage — the precondition for any destructive local cleanup — is a separate, key-free query, the [storage-verification endpoint](/design/import/storage-verification/). A safe local release requires **both** to pass. - **Defensive failure handling.** A verification failure is *never* silently dropped and *never* silently accepted. The asset is quarantined and surfaced in the [provenance/audit trail](/design/cryptography/provenance/#provenance-of-library-modifications) so an operator can distinguish a bug from an attack after the fact. This bounds the blast radius of an implementation bug. - **Transient vs. terminal outcomes.** `verify_asset` returns one of three outcomes, not two: **accept**, **terminal-reject** (reader-signed, removed-writer, wrong-epoch, forged chain, suite-downgrade → quarantined as above), and **pending**. *Pending* is the narrow, recoverable case where the manifest's `amk_version` is within the MLS-attested epoch range but the corresponding AMK content key has not yet arrived over the in-band [AlbumKeyDistribution](/design/cryptography/encryption/#asset-key-derivation) message (an epoch bump whose key broadcast is still in flight). A pending asset is **held and retried** as MLS state catches up — never quarantined as forged and never accepted unverified — until the key arrives or a configurable timeout elapses, after which it escalates to a surfaced quarantine. This distinction stops an in-flight epoch bump from flagging honest concurrent uploads as attacks; see [Failure Modes](/design/cryptography/failure-modes/#failure-mode-catalog). @@ -102,6 +102,22 @@ This is **the** contract every consumer of `capsule-core::crypto` depends on. It - **Key delivery is authorization-neutral.** An adopted [web-upload drop](/design/web-upload/#why-adopt-in-place) carries its file key *wrapped* under the AMK ([`key_mode = wrapped`](/design/cryptography/provenance/#asset-manifest)) instead of deriving it, but it is authorized **identically** to any other write — `verify_asset` checks the adopter's `device_sig` and epoch `write_sig`, and the guest who supplied the bytes is never a signer. `key_mode` and `wrapped_file_key` are ordinary signed manifest fields covered by both signatures, so a tampered wrapped key, or a `key_mode` inconsistent with the manifest, is a terminal-reject like any other forged field. - **Timestamp is audit-only.** A manifest's `timestamp` is the client's self-asserted capture/write time and is **never** load-bearing for authorization or ordering — those ride the epoch and the chain above. The server stamps its own trusted `received_at` in the [server-visible envelope](/design/filesystem/server/#postgresql-what-the-server-knows) as the authoritative wall-clock for any time-based policy (retention, rate limits); the client `timestamp` is preserved verbatim for display and audit. A server-side *sanity* bound on `timestamp` is a gross-drift guard for honest clients, **not** an authorization control — it surfaces a wildly-wrong clock rather than silently distorting the audit trail. Grammar owned by [Threat Model — Schema Rules](/design/threat-model/schema-rules/) and mirrored in [Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). +### Write Authority Interface + +"Who could write at which epoch" is consumed through one interface, implemented in `capsule-core::crypto::authority`. `verify_asset` takes the authority as a parameter (`&dyn AlbumAuthority`) and asks it exactly four things: + +- the album's identity (`album_id`); +- the **epoch ceiling** — the highest `amk_version` the admin-signed attestation chain covers; +- the **per-epoch write-tier public key** for any attested epoch; +- whether the local device **holds the AMK** for an epoch (the input separating *pending* from *terminal-reject* above) — plus whether the admin attestation chain itself verifies. + +Two implementations exist or are planned: + +- **`ReferenceAuthority` (implemented).** An admin-signed epoch ledger: each epoch entry (AMK version + write-tier public key) is attested by the album admin key, and the ledger serializes to a `SignedEpochLedger` whose admin chain is re-verified on load. This is the shipped offline authority — it powers multi-epoch rotation (`Workspace::rotate_epoch`) and the exhaustive `verify_asset` negative-case suite today, with no MLS dependency. +- **`OpenMlsAuthority` (blocked).** The design-target authority backed by live MLS group state — the admin-signed commit chain described throughout this doc. It is blocked on an upstream OpenMLS backend for the chosen ciphersuite (see the [status note in Cryptography — MLS](/design/cryptography/mls/)) and drops in behind the same interface without touching `verify_asset`. + +Everywhere this doc says an epoch or write-tier key is "MLS-attested", the code-level statement is "attested by the album's `AlbumAuthority`"; MLS is how that attestation is *produced and distributed* once live. + ## Device Directory Each user publishes a signed device directory. Other users (and federated peers) read it to learn which devices belong to whom and which public keys to trust. diff --git a/capsule-docs/src/content/docs/design/cryptography/mls.md b/capsule-docs/src/content/docs/design/cryptography/mls.md index 567d1efe..f5e54738 100644 --- a/capsule-docs/src/content/docs/design/cryptography/mls.md +++ b/capsule-docs/src/content/docs/design/cryptography/mls.md @@ -3,7 +3,9 @@ title: MLS Group Membership description: How Capsule binds MLS (RFC 9420) to its identity layer and uses it for album membership --- -Capsule's group layer is the [MLS ciphersuite](/design/cryptography/primitives/#mls-ciphersuite) from the inventory. It is implemented in `capsule-core::crypto::mls` as a thin wrapper over OpenMLS — the wrapper is what binds MLS to Capsule's identity layer ([Keys](/design/cryptography/keys/)) and to the in-band AMK distribution. +Capsule's group layer is the [MLS ciphersuite](/design/cryptography/primitives/#mls-ciphersuite) from the inventory. It will be implemented in `capsule-core::crypto::mls` (planned) as a thin wrapper over OpenMLS — the wrapper is what binds MLS to Capsule's identity layer ([Keys](/design/cryptography/keys/)) and to the in-band AMK distribution. + +**Status: blocked upstream.** The chosen ciphersuite (`MLS_256_XWING_CHACHA20POLY1305_SHA256_Ed25519`, `0x004D`) rides a non-final IETF draft with no IANA codepoint, and OpenMLS supports it only through a C (`libcrux`) backend — there is no RustCrypto PQ backend yet (openmls#1940). Until that lands, the epoch-authority role this doc assigns to the MLS commit chain is filled offline by the admin-signed epoch ledger behind the [`AlbumAuthority` interface](/design/cryptography/keys/#write-authority-interface); the membership ceremonies, `Welcome`/history delivery, and the [album upgrade ceremony](/design/versioning/#album-upgrade-ceremony) wait for live MLS. Docs that depend on live MLS link to this note rather than restating it. The ciphersuite's choice of [ChaCha20-Poly1305](/design/cryptography/primitives/#mls-control-aead) (rather than the [AES-GCM](/design/cryptography/primitives/#bulk-aead) used for user data) is acceptable because: diff --git a/capsule-docs/src/content/docs/design/device-enrollment.md b/capsule-docs/src/content/docs/design/device-enrollment.md index 0614e19d..f4a5c1fa 100644 --- a/capsule-docs/src/content/docs/design/device-enrollment.md +++ b/capsule-docs/src/content/docs/design/device-enrollment.md @@ -10,7 +10,7 @@ A Capsule account has one or more devices, each holding a hardware-bound DSK + D These are distinct from **[cross-device recovery](/design/backup-recovery/#default-mechanisms)** (which is also a way to bring up a new device, but in the recovery context — the user has lost their other devices and is using the recovery passphrase + master-key escrow to restore). -Implementation will live in `capsule-core::crypto::keys` (key generation and wrapping) and `capsule-api-auth::devices` (the device directory and the enrollment authentication surface). The ceremony glue lives in per-platform native client code (QR scan, biometric prompt). +Implementation lives in `capsule-core::crypto::keys` (key generation and wrapping — implemented) and `capsule-api-auth::devices` (the device directory and enrollment authentication surface — planned). The ceremony glue lives in per-platform native client code (QR scan, biometric prompt). The MLS group joins these ceremonies invoke are blocked upstream — see the [MLS status note](/design/cryptography/mls/). ## First-Device Enrollment diff --git a/capsule-docs/src/content/docs/design/federation.md b/capsule-docs/src/content/docs/design/federation.md index a9ad34c4..1390b9b1 100644 --- a/capsule-docs/src/content/docs/design/federation.md +++ b/capsule-docs/src/content/docs/design/federation.md @@ -5,7 +5,7 @@ description: How Capsule servers share albums across users on different home ser Federation lets an album owned on one Capsule server be shared with users whose accounts live on another. This document covers **server-to-server** federation only; direct device-to-device sync for a single user is [Peering](/design/peering/). -Federation reuses the existing read primitives — `/sync`, `/blob/{hash}`, the standard manifest envelope. The only new things federation introduces are a **capability token** (the contract that gates which peers may fetch what) and a **per-peer compartmentalization layer**. Implemented in `capsule-api-sync::federation`: capability issuance, verification, the pull path, and per-peer rate budgeting. +Federation reuses the existing read primitives — `/sync`, `/blob/{hash}`, the standard manifest envelope. The only new things federation introduces are a **capability token** (the contract that gates which peers may fetch what) and a **per-peer compartmentalization layer**. Planned to live in `capsule-api-sync::federation` (today a stub crate): capability issuance, verification, the pull path, and per-peer rate budgeting. ## Threat Model diff --git a/capsule-docs/src/content/docs/design/filesystem/client.md b/capsule-docs/src/content/docs/design/filesystem/client.md index b77ecd38..91f9e4e6 100644 --- a/capsule-docs/src/content/docs/design/filesystem/client.md +++ b/capsule-docs/src/content/docs/design/filesystem/client.md @@ -58,6 +58,8 @@ What is eligible for reclamation is exactly the rebuildable-or-refetchable set: ### Automatic cache management +Implemented (issue #23): last-access tracking lives in the `cached_representations` table in `capsule-core::db`, and the sweep is `capsule-core::library::cache_sweep`. The connection-class detection that would *drive* the byte budget from `capsule-sdk` is planned — the budget is a plain parameter today. + The reclaimable set is held within a **user-configurable cache budget**. When it grows past budget — typically while browsing a large library on a device that cannot hold everything — Capsule reclaims space itself rather than waiting for the user or letting the OS decide: - **Recency promotion.** Viewing or opening an asset stamps a last-access time on its fetched representations in `library.sqlite`. Recently-viewed content is therefore the *last* to go, so scrolling back through an album already browsed on a high-latency or metered connection hits local cache instead of the network. diff --git a/capsule-docs/src/content/docs/design/import/pipeline.md b/capsule-docs/src/content/docs/design/import/pipeline.md index 4016dfb6..bc3a82bb 100644 --- a/capsule-docs/src/content/docs/design/import/pipeline.md +++ b/capsule-docs/src/content/docs/design/import/pipeline.md @@ -30,7 +30,7 @@ The server independently enforces a closed-enum `content_type` allow-list at ses The planner is **pure**: given the scanned files and their extracted metadata, it produces an `ImportPlan { added: [..], skipped: [..], conflicts: [..], total_size }` deterministically. The plan is shown to the user (summary of what will be imported, total size, any issues), and the user confirms or adjusts. -**Free-space probe.** Alongside `total_size`, the client probes the library volume's available space (`capsule-core::library::available_bytes()`) and sets `streaming_recommended: bool` on the plan when `total_size` is near or over free space (a configurable headroom margin). The probe is I/O, so it runs *outside* the pure planner and is attached at confirmation — the planner stays deterministic — exactly as the destination-pointer snapshot is a planner input rather than a discovered-later value. `streaming_recommended` is what surfaces [import-upload streaming mode](#import-upload-streaming-mode) to the user before execution begins. +**Free-space probe (planned).** Alongside `total_size`, the client probes the library volume's available space (`capsule-core::library::available_bytes()`, planned) and sets `streaming_recommended: bool` on the plan when `total_size` is near or over free space (a configurable headroom margin). The probe is I/O, so it runs *outside* the pure planner and is attached at confirmation — the planner stays deterministic — exactly as the destination-pointer snapshot is a planner input rather than a discovered-later value. `streaming_recommended` is what surfaces [import-upload streaming mode](#import-upload-streaming-mode) to the user before execution begins. - If an asset is already uploaded *locally* in the library, import refuses it — no merge needed. - If an asset already exists *remotely* under a different ciphertext (e.g. re-encrypted under a newer album key), import still admits it; the [upload protocol](/design/import/upload-protocol/#deduplication-and-merge) then resolves it as a merge (the existing blob is linked rather than re-uploaded). @@ -50,8 +50,12 @@ For each file in the plan, in [upload prioritization](#upload-prioritization) or Step 1–3 can be parallelized across files. The executor is cancellation-aware: a partially-executed plan can be aborted cleanly and resumed (re-running the import re-derives the plan and skips already-completed work via the deterministic planner). +**Status note.** The signed path in step 2 — encrypt, manifest, provenance — is implemented in `capsule-core::lifecycle::Workspace` and writes through to the shared `library.sqlite` index. The standalone `import::executor` still writes the legacy **unsigned** `AssetSidecar`; unifying it onto the signed path waits on derivative generation (step 3, planned with [Thumbnails](/design/thumbnails/)). + ## Import-Upload Streaming Mode +**Status: planned.** The offline pipeline above (scan → plan → execute) is implemented in `capsule-core::import`; streaming mode, the free-space probe, and the verify-and-release loop are the planned seam over it — a new executor drive mode, with no change to the upload wire protocol. + The default pipeline imports every file into the local library *before* upload, so the device temporarily holds the whole import on disk. That is impossible on a storage-constrained device — a laptop with a near-full SSD importing a library larger than its free space. **Streaming mode** removes the requirement that the full import ever land locally at once. It is **auto-detected**: when the [free-space probe](#plan--confirm) reports `total_size` is near or over available space, the plan is marked `streaming_recommended` and the user confirms a streaming import. Instead of executing all files and then uploading, the executor runs a bounded **sliding window**: @@ -85,7 +89,7 @@ The pipeline decides which assets to *start*; the [upload protocol](/design/impo What the rest of the system depends on this module for: - `ImportPlan` — the deterministic output of the planner; rendered to the UI for confirmation. Schema fields: `added` (each entry carrying its resolved destination `album_id`), `skipped`, `conflicts`, `total_size`, `import_id` (UUIDv7), and `streaming_recommended` (set at confirmation from the [free-space probe](#plan--confirm), not by the pure planner). -- `available_bytes() → u64` — the library volume's free space (a thin `statvfs` / `GetDiskFreeSpaceEx` wrapper in `capsule-core::library`); the input that decides `streaming_recommended`. +- `available_bytes() → u64` (planned) — the library volume's free space (a thin `statvfs` / `GetDiskFreeSpaceEx` wrapper in `capsule-core::library`); the input that decides `streaming_recommended`. - `execute(plan, cancel_token) → ImportExecutionReport` — the executor entry-point. Honors the cancel token at every file boundary. Returns per-file status. In [streaming mode](#import-upload-streaming-mode) it drives the per-asset import→upload→verify→release window instead of executing-then-uploading in bulk. - A stable progress event stream so the UI can report per-asset state (queued / encrypting / uploading / done / failed). diff --git a/capsule-docs/src/content/docs/design/mls-resilience.md b/capsule-docs/src/content/docs/design/mls-resilience.md index b377d78a..06fae6d8 100644 --- a/capsule-docs/src/content/docs/design/mls-resilience.md +++ b/capsule-docs/src/content/docs/design/mls-resilience.md @@ -5,7 +5,7 @@ description: How Capsule's MLS layer recovers from lost commits, state divergenc OpenMLS handles MLS (RFC 9420) correctly under normal operation — commits ordered by the group's chain, duplicates rejected, ratchet advanced atomically. But MLS can still hit scenarios the base protocol does not resolve on its own: a commit lost in transit, two devices proposing concurrently with the wrong ordering, a member whose local state has diverged from the server's. This doc owns Capsule's recovery contracts for those edge cases. -It is kept **separate** from [Cryptography — MLS](/design/cryptography/mls/) (which owns the ciphersuite binding and the four standard ceremonies) because recovery is a distinct, cross-cutting concern — it reaches into the [OGK](/design/cryptography/keys/#owner-group-keys-ogks), backup, and quarantine UX, not the steady-state membership protocol. The recovery surfaces here are exercised in `capsule-core::crypto::mls` (the OpenMLS wrapper) and surface to users through quarantine + reconciliation UX in the native clients. +It is kept **separate** from [Cryptography — MLS](/design/cryptography/mls/) (which owns the ciphersuite binding and the four standard ceremonies) because recovery is a distinct, cross-cutting concern — it reaches into the [OGK](/design/cryptography/keys/#owner-group-keys-ogks), backup, and quarantine UX, not the steady-state membership protocol. The recovery surfaces here will be exercised in `capsule-core::crypto::mls` (the OpenMLS wrapper — blocked upstream with the rest of the MLS layer; see the [status note](/design/cryptography/mls/)) and surface to users through quarantine + reconciliation UX in the native clients. ## Failure Modes diff --git a/capsule-docs/src/content/docs/design/moderation.md b/capsule-docs/src/content/docs/design/moderation.md index 8e884de0..e1785b0e 100644 --- a/capsule-docs/src/content/docs/design/moderation.md +++ b/capsule-docs/src/content/docs/design/moderation.md @@ -32,7 +32,7 @@ A report against `alice@other.tld`'s asset is routed to her home server's admini Server-level blocklists, plus per-user blocks that federate: - **Server-level blocklist.** A server admin publishes a list of peer servers that this server refuses to accept federated requests from. Operates at the [federation capability](/design/federation/#federation-capabilities) layer. -- **Per-user block.** A user can block another user; the block is enforced by the blocker's home server — the blocked user is removed from albums shared with the blocker and cannot share new albums with them. Removal is an ordinary MLS `Remove` + AMK epoch bump applied at the blocked user's next sync; the prior epochs' keys they already hold are not retroactively clawed back (consistent with [removal semantics](/design/cryptography/mls/#remove-user-charlie)). A per-user block is **scoped to that user**: it does **not** propagate as a server-wide federation block, so one user (or a coordinated group) cannot weaponize blocks to sever an entire peer server from the federation. Each home server enforces only its own users' blocks. +- **Per-user block.** A user can block another user; the block is enforced by the blocker's home server — the blocked user is removed from albums shared with the blocker and cannot share new albums with them. Removal is an ordinary MLS `Remove` + AMK epoch bump applied at the blocked user's next sync; the prior epochs' keys they already hold are not retroactively clawed back (consistent with [removal semantics](/design/cryptography/mls/#remove-user-charlie)). A per-user block is **scoped to that user**: it does **not** propagate as a server-wide federation block, so one user (or a coordinated group) cannot weaponize blocks to sever an entire peer server from the federation. Each home server enforces only its own users' blocks. (The MLS `Remove` this rides is blocked upstream — see the [MLS status note](/design/cryptography/mls/).) - **Blocklist exchange (v2).** A peer-level mechanism for sharing *server-level* blocklists across federated servers (so a malicious server isn't pure whack-a-mole) is **deferred to v2**, but its shape is fixed now: signed, versioned blocklist documents an admin **opts into** consuming from peers they already trust — never auto-applied, and deliberately distinct from per-user blocks (which never propagate). v1 ships only the manual server-level blocklist above. ### Untrusted-Server Whitelist diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 3f95897f..5adbd75d 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -5,14 +5,15 @@ description: Index of every code module to its owning design doc and validation This is the developer's first stop. It maps every Capsule workspace crate and module to the design doc(s) that govern its behavior, and to the validation tier (Unit / Smoke / E2E — see [Validation Tiers](/design/principles/#validation-tiers)) it ships with. The E2E test surface at the bottom is **bounded**: adding a test there means adding the test to the relevant doc's Validation section and justifying why the cross-module surface is irreducible. -The mapping reflects the *design intent*. Some modules listed below are currently planned (annotated `(planned)`) rather than already implemented in the codebase — the doc structure already accounts for them so the boundary is set before code lands. +The mapping reflects the *design intent*. Modules not yet implemented are annotated `(planned)` — or `(blocked)` where a named upstream dependency gates them — per the [status convention](/design/principles/#implementation-status); every unannotated module exists in the codebase today. The repo-root `SLICES.md` indexes the planned modules as executable implementation slices. ## Crate Roster | Crate | Purpose | | ------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `capsule-core` | Shared logic across server and clients: cryptography, library layout, import pipeline, metadata, ML orchestration | -| `capsule-sdk` | Client SDK: auto-generated OpenAPI client, upload protocol, per-platform hardware-key + peering glue | +| `capsule-sdk` | Client SDK: auto-generated OpenAPI client, upload protocol, LAN-peering glue | +| `capsule-core-ffi` | uniffi bindings crate for native Swift/Kotlin consumers (sidecar `Catalog` surface; the `HardwareSigner` foreign trait ships under `capsule-core`'s `ffi` feature) | | `capsule-api` | Server entry-point + routing | | `capsule-api-auth` | Authentication, sessions, OIDC, device directory | | `capsule-api-library` | GraphQL API for UI queries (assets, albums, search) | @@ -29,6 +30,9 @@ The mapping reflects the *design intent*. Some modules listed below are currentl | `capsule-media` | Standalone media utility crate | | `capsule-i18n` | Runtime localization (locale negotiation + ICU message formatting) for the server and CLI | | `capsule-web` | Browser/WASM web-upload client: guest drop sealing + upload (planned) | +| `capsule-vision` | Python research package for on-device ML experimentation (not a cargo crate) | + +Native app and harness packages — `capsule-android`, `capsule-swift`, `capsule-core-swift`, `capsule-core-kotlin` — are not cargo crates and carry no design owner of their own; the two harness packages hold the per-platform `HardwareSigner` reference adapters ([Keys — Device Keys](/design/cryptography/keys/#device-keys)). ## Module → Design Doc @@ -36,15 +40,19 @@ The mapping reflects the *design intent*. Some modules listed below are currentl | Module | Owning design doc | Validation tier | | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | -| `crypto::primitives` (planned) | [Cryptography — Primitives](/design/cryptography/primitives/) | Unit (RFC vectors) | -| `crypto::keys` (planned) | [Cryptography — Keys](/design/cryptography/keys/), [Device Enrollment](/design/device-enrollment/) | Unit + Smoke (hardware per-platform) | -| `crypto::mls` (planned) | [Cryptography — MLS](/design/cryptography/mls/), [MLS Resilience](/design/mls-resilience/) | Unit + Smoke (protocol round-trip) | -| `crypto::encryption` (planned) | [Cryptography — Encryption](/design/cryptography/encryption/) | Unit (KAT, round-trip) | -| `crypto::provenance` (planned) | [Cryptography — Provenance](/design/cryptography/provenance/) | Unit (exhaustive negative cases) + Smoke | -| `crypto::verify_asset` (planned) | [Cryptography — Write Authorization](/design/cryptography/keys/#write-authorization) | Unit (the single chokepoint; exhaustive) | -| `backup` (planned) | [Backup and Recovery](/design/backup-recovery/) | Unit + Smoke | +| `crypto::primitives` | [Cryptography — Primitives](/design/cryptography/primitives/) | Unit (RFC vectors) | +| `crypto::keys` | [Cryptography — Keys](/design/cryptography/keys/), [Device Enrollment](/design/device-enrollment/) | Unit + Smoke (hardware per-platform) | +| `crypto::authority` | [Cryptography — Keys: Write Authority Interface](/design/cryptography/keys/#write-authority-interface) | Unit (epoch-ledger round-trip) | +| `crypto::mls` (blocked upstream — see the [MLS status note](/design/cryptography/mls/)) | [Cryptography — MLS](/design/cryptography/mls/), [MLS Resilience](/design/mls-resilience/) | Unit + Smoke (protocol round-trip) | +| `crypto::encryption` | [Cryptography — Encryption](/design/cryptography/encryption/) | Unit (KAT, round-trip) | +| `crypto::provenance` | [Cryptography — Provenance](/design/cryptography/provenance/) | Unit (exhaustive negative cases) + Smoke | +| `crypto::verify_asset` | [Cryptography — Write Authorization](/design/cryptography/keys/#write-authorization) | Unit (the single chokepoint; exhaustive) | +| `validation` | [Threat Model — Validation](/design/threat-model/validation/) | Unit (pure key-free invariants) | +| `cbor` | [Metadata — Canonical CBOR](/design/metadata/#canonical-cbor-encoding) | Unit (RFC 8949 §4.2 vectors) | +| `lifecycle` | [Filesystem — Client](/design/filesystem/client/), [Import — Pipeline](/design/import/pipeline/) | Unit + Smoke (`capsule demo` end-to-end) | +| `backup` | [Backup and Recovery](/design/backup-recovery/) | Unit + Smoke | | `library::{init,open,rebuild,lock,paths,scrub,trash}` | [Filesystem — Client](/design/filesystem/client/), [Filesystem — Maintenance](/design/filesystem/maintenance/) | Unit + Smoke | -| `import::{scanner,planner,executor,plan,upload,group,progress,special}` | [Import — Pipeline](/design/import/pipeline/) | Unit (planner determinism) + Smoke (executor) | +| `import::{scanner,planner,executor,plan,upload,group,progress,special}` (streaming upload loop planned) | [Import — Pipeline](/design/import/pipeline/) | Unit (planner determinism) + Smoke (executor) | | `metadata::{file,filter,types}` | [Metadata](/design/metadata/) | Unit (filtering) | | `sidecar::*` | [Metadata — Sidecar Schema](/design/metadata/#sidecar-schema-v1) | Unit (serde determinism) | | `exif::{extract,timezone}` | [Metadata](/design/metadata/) | Unit | @@ -60,9 +68,10 @@ The mapping reflects the *design intent*. Some modules listed below are currentl | Module | Owning design doc | Validation tier | | ------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------- | | (auto-generated client) | [Clients](/design/clients/) | Smoke (re-generated; not unit-tested) | -| `upload` | [Import — Upload Protocol](/design/import/upload-protocol/) | Unit + Smoke (client side) | +| `upload` (client stub) | [Import — Upload Protocol](/design/import/upload-protocol/) | Unit + Smoke (client side) | | `peering` (planned) | [Peering](/design/peering/) | Unit + Smoke per platform | -| `hardware-keys` (planned) | [Cryptography — Keys](/design/cryptography/keys/), [Device Enrollment](/design/device-enrollment/) | Smoke per platform | + +Hardware-key adapters do **not** live in `capsule-sdk`: the `Signer`/`HardwareSigner` seams and the software + TPM reference adapters are in `capsule-core::crypto::keys`, and the Secure Enclave / StrongBox adapters live in the `capsule-core-swift` / `capsule-core-kotlin` harness packages (see [Keys — Device Keys](/design/cryptography/keys/#device-keys)). ### `capsule-api` (root + sub-crates) @@ -94,7 +103,7 @@ The mapping reflects the *design intent*. Some modules listed below are currentl | Crate | Owning design doc | Validation tier | | --------------- | ----------------------------------------------------------------- | -------------------- | | `capsule-cli` | [Clients](/design/clients/) (treats CLI as a client) | Smoke | -| `capsule-media` | (small utility crate; no specific design owner) | Unit | +| `capsule-media` | [Thumbnails and Previews](/design/thumbnails/) (decode/encode utilities; only JPEG decode is implemented today) | Unit | | `capsule-i18n` | [Internationalization](/design/i18n/) | Unit + Smoke | | `capsule-web` | [Web Upload](/design/web-upload/), [Clients](/design/clients/) | Smoke (browser/WASM) | @@ -105,33 +114,34 @@ Navigation from a design doc back to where the code lives. | Design doc | Implementing modules | | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | [Principles](/design/principles/) | (meta — no specific code module) | -| [Cryptography — Primitives](/design/cryptography/primitives/) | `capsule-core::crypto::primitives` (planned) | -| [Cryptography — Keys](/design/cryptography/keys/) | `capsule-core::crypto::keys`, `capsule-sdk::hardware-keys` (both planned) | -| [Cryptography — MLS](/design/cryptography/mls/) | `capsule-core::crypto::mls` (planned, wraps OpenMLS) | -| [Cryptography — Encryption](/design/cryptography/encryption/) | `capsule-core::crypto::encryption` (planned) | -| [Cryptography — Provenance](/design/cryptography/provenance/) | `capsule-core::crypto::provenance` + `verify_asset` chokepoint (planned) | +| [Cryptography — Primitives](/design/cryptography/primitives/) | `capsule-core::crypto::primitives` | +| [Cryptography — Keys](/design/cryptography/keys/) | `capsule-core::crypto::keys` (`Signer`/`HardwareSigner` seams; software + TPM reference adapters; Secure Enclave / StrongBox adapters in `capsule-core-swift`/`-kotlin`; P-256 hybrid-DSK variant planned) | +| [Cryptography — Keys: Write Authority Interface](/design/cryptography/keys/#write-authority-interface) | `capsule-core::crypto::authority` (`AlbumAuthority` trait + `ReferenceAuthority` epoch ledger; `OpenMlsAuthority` blocked with MLS) | +| [Cryptography — MLS](/design/cryptography/mls/) | `capsule-core::crypto::mls` (blocked upstream — no usable OpenMLS backend for the `0x004D` ciphersuite; see the [MLS status note](/design/cryptography/mls/)) | +| [Cryptography — Encryption](/design/cryptography/encryption/) | `capsule-core::crypto::encryption` | +| [Cryptography — Provenance](/design/cryptography/provenance/) | `capsule-core::crypto::provenance` + `verify_asset` chokepoint | | [Cryptography — Failure Modes](/design/cryptography/failure-modes/) | Cross-cutting: `capsule-core::backup`, `capsule-core::library`, `capsule-core::crypto::*` | -| [MLS Resilience](/design/mls-resilience/) | `capsule-core::crypto::mls` (extends main MLS module) | +| [MLS Resilience](/design/mls-resilience/) | `capsule-core::crypto::mls` (extends main MLS module; blocked with it) | | [Device Enrollment](/design/device-enrollment/) | `capsule-core::crypto::keys`, `capsule-api-auth::devices` | | [Authentication](/design/authentication/) | `capsule-api-auth::{oidc,session,claims}` | | [Authorization](/design/authorization/) | `capsule-api-auth::roles`, `capsule-core::crypto::provenance` (verify_asset) | | [Clients](/design/clients/) | `capsule-sdk` + per-platform native code | | [Internationalization](/design/i18n/) | `capsule-i18n` (runtime) + `xtask::i18n` (codegen) + `locales/` source + per-platform generated catalogs | | [Versioning](/design/versioning/) | Cross-cutting: `capsule-api` (header enforcement), `capsule-core::crypto::mls` (upgrade ceremony), `capsule-api-migration` | -| [Backup and Recovery](/design/backup-recovery/) | `capsule-core::backup` (planned), `capsule-api-auth` (escrow surface) | +| [Backup and Recovery](/design/backup-recovery/) | `capsule-core::backup`, `capsule-api-auth` (escrow surface, planned) | | [Metadata](/design/metadata/) | `capsule-core::{metadata,sidecar,exif}`, `capsule-api-library::schema` | | [Filesystem — Server](/design/filesystem/server/) | `capsule-api`, `capsule-api-entity`, blob store glue | | [Filesystem — Client](/design/filesystem/client/) | `capsule-core::{library,db}`, per-platform native code | | [Filesystem — Maintenance](/design/filesystem/maintenance/) | `capsule-core::library::{scrub,rebuild,trash}`, server-side scrub in `capsule-api-upload` | -| [Import — Pipeline](/design/import/pipeline/) | `capsule-core::import::*` (incl. the streaming import-upload loop) + `capsule-core::library::available_bytes` (free-space probe) | +| [Import — Pipeline](/design/import/pipeline/) | `capsule-core::import::*`; the streaming import-upload loop and the `capsule-core::library::available_bytes` free-space probe are planned | | [Import — Upload Protocol](/design/import/upload-protocol/) | `capsule-sdk::upload` (client) + `capsule-api-upload` (server) | | [Import — Download & Sync](/design/import/download-sync/) | `capsule-sdk` (client) + `capsule-api-sync` (server) | | [Import — Storage Verification](/design/import/storage-verification/) | `capsule-api-media::verify` (route) + `capsule-sdk` (client) + `capsule-core` (verify-before-destroy predicate) | | [Federation](/design/federation/) | `capsule-api-sync::federation` | | [Peering](/design/peering/) | `capsule-sdk::peering` (planned) + `capsule-core::backup` (artifact format) | | [Organization](/design/organization/) | `capsule-core::domain::stack_type`, `capsule-api-service::{album,stack}` | -| [AI/ML Integrations](/design/ai/) | `capsule-core::ml` (planned), model registry + per-platform inference runners | -| [Thumbnails](/design/thumbnails/) | Client-side gen in `capsule-sdk` + serving in `capsule-api-media` | +| [AI/ML Integrations](/design/ai/) | `capsule-core::ml` (planned), `capsule-vision` (Python research), model registry + per-platform inference runners | +| [Thumbnails](/design/thumbnails/) | Client-side gen in `capsule-sdk` (planned) over `capsule-media` decode/encode utilities + serving in `capsule-api-media` | | [Share Links](/design/share-links/) | `capsule-core::sharing` (planned), `capsule-api-media::shares` (planned) | | [Web Upload](/design/web-upload/) | `capsule-core::drop` (planned), `capsule-api-media::drops` (planned), `capsule-web` (planned); reuses `capsule-api-upload` for drop chunks | | [Moderation](/design/moderation/) | `capsule-api::moderation` (planned) | @@ -147,6 +157,8 @@ The bounded global list of cross-module integration tests. Editing this list req Target count: ≤ 13 cases. Each one is named by what it proves — not "test X" but "X works through Y and Z." +**Status:** none of the 13 cases is runnable today — every one crosses at least one planned surface (the networked server/client wiring, MLS, or ML), even where its `capsule-core` half is implemented and unit/smoke-tested. The bound starts governing with the first implementation slice that makes a case live; until then this list is the contract the slices build toward. + 1. **Auth → Library query.** Log in via OIDC → access-token → GraphQL query for own albums returns expected list. Covers `capsule-api-auth::oidc + session` × `capsule-api-library::schema`. 2. **Full import + upload + finalize.** Local scan → plan → execute → upload session → finalize → blob present at `blobs/{hash}` + index row marked uploaded. Covers `capsule-core::import` × `capsule-sdk::upload` × `capsule-api-upload` × `capsule-api-entity`. 3. **Sync feed pickup.** Upload from device A → device B's `/sync` advances → device B fetches metadata blob and (per scope) the original. Covers `capsule-api-sync` × `capsule-sdk` download path × `capsule-core::library` write. diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index bdc7651c..9b796377 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -5,7 +5,7 @@ description: Albums (container and view), default-album resolution, asset stacks **Albums** are Capsule's organizational backbone: [container albums](#container-albums) are the cryptographic unit every asset belongs to, while [view albums](#system--smart-albums-views) are derived, key-free presentations. On top of albums, **stacks** group related files (RAW+JPEG pairs, bursts, live photos) so a library stays tidy, and **trash** stages every destructive operation behind a signed retention window so a buggy or hostile actor cannot silently destroy data. Stacks and trash are metadata-only — they never touch the underlying asset bytes. -Implemented across `capsule-core::domain::stack_type` (stack-type enums), `capsule-core::library` (default-album resolution and client-side view evaluation), the metadata sidecar layer for `stack_membership` (see [Metadata](/design/metadata/)), the signed `delete`-manifest envelope for `retention_until`, and the service layer in `capsule-api-service::album`/`stack` for server-side enforcement. The retention contract — the `retention_until` field signed into the `delete` manifest — is the load-bearing piece that prevents a hostile server from accelerating purges. +Implemented across `capsule-core::domain::stack_type` (stack-type enums), `capsule-core::library` (default-album resolution and client-side view evaluation), the metadata sidecar layer for `stack_membership` (see [Metadata](/design/metadata/)), the signed `delete`-manifest envelope for `retention_until`, and the service layer in `capsule-api-service::album`/`stack` for server-side enforcement (planned). The retention contract — the `retention_until` field signed into the `delete` manifest — is the load-bearing piece that prevents a hostile server from accelerating purges. ## Albums @@ -19,7 +19,7 @@ The UI calls two different things "albums," and the design keeps them strictly s A container album is Capsule's primary organizational unit and its primary **sharing and access-control boundary**. An album *is* an MLS group: its cryptographic identity (the per-epoch [AMK](/design/cryptography/keys/#album-master-keys-amks)) and membership operations are owned by [Cryptography — Keys](/design/cryptography/keys/) and [MLS](/design/cryptography/mls/), and its server-side storage shape (rows, blob references, `protocol_version` pin) lives in the [Filesystem — Server](/design/filesystem/server/) Postgres schema. This section owns the *interaction surface* over that machinery. - **Membership and roles.** Each member holds one of the album's three capabilities — read (AMK only), write (AMK + write-tier key), or admin (also the admin-tier key) — delivered over MLS to that member's devices ([Keys — Album Master Keys](/design/cryptography/keys/#album-master-keys-amks)). A role change is an MLS commit and bumps the AMK epoch. -- **Invitation and join.** An admin invites a user by fetching and verifying their [device directory](/design/cryptography/keys/#device-directory) and issuing an MLS `Add` for all their devices; the `Welcome` delivers the AMK range set by the album's `history_policy` ([MLS — History Delivery](/design/cryptography/mls/#history-delivery-for-new-joiners)). Inviting a user on another home server also issues a [federation capability](/design/federation/#federation-capabilities); inviting a non-account recipient uses a [share link](/design/share-links/). Joining is acceptance of the `Welcome`; leaving or removal is an MLS `Remove` + epoch bump. +- **Invitation and join.** An admin invites a user by fetching and verifying their [device directory](/design/cryptography/keys/#device-directory) and issuing an MLS `Add` for all their devices; the `Welcome` delivers the AMK range set by the album's `history_policy` ([MLS — History Delivery](/design/cryptography/mls/#history-delivery-for-new-joiners)). Inviting a user on another home server also issues a [federation capability](/design/federation/#federation-capabilities); inviting a non-account recipient uses a [share link](/design/share-links/). Joining is acceptance of the `Welcome`; leaving or removal is an MLS `Remove` + epoch bump. (The MLS membership operations this surface invokes are blocked upstream — see the [MLS status note](/design/cryptography/mls/).) - **Album-level policy** — `history_policy`, the `protocol_version` pin, and the default `retention_until` — is fixed at creation and changed only through an [album upgrade ceremony](/design/versioning/#album-upgrade-ceremony), never ad hoc. Dialog copy and on-screen presentation remain a client-UX detail. diff --git a/capsule-docs/src/content/docs/design/peering.md b/capsule-docs/src/content/docs/design/peering.md index d7e39518..d71b595a 100644 --- a/capsule-docs/src/content/docs/design/peering.md +++ b/capsule-docs/src/content/docs/design/peering.md @@ -11,7 +11,7 @@ Peering exists as an **accelerator, never a replacement** for normal [server syn - **Offline operation.** When the server is unreachable, devices on a shared LAN still converge. This satisfies the [offline/online divide](/design/principles/) — peering works fully offline. - **Best-effort opportunism.** If no peer is found, peering simply does nothing and the device falls back to server sync. Nothing depends on it succeeding. -Peering is the one module here that lives entirely on the client. It is implemented in `capsule-sdk::peering` (discovery, channel, transfer) over `capsule-core::backup` (the artifact format it ingests). The three contract surfaces — mDNS descriptor, TLS handshake parameters, delta-fetch protocol — are the only new primitives peering introduces; everything else is borrowed. +Peering is the one module here that lives entirely on the client. It will live in `capsule-sdk::peering` (planned: discovery, channel, transfer) over `capsule-core::backup` (implemented — the artifact format it ingests). The three contract surfaces — mDNS descriptor, TLS handshake parameters, delta-fetch protocol — are the only new primitives peering introduces; everything else is borrowed. ## Peering Reuses, Not Reinvents diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 50c6f417..497f170d 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -82,6 +82,16 @@ These are goals, not required headings. Some docs hit all four in a single intro The [Module Map](/design/module-map/) is the cross-cutting index: every code module → owning design doc → validation tier. It is the developer's first stop. +### Implementation Status + +Design intent and shipped code drift apart unless the docs say which is which. Every design doc that names an implementing module states, where that module is introduced, one of three statuses per surface: + +- **Implemented** — the named module exists and its validation tier runs in CI. +- **Planned** — the contract is designed; no code exists yet. Marked `(planned)` in the [Module Map](/design/module-map/). +- **Blocked** — planned, plus a named external blocker (e.g. an upstream dependency), stated where the surface is declared and marked `(blocked)` in the map. + +The repo-root `SLICES.md` (a plain repository file, not part of this site) is the executable index of every planned surface — its slice IDs are the unit of implementation work. A doc describing a planned surface writes the *contract* in normative present tense, but must not claim the code exists. + ## Validation Tiers The three test tiers a design doc may reference: diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index 0562650b..e9a8ddbb 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -5,7 +5,7 @@ description: Server and client refuse-by-default checklists; protocol handshake; The cross-cutting refuse-by-default rules every Capsule receiver runs before persisting any incoming write. These are the operational core of the threat model — a server or client that skips one of them silently widens the blast radius for the entire client class taxonomy. -The server-side invariants are enforced in `capsule-api` (every write path passes through them); the client-side invariants are enforced via the single `verify_asset` chokepoint in `capsule-core::crypto` plus the per-receiver decoder paths. The protocol handshake is a one-shot pre-flight check on every request; idempotency and atomicity invariants are properties of specific write surfaces, each cross-linked to the doc that owns the surface. +The server-side invariants are implemented as pure, key-free checks in `capsule-core::validation` (protocol gate, manifest-envelope check, idempotency keys); wiring them into every `capsule-api` write path lands with the networked surface (planned). The client-side invariants are enforced via the single `verify_asset` chokepoint in `capsule-core::crypto` (implemented) plus the per-receiver decoder paths. The protocol handshake is a one-shot pre-flight check on every request; idempotency and atomicity invariants are properties of specific write surfaces, each cross-linked to the doc that owns the surface. ## Server-Side Validation Invariants diff --git a/capsule-docs/src/content/docs/design/versioning.md b/capsule-docs/src/content/docs/design/versioning.md index bc0566fd..1398846b 100644 --- a/capsule-docs/src/content/docs/design/versioning.md +++ b/capsule-docs/src/content/docs/design/versioning.md @@ -5,7 +5,7 @@ description: How Capsule pins each album to a protocol version, upgrades safely, Changes are inevitable. Capsule minimizes breaking changes but generously accepts compatible ones. The aim is backward-compatible reads forever and a deliberately fail-closed write path — a [version-mismatched client](/design/threat-model/) never silently corrupts state; it is rejected at the handshake. -The enforcement is cross-cutting: every wire request, every album commit, and every sidecar carries a version identifier. The header set below is the **contract** that lets two implementations agree (or fail-closed) without negotiating. Album pinning is implemented in the album metadata model (`capsule-api` + `capsule-core`); the upgrade ceremony is an MLS application-layer flow in `capsule-core::crypto::mls` driven by client UI. The min-supported-client window is enforced server-side in `capsule-api`. +The enforcement is cross-cutting: every wire request, every album commit, and every sidecar carries a version identifier. The header set below is the **contract** that lets two implementations agree (or fail-closed) without negotiating. Album pinning lands in the album metadata model (`capsule-api` + `capsule-core`; planned with the networked surface); the upgrade ceremony is an MLS application-layer flow in `capsule-core::crypto::mls` driven by client UI (blocked with MLS — see the [status note](/design/cryptography/mls/)). The min-supported-client window is enforced server-side in `capsule-api` (planned). ## Versioned Surfaces From 270bd0f3a497c2e3f019f80a5f2e5a81da2aa2c0 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:07:31 -0400 Subject: [PATCH 02/58] docs(design): bind the API surface to REST/OpenAPI plus gRPC sync MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs never stated which server surface speaks which transport, and contradicted each other where they tried: download-sync specified an HTTP GET /sync while the module map called capsule-api-sync gRPC, the threat-model handshake was HTTP-only across three transports, and the OpenAPI-generated SDK could never drive the GraphQL library API. Decide and document the split: REST/OpenAPI for every request/response surface (auth, upload, blob fetch, shares, drops, storage-verify), gRPC (capsule.sync.v1.SyncService) for the sync feed and federation pull, and no server surface at all for rich queries — they run client-side over library.sqlite, which is what the E2EE key-free server model demands. GraphQL (capsule-api-library) is frozen and retiring. - new owner doc design/api-surfaces.md: surface-to-transport map, REST-header to gRPC-metadata mapping, HTTP-status to gRPC-status rejection mapping with the error.* code as the cross-transport discriminator, and the GraphQL retirement rationale - download-sync: the feed is the gRPC Sync RPC (manifest travels as opaque canonical CBOR); blob fetch stays REST ranged GET; sync_seq typed u64; the cursor HMAC noted as a server-internal construction - federation: primitives table gains the transport column; capability rides gRPC authorization metadata - threat-model/validation: the universal gate is stated once in REST terms with the gRPC carriage delegated to api-surfaces - module-map: capsule-api-library marked legacy; E2E case 1 rewritten to auth -> gRPC sync -> client-side library query - principles/index: owner-table rows and overview entries for the new doc plus the previously unlisted web-upload, quota, storage-verification, and mls-resilience layers --- capsule-docs/astro.config.mjs | 1 + .../src/content/docs/design/api-surfaces.md | 80 +++++++++++++++++++ .../src/content/docs/design/federation.md | 16 ++-- .../docs/design/import/download-sync.md | 16 ++-- .../content/docs/design/import/pipeline.md | 2 +- .../design/import/storage-verification.md | 2 +- capsule-docs/src/content/docs/design/index.md | 10 +-- .../src/content/docs/design/module-map.md | 10 +-- .../src/content/docs/design/peering.md | 2 +- .../src/content/docs/design/principles.md | 2 + .../docs/design/threat-model/validation.md | 4 +- .../src/content/docs/design/web-upload.md | 4 +- 12 files changed, 118 insertions(+), 31 deletions(-) create mode 100644 capsule-docs/src/content/docs/design/api-surfaces.md diff --git a/capsule-docs/astro.config.mjs b/capsule-docs/astro.config.mjs index b97dd3a2..4eff2135 100644 --- a/capsule-docs/astro.config.mjs +++ b/capsule-docs/astro.config.mjs @@ -52,6 +52,7 @@ export default defineConfig({ { slug: 'design' }, { slug: 'design/principles' }, { slug: 'design/module-map' }, + { slug: 'design/api-surfaces' }, ], }, { diff --git a/capsule-docs/src/content/docs/design/api-surfaces.md b/capsule-docs/src/content/docs/design/api-surfaces.md new file mode 100644 index 00000000..117a4c46 --- /dev/null +++ b/capsule-docs/src/content/docs/design/api-surfaces.md @@ -0,0 +1,80 @@ +--- +title: API Surfaces +description: Which server surface speaks which transport, and how negotiation and rejection map across them +--- + +Capsule's server exposes exactly **two** wire transports, chosen per surface: **REST with an OpenAPI schema** for request/response surfaces, and **gRPC** for the sync feed and federation pull. This doc owns the surface ↔ transport map and the cross-transport mapping of the [universal handshake](/design/threat-model/validation/#protocol-and-capability-negotiation) and rejection semantics. The handshake *rules* stay owned by [Threat Model — Validation](/design/threat-model/validation/); error-code identity stays owned by [Internationalization](/design/i18n/#server-error-codes); this doc owns only how both ride each transport. + +## Surface ↔ Transport Map + +| Surface | Transport | Crate | Owner doc | +| --------------------------------------------------------- | -------------------------------------------- | ------------------------------------ | ----------------------------------------------------------------------- | +| Authentication (sessions, passkeys, TOTP, OIDC) | REST | `capsule-api-auth` | [Authentication](/design/authentication/) | +| Resumable upload (`POST/HEAD/PATCH /upload`) | REST | `capsule-api-upload` | [Import — Upload Protocol](/design/import/upload-protocol/) | +| Blob fetch (`GET /blob/{hash}`, HTTP `Range`) | REST | `capsule-api-media` | [Import — Download & Sync](/design/import/download-sync/) | +| Share serving (`/s/{opaque-id}`) | REST | `capsule-api-media::shares` (planned) | [Share Links](/design/share-links/) | +| Guest drops (`/u/{opaque-id}/drop`, inbox, adoption) | REST | `capsule-api-media::drops` (planned) | [Web Upload](/design/web-upload/) | +| Storage verification (`POST /storage/verify`) | REST | `capsule-api-media::verify` (planned) | [Import — Storage Verification](/design/import/storage-verification/) | +| Device enrollment (`/auth/devices/enroll…`) | REST | `capsule-api-auth::devices` (planned) | [Device Enrollment](/design/device-enrollment/) | +| Version/handshake surface | REST | `capsule-api` | [Threat Model — Validation](/design/threat-model/validation/) | +| **Sync feed** (change discovery after a cursor) | **gRPC** — `capsule.sync.v1.SyncService` | `capsule-api-sync` (stub today) | [Import — Download & Sync](/design/import/download-sync/) | +| **Federation pull** (peer server fetches an album's feed) | **gRPC** — same service, capability-gated | `capsule-api-sync::federation` (planned) | [Federation](/design/federation/) | +| Library queries (timeline, albums, search) | **none — client-side** over `library.sqlite` | `capsule-core::library` + `db` | [Organization](/design/organization/#system--smart-albums-views) | + +Two consequences worth stating plainly: + +- **Blob bytes always ride REST.** Ranged, resumable byte transfer is what HTTP is best at (`Range`, caches, proxies); gRPC has no ranged-read idiom. The gRPC surface carries only the small, typed feed — content hashes, envelopes, encrypted metadata blobs — never original or derivative bytes. +- **Rich queries have no server surface at all.** The server is key-free and cannot evaluate a content predicate; every timeline, filter, and smart-album query runs client-side against the rebuildable local index. What the server offers is exactly the feed needed to *build* that index. + +## Why Two Transports + +- **REST + OpenAPI** for every request/response surface: the OpenAPI schema is the machine-readable contract that generates `capsule-sdk` (progenitor), and a plain HTTP surface stays debuggable with nothing but `curl` — which matters for a self-hosted product. +- **gRPC** only where a typed, paged feed contract earns it: the sync feed and its federation twin are one proto contract consumed by every client and every peer server, with the signed manifest traveling as opaque canonical CBOR inside it (never re-modeled as proto fields — re-encoding would detach it from its signatures). + +## Legacy: GraphQL (retiring) + +`capsule-api-library` exposes an `async-graphql` schema at `/v1/library`. It predates the E2EE key-free server model and is **retiring**, not evolving: + +- Its resolvers presume a server that can read content (people, faces, smart tags, memories server-side) — structurally impossible under the [threat model](/design/threat-model/); the key-free replacements are client-side ML ([AI/ML](/design/ai/)) and client-side views ([Organization](/design/organization/#system--smart-albums-views)). +- The generated SDK is OpenAPI-derived and cannot drive GraphQL, so the surface was never consumable by the client stack the design commits to. +- The query role it aimed at is served client-side over `library.sqlite` (above), fed by the sync feed. + +The crate stays frozen (compiling, unmounted from new work) until the client-side query path reaches parity; retirement is an explicit slice in the repo-root `SLICES.md`. New surfaces MUST NOT be added to it. + +## Negotiation Across Transports + +The [universal headers](/design/threat-model/validation/#universal-headers) are defined once, as REST headers. On gRPC they ride call metadata with the lowercased key names; the values and fail-closed rules are identical. + +| REST header | gRPC call metadata | Direction | +| ---------------------------- | ---------------------------- | ----------------- | +| `X-Capsule-Protocol` | `x-capsule-protocol` | request | +| `X-Capsule-Crypto-Suite` | `x-capsule-crypto-suite` | request (writes) | +| `X-Capsule-Sidecar-Schema` | `x-capsule-sidecar-schema` | request | +| `X-Capsule-Protocol-Min` | `x-capsule-protocol-min` | response metadata | +| `X-Capsule-Protocol-Max` | `x-capsule-protocol-max` | response metadata | +| `X-Capsule-Min-Client-Build` | `x-capsule-min-client-build` | response metadata | + +Credentials map the same way: the REST `Authorization: Bearer` access token and the federation capability token both ride the gRPC `authorization` metadata key unchanged — one token format ([EdDSA-JWT](/design/authentication/#access-token)), two carriages. + +## Rejection Mapping + +REST rejections are HTTP statuses; gRPC rejections are gRPC status codes. The mapping is deliberately coarse — the precise discriminator on **both** transports is the machine-readable [`error.*` code](/design/i18n/#server-error-codes), carried in the REST `ApiError` body and in the gRPC status detail (plus the `x-capsule-error-code` trailing metadata). + +| Rejection class | REST | gRPC | +| ---------------------------------------------- | ----- | --------------------- | +| Structural (bad envelope, unknown enum, sizes) | `400` | `INVALID_ARGUMENT` | +| Unauthenticated / expired token | `401` | `UNAUTHENTICATED` | +| Unauthorized (capability, quota-hard, suspend) | `403` | `PERMISSION_DENIED` | +| Not found (and indistinguishable-404 surfaces) | `404` | `NOT_FOUND` | +| Stale state (chain/cursor/directory regress) | `409` | `FAILED_PRECONDITION` | +| Payload too large | `413` | `RESOURCE_EXHAUSTED` | +| Protocol version outside `[Min, Max]` | `426` | `FAILED_PRECONDITION` | +| Rate limited | `429` | `RESOURCE_EXHAUSTED` | + +Where two rejection classes share a gRPC code (`409`/`426`, `413`/`429`), the `error.*` code disambiguates — a client switches on the code, never on the transport status alone. + +## Validation + +- **Handshake parity (unit).** For each fail-closed rule in [Protocol and Capability Negotiation](/design/threat-model/validation/#protocol-and-capability-negotiation), drive the same bad input through a REST route and the gRPC service; assert the rejection maps per the table above and carries the same `error.*` code. +- **Metadata round-trip (unit).** Send the universal values as gRPC metadata; assert the server's negotiation gate reads them identically to the REST headers. +- **Capability carriage (unit).** Present the same federation capability as a REST bearer and as gRPC `authorization` metadata; assert identical verification outcomes. diff --git a/capsule-docs/src/content/docs/design/federation.md b/capsule-docs/src/content/docs/design/federation.md index 1390b9b1..a5546707 100644 --- a/capsule-docs/src/content/docs/design/federation.md +++ b/capsule-docs/src/content/docs/design/federation.md @@ -5,7 +5,7 @@ description: How Capsule servers share albums across users on different home ser Federation lets an album owned on one Capsule server be shared with users whose accounts live on another. This document covers **server-to-server** federation only; direct device-to-device sync for a single user is [Peering](/design/peering/). -Federation reuses the existing read primitives — `/sync`, `/blob/{hash}`, the standard manifest envelope. The only new things federation introduces are a **capability token** (the contract that gates which peers may fetch what) and a **per-peer compartmentalization layer**. Planned to live in `capsule-api-sync::federation` (today a stub crate): capability issuance, verification, the pull path, and per-peer rate budgeting. +Federation reuses the existing read primitives — the sync feed, `/blob/{hash}`, the standard manifest envelope. The only new things federation introduces are a **capability token** (the contract that gates which peers may fetch what) and a **per-peer compartmentalization layer**. Planned to live in `capsule-api-sync::federation` (today a stub crate): capability issuance, verification, the pull path, and per-peer rate budgeting. ## Threat Model @@ -15,13 +15,15 @@ This extends the security posture established in the [cryptography](/design/cryp ## Federation Reuses Existing Primitives -Federation deliberately introduces **no new data protocol**. A remote server fetches exactly the same content-addressed primitives a client uses (see [Import — Download & Sync](/design/import/download-sync/#discovering-what-changed)): +Federation deliberately introduces **no new data protocol**. A remote server fetches exactly the same content-addressed primitives a client uses, over the same transports (see [Import — Download & Sync](/design/import/download-sync/#discovering-what-changed) and the [API surface map](/design/api-surfaces/#surface--transport-map)): -| Operation | Purpose | -| -------------------------- | --------------------------------------------------------------------------------------------- | -| `GET /sync` (album-scoped) | A page of metadata-blob changes after a cursor, for an album the peer holds a capability for. | -| `GET /blob/{hash}` | Fetch an opaque ciphertext blob by its content address. | -| `POST` capability proof | Present a [federation capability](#federation-capabilities) to establish or refresh access. | +| Operation | Transport | Purpose | +| ------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `Sync` (album-scoped) | gRPC (`capsule.sync.v1.SyncService`) | A page of metadata-blob changes after a cursor, for an album the peer holds a capability for. | +| `GET /blob/{hash}` | REST (HTTP `Range`) | Fetch an opaque ciphertext blob by its content address. | +| Capability presentation | gRPC `authorization` metadata / REST bearer | Present a [federation capability](#federation-capabilities) to establish or refresh access. | + +Rejection semantics are stated in REST terms throughout this doc; on the gRPC feed they map to gRPC status codes per [API Surfaces — Rejection Mapping](/design/api-surfaces/#rejection-mapping). Everything else — notifications, presence — rides a separate, lower-trust channel and never feeds the validation pipeline directly. diff --git a/capsule-docs/src/content/docs/design/import/download-sync.md b/capsule-docs/src/content/docs/design/import/download-sync.md index 42383a46..a213a688 100644 --- a/capsule-docs/src/content/docs/design/import/download-sync.md +++ b/capsule-docs/src/content/docs/design/import/download-sync.md @@ -5,22 +5,22 @@ description: How Capsule clients discover changes, fetch blobs on demand, and au Download is the inverse of [upload](/design/import/upload-protocol/), and rests on the same two foundations: blobs are **content-addressed by ciphertext hash**, and the server never holds a key, so it serves only opaque ciphertext. Where the upload path optimises for correctness under interruption, the download path optimises for **bandwidth and storage frugality** — a client fetches the smallest representation that satisfies the user's current intent, and nothing more. -The download client lives in `capsule-sdk` (per-platform glue handles cache placement and connection-class detection); the server side — the sync feed and blob fetch — lives in `capsule-api-sync`. The `/sync` feed format is the **contract** other modules consume; its versioning and per-album monotonic ordering are what defeats the stale-rewind attack class. +The download client lives in `capsule-sdk` (per-platform glue handles cache placement and connection-class detection; planned). Server-side, the two surfaces ride different transports per the [API surface map](/design/api-surfaces/#surface--transport-map): the **sync feed** is the gRPC `capsule.sync.v1.SyncService` in `capsule-api-sync` (a stub today), and **blob fetch** is a REST ranged `GET` served by `capsule-api-media`. The sync feed format is the **contract** other modules consume; its versioning and per-album monotonic ordering are what defeats the stale-rewind attack class. ## Discovering What Changed A client never polls assets individually. It holds a single opaque **sync cursor** and asks the server for everything that changed after it: -| Method | Path | Purpose | -| ------ | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -| `GET` | `/sync` | Returns a page of asset changes (created, metadata-updated, deleted) after `cursor`, with a `next_cursor`. The feed is monotonic and resumable. | -| `GET` | `/blob/{hash}` | Fetch a ciphertext blob by its content address. Supports HTTP `Range` for resumable and partial reads. | +| Surface | Transport | Purpose | +| --------------------------------------------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| `Sync(cursor, page_size)` | gRPC (`capsule.sync.v1.SyncService`) | Returns a page of asset changes (created, metadata-updated, deleted) after `cursor`, with a `next_cursor`. The feed is monotonic and resumable. | +| `GET /blob/{hash}` | REST (HTTP `Range`) | Fetch a ciphertext blob by its content address; ranged for resumable and partial reads. | -The `/sync` feed carries only the small encrypted **metadata blobs** and each asset's **blob manifest** — the content hashes of its original and derivative blobs — never original or derivative bytes. Discovering a thousand new assets costs a few hundred kilobytes. The client decrypts each metadata blob, learns the asset's dimensions, capture date, and LQIP, and only *then* decides what else, if anything, to fetch. A deleted or modified asset arrives as a tombstone or an updated metadata reference; the client reconciles local state against it (see [Synchronization Scope](#synchronization-scope)). +Each sync entry carries the asset's signed manifest as **opaque canonical CBOR** (never re-modeled as proto fields — re-encoding would detach it from its signatures), the small encrypted **metadata blob**, and the asset's **blob manifest** — the content hashes of its original and derivative blobs — never original or derivative bytes. Discovering a thousand new assets costs a few hundred kilobytes. The client decrypts each metadata blob, learns the asset's dimensions, capture date, and LQIP, and only *then* decides what else, if anything, to fetch. A deleted or modified asset arrives as a tombstone or an updated metadata reference; the client reconciles local state against it (see [Synchronization Scope](#synchronization-scope)). -**Cursor authenticity.** The opaque sync cursor is **MAC'd by the server** (HMAC-SHA256) under a server-only key and verified on every `/sync` (and [federation pull](/design/federation/#federation-reuses-existing-primitives)) request, so a client cannot forge or mutate a cursor and a cursor lifted from another context is rejected at the boundary. The MAC is the *authenticity* layer; the per-album monotonic `sync_seq` check below is the independent *anti-rewind* layer. They are separate on purpose: a malicious server can always hand back one of its own *older*, validly-MAC'd cursors, and only the client-held high-water mark defeats that. Together they close the [sync-cursor rewind class](/design/threat-model/scenarios/#damage-scenario--invariant-map). +**Cursor authenticity.** The opaque sync cursor is **MAC'd by the server** (HMAC-SHA256 — a server-internal construction: the cursor is opaque to clients and never verified by them, so it sits outside the client-facing [primitives inventory](/design/cryptography/primitives/#primitives-inventory)) under a server-only key and verified on every `Sync` call (and [federation pull](/design/federation/#federation-reuses-existing-primitives)), so a client cannot forge or mutate a cursor and a cursor lifted from another context is rejected at the boundary. The MAC is the *authenticity* layer; the per-album monotonic `sync_seq` check below is the independent *anti-rewind* layer. They are separate on purpose: a malicious server can always hand back one of its own *older*, validly-MAC'd cursors, and only the client-held high-water mark defeats that. Together they close the [sync-cursor rewind class](/design/threat-model/scenarios/#damage-scenario--invariant-map). -**Sync feed validation.** Every entry in a `/sync` response carries a `protocol_version` (matching the album's pin) and a per-album monotonic `sync_seq`. The client refuses to apply an entry whose `protocol_version` is above its max known (per the [tightened Postel's Law](/design/principles/#postels-law-asymmetric)) and refuses any page whose `sync_seq` regresses against what the client has already seen for that album — a regressing `sync_seq` indicates a malicious or buggy server attempting to rewind the client's view, and the client surfaces it rather than applying it. +**Sync feed validation.** Every entry in a `Sync` response carries a `protocol_version` (matching the album's pin) and a per-album monotonic `sync_seq` (a `u64`, strictly increasing per album). The client refuses to apply an entry whose `protocol_version` is above its max known (per the [tightened Postel's Law](/design/principles/#postels-law-asymmetric)) and refuses any page whose `sync_seq` regresses against what the client has already seen for that album — a regressing `sync_seq` indicates a malicious or buggy server attempting to rewind the client's view, and the client surfaces it rather than applying it. ## Stale-Revival Detection diff --git a/capsule-docs/src/content/docs/design/import/pipeline.md b/capsule-docs/src/content/docs/design/import/pipeline.md index bc3a82bb..9d23dfc8 100644 --- a/capsule-docs/src/content/docs/design/import/pipeline.md +++ b/capsule-docs/src/content/docs/design/import/pipeline.md @@ -103,4 +103,4 @@ What the rest of the system depends on this module for: - **Streaming release gating (smoke).** Run a streaming import with `/storage/verify` mocked: assert each local original (and Move-mode source) is released *only* after its `durable` verdict, and that a non-`durable` verdict leaves the local copy in place. - **Streaming halt-on-disconnect (smoke).** Drop the server connection mid-stream; assert the pipeline stops admitting new source files into the library (no unbounded local growth) and resumes uploading via `HEAD` on reconnect without re-importing completed assets. -The cross-module case — pipeline → upload protocol → server finalization → assets visible in `/sync` — is bounded E2E surface listed in [Module Map](/design/module-map/#e2e-test-surface). +The cross-module case — pipeline → upload protocol → server finalization → assets visible in the sync feed — is bounded E2E surface listed in [Module Map](/design/module-map/#e2e-test-surface). diff --git a/capsule-docs/src/content/docs/design/import/storage-verification.md b/capsule-docs/src/content/docs/design/import/storage-verification.md index 1362f040..89893527 100644 --- a/capsule-docs/src/content/docs/design/import/storage-verification.md +++ b/capsule-docs/src/content/docs/design/import/storage-verification.md @@ -82,7 +82,7 @@ A non-`durable` verdict never triggers a destructive action: the client retains - **vs. upload finalization.** Finalization's `Completed` is a one-time receipt for one transfer; `/storage/verify` is the re-checkable, after-the-fact confirmation that the receipt still holds — across app restarts, across devices, and after server-side GC or migration could have changed state. - **vs. `verify_asset`.** `verify_asset` is local and key-aware (signatures, provenance chain, write authorization); `/storage/verify` is remote and key-free (does the server physically have and serve these bytes). Neither subsumes the other. -- **vs. the sync feed.** `/sync` tells a client *what exists*; `/storage/verify` tells it *whether a specific blob is durably serveable right now* — the question that gates destruction. +- **vs. the sync feed.** The sync feed tells a client *what exists*; `/storage/verify` tells it *whether a specific blob is durably serveable right now* — the question that gates destruction. ## Validation diff --git a/capsule-docs/src/content/docs/design/index.md b/capsule-docs/src/content/docs/design/index.md index 12cb88bc..74c33392 100644 --- a/capsule-docs/src/content/docs/design/index.md +++ b/capsule-docs/src/content/docs/design/index.md @@ -9,12 +9,12 @@ Capsule is an end-to-end-encrypted personal photo and media store with optional The design stacks in layers, each building on the one below — the sidebar groups follow this order: -- **Foundations** — the [core principles](/design/principles/) every component obeys, and the [module map](/design/module-map/) from code module to owning doc. -- **Cryptography** — the [primitives](/design/cryptography/primitives/) inventory, the [key hierarchy](/design/cryptography/keys/), [MLS](/design/cryptography/mls/) group membership, asset/metadata [encryption](/design/cryptography/encryption/), and signed [provenance](/design/cryptography/provenance/). The server holds only opaque ciphertext — never a key. +- **Foundations** — the [core principles](/design/principles/) every component obeys, the [module map](/design/module-map/) from code module to owning doc, and the [API surfaces](/design/api-surfaces/) map from server surface to transport. +- **Cryptography** — the [primitives](/design/cryptography/primitives/) inventory, the [key hierarchy](/design/cryptography/keys/), [MLS](/design/cryptography/mls/) group membership with its [resilience contracts](/design/mls-resilience/), asset/metadata [encryption](/design/cryptography/encryption/), and signed [provenance](/design/cryptography/provenance/). The server holds only opaque ciphertext — never a key. - **Identity & access** — [authentication](/design/authentication/), [authorization](/design/authorization/), and [device enrollment](/design/device-enrollment/). -- **Storage** — the [server](/design/filesystem/server/) and [client](/design/filesystem/client/) filesystems, the [metadata](/design/metadata/) sidecar schema, and [thumbnails](/design/thumbnails/). -- **Import & sync** — the [import pipeline](/design/import/pipeline/), [upload protocol](/design/import/upload-protocol/), [download & sync](/design/import/download-sync/), [backup](/design/backup-recovery/), and [versioning](/design/versioning/). -- **Sharing & federation** — server-to-server [federation](/design/federation/), device-to-device [peering](/design/peering/), [share links](/design/share-links/), and [moderation](/design/moderation/). +- **Storage** — the [server](/design/filesystem/server/) and [client](/design/filesystem/client/) filesystems, the [metadata](/design/metadata/) sidecar schema, [thumbnails](/design/thumbnails/), and [quota](/design/quota/). +- **Import & sync** — the [import pipeline](/design/import/pipeline/), [upload protocol](/design/import/upload-protocol/), [download & sync](/design/import/download-sync/), [storage verification](/design/import/storage-verification/), [backup](/design/backup-recovery/), and [versioning](/design/versioning/). +- **Sharing & federation** — server-to-server [federation](/design/federation/), device-to-device [peering](/design/peering/), [share links](/design/share-links/), [web upload](/design/web-upload/) for guest drops, and [moderation](/design/moderation/). - **Organization & clients** — [albums and stacks](/design/organization/), native [client duties](/design/clients/), [internationalization](/design/i18n/), and on-device [AI/ML](/design/ai/). - **Threat model** — the cross-cutting [damage-scenario map](/design/threat-model/scenarios/), [validation invariants](/design/threat-model/validation/), and [schema rules](/design/threat-model/schema-rules/) that bound what a faulty or hostile client can do. diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 5adbd75d..1b32cca2 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -16,7 +16,7 @@ The mapping reflects the *design intent*. Modules not yet implemented are annota | `capsule-core-ffi` | uniffi bindings crate for native Swift/Kotlin consumers (sidecar `Catalog` surface; the `HardwareSigner` foreign trait ships under `capsule-core`'s `ffi` feature) | | `capsule-api` | Server entry-point + routing | | `capsule-api-auth` | Authentication, sessions, OIDC, device directory | -| `capsule-api-library` | GraphQL API for UI queries (assets, albums, search) | +| `capsule-api-library` | **Legacy** GraphQL API for UI queries — frozen and retiring; rich queries move client-side onto `library.sqlite` (see [API Surfaces](/design/api-surfaces/#legacy-graphql-retiring)) | | `capsule-api-upload` | TUS-like resumable upload protocol server | | `capsule-api-media` | Media serving (ciphertext blobs, public shares) | | `capsule-api-sync` | gRPC sync API + federation | @@ -80,8 +80,8 @@ Hardware-key adapters do **not** live in `capsule-sdk`: the `Signer`/`HardwareSi | `capsule-api` (routing) | [Filesystem — Server](/design/filesystem/server/) | Smoke | | `capsule-api-auth::{oidc,session,claims,roles}` | [Authentication](/design/authentication/), [Authorization](/design/authorization/) | Unit + Smoke (testcontainer Postgres/Redis) | | `capsule-api-auth::devices` (planned for enrollment) | [Device Enrollment](/design/device-enrollment/) | Smoke | -| `capsule-api-library::schema::*` | [Metadata](/design/metadata/), [Organization](/design/organization/) | Smoke (GraphQL) | -| `capsule-api-library::loaders` | [Filesystem — Server](/design/filesystem/server/) | Unit (DataLoader) | +| `capsule-api-library::schema::*` (legacy, retiring) | [API Surfaces](/design/api-surfaces/#legacy-graphql-retiring) | frozen (no new surface) | +| `capsule-api-library::loaders` (legacy, retiring) | [API Surfaces](/design/api-surfaces/#legacy-graphql-retiring) | frozen (no new surface) | | `capsule-api-upload` | [Import — Upload Protocol](/design/import/upload-protocol/) | Unit + Smoke + 1 E2E | | `capsule-api-media::routes` | [Filesystem — Server](/design/filesystem/server/), [Thumbnails](/design/thumbnails/) | Smoke | | `capsule-api-media::verify` (planned) | [Import — Storage Verification](/design/import/storage-verification/) | Unit + Smoke | @@ -159,9 +159,9 @@ Target count: ≤ 13 cases. Each one is named by what it proves — not "test X" **Status:** none of the 13 cases is runnable today — every one crosses at least one planned surface (the networked server/client wiring, MLS, or ML), even where its `capsule-core` half is implemented and unit/smoke-tested. The bound starts governing with the first implementation slice that makes a case live; until then this list is the contract the slices build toward. -1. **Auth → Library query.** Log in via OIDC → access-token → GraphQL query for own albums returns expected list. Covers `capsule-api-auth::oidc + session` × `capsule-api-library::schema`. +1. **Auth → sync → client-side library query.** Log in via OIDC → access token → gRPC `Sync` returns the account's album entries → the client applies them and a local `library.sqlite` query lists the expected albums (rich queries are client-side per [API Surfaces](/design/api-surfaces/)). Covers `capsule-api-auth::oidc + session` × `capsule-api-sync` × `capsule-core::library`. 2. **Full import + upload + finalize.** Local scan → plan → execute → upload session → finalize → blob present at `blobs/{hash}` + index row marked uploaded. Covers `capsule-core::import` × `capsule-sdk::upload` × `capsule-api-upload` × `capsule-api-entity`. -3. **Sync feed pickup.** Upload from device A → device B's `/sync` advances → device B fetches metadata blob and (per scope) the original. Covers `capsule-api-sync` × `capsule-sdk` download path × `capsule-core::library` write. +3. **Sync feed pickup.** Upload from device A → device B's sync feed advances → device B fetches metadata blob and (per scope) the original. Covers `capsule-api-sync` × `capsule-sdk` download path × `capsule-core::library` write. 4. **Federation cross-server pull.** Alice on `home.tld` shares to Bob on `other.tld` → capability token → Bob's server pulls metadata + blobs → Bob's client renders. Covers `capsule-api-sync::federation` (both sides) × `capsule-api-auth` (capability issue). 5. **LAN peering A→B.** Two devices on the same LAN; mDNS discovery → TLS handshake → delta-scoped artifact → restore on receiver → byte-equal libraries. Covers `capsule-sdk::peering` × `capsule-core::backup` × `capsule-core::library`. 6. **Backup → restore on a fresh device.** Export full backup → bootstrap new device via passphrase + escrow → import backup → assert every asset present and verifiable. Covers `capsule-core::backup` × `capsule-core::crypto::keys` × `capsule-core::library`. diff --git a/capsule-docs/src/content/docs/design/peering.md b/capsule-docs/src/content/docs/design/peering.md index d71b595a..b7657412 100644 --- a/capsule-docs/src/content/docs/design/peering.md +++ b/capsule-docs/src/content/docs/design/peering.md @@ -49,7 +49,7 @@ This doc covers sync between devices that are **already provisioned** — both a ## Determining the Delta -Before building an artifact, the two devices must agree on what is missing. Peering reuses the [sync cursor](/design/import/download-sync/#discovering-what-changed) model rather than inventing a diff: each side offers its set of held [ciphertext content addresses](/design/cryptography/primitives/) and its cursor, and the delta is the complement. "What changed" is already defined by the `/sync` feed — peering borrows that definition wholesale. +Before building an artifact, the two devices must agree on what is missing. Peering reuses the [sync cursor](/design/import/download-sync/#discovering-what-changed) model rather than inventing a diff: each side offers its set of held [ciphertext content addresses](/design/cryptography/primitives/) and its cursor, and the delta is the complement. "What changed" is already defined by the sync feed — peering borrows that definition wholesale. ## What Moves Over the Wire diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 497f170d..0dc97330 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -64,6 +64,8 @@ The owner docs are: | Client validation duties + sandboxed decoder | [Clients](/design/clients/) | | Translation catalog format + locale resolution + error-code scheme | [Internationalization](/design/i18n/) | | Code module → design doc mapping + bounded E2E test surface | [Module Map](/design/module-map/) | +| API surface ↔ transport map + cross-transport negotiation/rejection mapping | [API Surfaces](/design/api-surfaces/) | +| Storage durability verdict + verify-before-destroy rule | [Import — Storage Verification](/design/import/storage-verification/) | **Permitted secondary mentions.** Mechanism-explanatory phrasing inside a non-owner doc is fine — for example, "STREAM tags catch chunk reordering" inside [Peering](/design/peering/) is explaining a *behavior*, not declaring a *choice*. What the rule forbids is restating the choice itself ("we use SHA-256") outside the owner doc. When in doubt, link. diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index e9a8ddbb..f3d14be4 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -51,7 +51,7 @@ Invariants carry **stable numbers** (referenced across docs as "invariant 17", " - **20.** All checks (1)–(18) re-applied — federation does not unlock looser rules. - **21.** Per-peer rate budgets unbroken (events/hour, bytes/hour, CPU/hour). Otherwise `429`. -### On the `/sync` feed, directory publish, and federated reports +### On the sync feed, directory publish, and federated reports - **22.** The `sync_cursor` carries a server MAC under a server-only key; a forged or mutated cursor is rejected (`400`). This is the authenticity layer; the client independently enforces per-album `sync_seq` monotonicity (client-side invariants below). Owner: [Import — Download & Sync](/design/import/download-sync/#discovering-what-changed). - **23.** A published `DeviceDirectory` has `directory_version` **strictly greater** than the version currently stored for that user, and the master signature covers it. A non-advancing or regressing publish is rejected (`409`). Owner: [Cryptography — Device Directory](/design/cryptography/keys/#device-directory). @@ -97,6 +97,8 @@ Mirror checklist that every client implements before applying any received data Every versioned API surface — client-to-server uploads, sync feed, federation pull, peering — runs the same compatibility gate. The gate is **fail-closed**: a mismatch is a hard reject before any state is written, never a silent degrade. +The rules are stated once, in REST terms (headers + HTTP statuses). On the gRPC surfaces the same values ride call metadata and the same rejections map onto gRPC status codes per [API Surfaces](/design/api-surfaces/#negotiation-across-transports) — one gate, two carriages. + ### Universal Headers | Header | Sent by | Meaning | diff --git a/capsule-docs/src/content/docs/design/web-upload.md b/capsule-docs/src/content/docs/design/web-upload.md index 6b9628b0..d7142138 100644 --- a/capsule-docs/src/content/docs/design/web-upload.md +++ b/capsule-docs/src/content/docs/design/web-upload.md @@ -40,7 +40,7 @@ Out of scope for v1 (deliberate non-goals): These are **normative** — the security-relevant decisions are committed; only UX presentation remains open. - **Upload-link URL format.** `https://server.tld/u/{opaque-id}#{drop_pubkey}`. `{opaque-id}` is a **random 128-bit value** from the CSPRNG — full 128-bit entropy, *not* a UUIDv7 or other structured id (identical rule to the [share-link opaque-id](/design/share-links/#security-contract)); it is fully opaque and carries no scope. `{drop_pubkey}` is the Drop Key public half, carried in the **fragment** so the server never receives it (see [Server-blind](#two-confidentiality-properties)). -- **Drops never enter the library.** A drop is written only to the provisioning user's **inbox**; it is never an album asset, never appears in any album member's [`/sync`](/design/import/download-sync/#discovering-what-changed) feed, and is served only to the provisioning user's own authenticated devices. A drop carries **no `AssetManifest`** — no `device_sig`, no `write_sig`, no `album_id`, no provenance — and therefore never flows through [`verify_asset`](/design/cryptography/keys/#write-authorization). Library state is reachable only through adoption by a trusted client. +- **Drops never enter the library.** A drop is written only to the provisioning user's **inbox**; it is never an album asset, never appears in any album member's [sync feed](/design/import/download-sync/#discovering-what-changed), and is served only to the provisioning user's own authenticated devices. A drop carries **no `AssetManifest`** — no `device_sig`, no `write_sig`, no `album_id`, no provenance — and therefore never flows through [`verify_asset`](/design/cryptography/keys/#write-authorization). Library state is reachable only through adoption by a trusted client. - **Per-link caps are enforced server-side at the no-key layer.** Expiry, cumulative-byte cap, file-count cap, and per-file size cap are checked on every drop-session creation; an over-cap or expired/revoked link is refused. These bound a leaked link to *wasted quota and inbox space*, never to library corruption. - **Quota is charged to the provisioning user at drop-session creation.** A drop debits the link owner's quota at session creation — the single hard [enforcement point](/design/quota/#enforcement-points) — using the [`upload_user_id = owner_id` attribution](/design/quota/#accounting-model). A link cannot be used to push the owner past their hard limit. - **Serving-endpoint rate limits.** Drop-session creation is rate-limited **per source IP and per `{opaque-id}`** (two independent limiters), and a not-found, revoked, or expired link returns an **indistinguishable `404`** — never `410 Gone` — exactly as the [share-link serve path](/design/share-links/#security-contract), so probing reveals nothing. @@ -154,7 +154,7 @@ The drop sealing and adoption rewrap live in `capsule-core::drop` (so they apply - **Opaque-id entropy (unit).** Assert generated upload-link ids are ≥128-bit and non-sequential, identical to the [share-link check](/design/share-links/#validation). - **Adoption rewrap accepts (unit).** Decapsulate a drop, rewrap `K` under a test AMK, build the `create` manifest with `key_mode = wrapped`; assert [`verify_asset`](/design/cryptography/keys/#write-authorization) accepts and a second member can unwrap `wrapped_file_key` and STREAM-decrypt the unchanged ciphertext. - **Wrapped-mode negative cases (unit).** A manifest with `key_mode = wrapped` but a forged `wrapped_file_key` (failing the `metadata_blob_hash` binding), or `key_mode` disagreeing with the manifest, is rejected at `verify_asset`. Owned alongside the [Provenance negative-case suite](/design/cryptography/provenance/#validation). -- **No library injection (unit).** Assert a `DropDescriptor` cannot be presented on any album-write path: the drop endpoints accept no `album_id`/manifest fields, and the inbox is never emitted on `/sync`. +- **No library injection (unit).** Assert a `DropDescriptor` cannot be presented on any album-write path: the drop endpoints accept no `album_id`/manifest fields, and the inbox is never emitted on the sync feed. - **Drop session lifecycle (smoke).** Against a real Postgres + blob store: open a drop session, exceed each per-link cap in turn, exhaust the owner's quota, and probe a revoked/expired/unknown link — assert caps and quota refuse, the rate limiter engages, and not-found/revoked/expired all return an indistinguishable `404`. - **Adoption atomicity (smoke).** Inject a crash between blob promotion and the Postgres commit; assert the asset row, provenance record, and inbox-row deletion either all land or all roll back — no half-adopted drop, no orphaned blob, no zombie inbox row. From 94291a071f2fab2a8d59ccd4557d8f723cc39f0a Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:16:02 -0400 Subject: [PATCH 03/58] docs(design): fix data-plane contract defects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Several contracts in the crypto/metadata SSoT docs were ambiguous or self-contradictory in ways an implementer could not code against: - metadata: provenance_chain_hash was defined as the hash of the latest provenance record — a structure that itself commits to a hash of the sidecar, i.e. a hash containing itself. Redefine it as the PRIOR chain head (always equal to the sealing manifest's prior_provenance_hash) and pin the author/seal/sign ordering in a new Provenance Binding and Sealing Order section - metadata: declare the single canonical asset identity (sidecar uuid = manifest file_id = asset_id = blob_id, one UUIDv7) plus the UUID version per identifier; enumerate the previously undeclared content_type and GpsSource closed value sets; bound caption_lww; define the add_id reseed source when local sidecars are absent; and state how the op-log path preserves _unknown byte-fidelity - provenance: metadata_blob_hash becomes Option with explicit presence-by-action; normative wire-presence rules (legacy options encode present-null, later v1 fields encode absent-key — absent key_mode means derived) since presence is signature-visible in canonical CBOR; DerivativeManifest gains the amk_version and protocol_version it needs to be authorization-verifiable; the record's duplicated prior_provenance_hash becomes a checked mirror invariant; chain-walk cost bounded via the cached verified head - provenance + filesystem/server: resolve the manifest's contradicted physical location — a small server-visible envelope object stored content-addressed beside fully opaque ciphertext blobs (never an embedded header), which is also what the keyless index rebuild walks; the server-side provenance chain is that envelope sequence, not an encrypted blob the no-key invariants could never read - encryption: normative ciphertext offset for ranged reads (chunk_index x 65,536 — the plaintext-stride confusion corrupts every ranged read); wrapped-key mode marked planned; the AlbumKeyDistribution message re-homed to its MLS owner - keys: verify_asset takes the ciphertext hash, never the bytes; sponsoree recovery matrix stated (all paths route through the sponsor); directory TOFU first-contact corroborated out-of-band; KDF formulas single-owned by encryption - primitives: crypto_suite_id vs MLS ciphersuite codepoint namespaces disambiguated - maintenance + client: rewrite bundles stage the displaced prior version aside so rollback restores rather than deletes; date buckets fixed at import (capture-date edits do not relocate files); one suite id governs both digests in lockstep - backup: deterministic export requires normalized tar headers - versioning: ceremony deadline evaluated on the server's trusted clock; frozen_state_hash follows the suite hash; MLS group naming deferred to the MLS layer --- .../content/docs/design/backup-recovery.md | 2 +- .../docs/design/cryptography/encryption.md | 6 +-- .../content/docs/design/cryptography/keys.md | 13 +++--- .../content/docs/design/cryptography/mls.md | 2 +- .../docs/design/cryptography/primitives.md | 2 + .../docs/design/cryptography/provenance.md | 40 ++++++++++++------- .../content/docs/design/filesystem/client.md | 2 +- .../docs/design/filesystem/maintenance.md | 6 +-- .../content/docs/design/filesystem/server.md | 10 ++--- .../src/content/docs/design/metadata.md | 34 ++++++++++++++-- .../src/content/docs/design/versioning.md | 6 +-- 11 files changed, 83 insertions(+), 40 deletions(-) diff --git a/capsule-docs/src/content/docs/design/backup-recovery.md b/capsule-docs/src/content/docs/design/backup-recovery.md index 133b9a16..f86f577b 100644 --- a/capsule-docs/src/content/docs/design/backup-recovery.md +++ b/capsule-docs/src/content/docs/design/backup-recovery.md @@ -43,7 +43,7 @@ The container properties below are what make it both safe and portable: - **Uncompressed.** Asset ciphertext is incompressible (it is the output of [AES-256-GCM-STREAM](/design/cryptography/primitives/#bulk-aead)); compressing it buys nothing and adds CPU cost. Metadata blobs are likewise encrypted before they hit the archive, so the same applies. - **Streamable.** Tar is append-friendly and has no central directory, so a backup of arbitrary size can be written and read end-to-end without seeking — important when exporting a terabyte-scale library to spinning rust or an external drive. -- **Deterministic ordering.** Entries are written in sorted order by `(album_id, asset_id, blob_role)`, so two exports of the same logical content produce byte-identical archives. This lets the integrity manifest's signature verify across re-exports. +- **Deterministic ordering.** Entries are written in sorted order by `(album_id, asset_id, blob_role)`, so two exports of the same logical content produce byte-identical archives. This lets the integrity manifest's signature verify across re-exports. Determinism also requires **normalized tar headers** — `mtime = 0`, `uid = gid = 0`, empty user/group names, fixed mode bits, plain ustar with no PAX extension headers — so every entry's bytes are a pure function of its content and path, never of the exporting machine. - **Top-level integrity manifest.** Written before any blob entry (right after the tiny `VERSION` header), `MANIFEST.cbor` lists every entry's path, [content hash](/design/cryptography/primitives/), declared size, and the exporting device's identity — so a streaming reader holds the full integrity list before the first blob arrives. The manifest is authenticated **two ways**: - An **HMAC** keyed by the backup's wrap key (derived from the user passphrase via the [password-based KDF](/design/cryptography/primitives/#password-based-kdf)) catches truncation, reordering, and corruption *before* any decrypt is attempted. - A **hybrid Ed25519 + ML-DSA-65 signature** from the exporting device's [DSK](/design/cryptography/keys/#device-keys) — the same [signature scheme](/design/cryptography/primitives/#signature-scheme) used for asset manifests. The signature defeats a symmetric-key attacker who could otherwise re-HMAC after tampering: an attacker who steals the wrap key can re-HMAC but cannot forge the device signature. diff --git a/capsule-docs/src/content/docs/design/cryptography/encryption.md b/capsule-docs/src/content/docs/design/cryptography/encryption.md index e4eb9a9b..98dfc606 100644 --- a/capsule-docs/src/content/docs/design/cryptography/encryption.md +++ b/capsule-docs/src/content/docs/design/cryptography/encryption.md @@ -46,7 +46,7 @@ file_key = HKDF_SHA512( The salt folds in `nonce_prefix` — the fresh 7-byte STREAM nonce prefix drawn for *this* encryption (below). Because a new `nonce_prefix` is drawn on **every** encryption of a file, the derived `file_key` is unique to that encryption even when `file_id` and `amk_version` are unchanged — so a same-epoch [`replace`](/design/authorization/#the-closed-action-set) (which keeps the same `file_id`) re-rolls the *key*, not merely the nonce. This is what lets the STREAM nonce safely start at zero per encryption: no `(file_key, nonce_prefix)` pair is ever reused across two encryptions of the same file, so AES-GCM nonce reuse is structurally impossible. See [Re-keying on Rewrite](#re-keying-on-rewrite). -**Wrapped file keys (external-origin assets).** The derivation above is the **derived** mode, where the file key is recomputed from the AMK and never stored. An asset a client did *not* author — a [web-upload drop](/design/web-upload/) adopted in place — arrives already encrypted under a random key `K` the guest chose, which no member can re-derive from the AMK. The adopting client therefore *carries* `K` wrapped under the AMK instead of deriving it: +**Wrapped file keys (external-origin assets; planned — lands with the [web-upload](/design/web-upload/) slice).** The derivation above is the **derived** mode, where the file key is recomputed from the AMK and never stored. An asset a client did *not* author — a [web-upload drop](/design/web-upload/) adopted in place — arrives already encrypted under a random key `K` the guest chose, which no member can re-derive from the AMK. The adopting client therefore *carries* `K` wrapped under the AMK instead of deriving it: ```rust wrap_key = HKDF_SHA512(ikm: AMK_v{amk_version}, salt: file_id || wrap_nonce, info: "asset-keywrap/v1", length: 32) @@ -55,7 +55,7 @@ wrapped_file_key = wrap_nonce || AES-256-GCM(wrap_key, plaintext: K) // 12-byt `wrapped_file_key` is stored in the signed [manifest](/design/cryptography/provenance/#asset-manifest) under [`key_mode = wrapped`](/design/cryptography/provenance/#asset-manifest). The fresh `wrap_nonce` is folded into the salt exactly as `nonce_prefix` is for derived keys, so no `(wrap_key, wrap_nonce)` pair repeats. Decryption is identical once `K` is recovered: a reader holding the AMK unwraps `wrapped_file_key` to obtain `K`, then runs the unchanged [STREAM construction](#stream-construction). Wrapped mode is set only by the adopting `create`; any later [`replace`](#re-keying-on-rewrite) re-encrypts the bytes under a freshly *derived* key. This is the sole case where a per-file key is stored rather than recomputed, and it is owned jointly with [Keys — Key Chain](/design/cryptography/keys/#key-chain). -AMKs are delivered over MLS application messages. When epoch N's MLS group is established, the creating device sends an `AlbumKeyDistribution { amk_version, amk_bytes }` message through MLS. Every current member's device receives and stores it locally (hardware-wrapped). +AMKs are delivered over MLS application messages; the distribution message's shape is owned by [MLS — History Delivery](/design/cryptography/mls/#history-delivery-for-new-joiners). Every current member's device receives the epoch's AMK over that channel and stores it locally (hardware-wrapped). **Distribution lag is expected and is not a failure.** An epoch bump and its `AlbumKeyDistribution` broadcast are separate MLS messages, so during a bump a device can legitimately receive an asset manifest referencing an `amk_version` whose key bytes have not yet arrived. A device that lacks the AMK for an `amk_version` that is otherwise **within the [MLS-attested epoch range](/design/cryptography/keys/#write-authorization)** treats the asset as *pending* — held and retried as MLS state catches up — rather than as a decryption failure or a forged manifest. Only an `amk_version` beyond the MLS-attested epoch, or one still missing after the retry timeout, is escalated. This is the `verify_asset` *pending* outcome and the matching [Failure Modes](/design/cryptography/failure-modes/#failure-mode-catalog) row; it is what keeps a concurrent upload during an epoch bump from being misread as an attack. @@ -73,7 +73,7 @@ Encrypting an asset for upload: Streaming download / ranged reads: - **Sequential:** `DecryptorBE32` consumes chunks in order, verifying each tag. -- **Ranged:** to start at plaintext byte `B`, compute `chunk_index = B / 65,520`. Because the [STREAM construction](#stream-construction) derives each chunk's nonce deterministically, chunk `i` decrypts independently given `file_key` and `i` — the server need only serve that 64 KiB ciphertext chunk, which the client decrypts and verifies. +- **Ranged:** to start at plaintext byte `B`, compute `chunk_index = B / 65,520` (the **plaintext** stride). Because the [STREAM construction](#stream-construction) derives each chunk's nonce deterministically, chunk `i` decrypts independently given `file_key` and `i` — the server need only serve the 64 KiB ciphertext chunk at **ciphertext byte offset `chunk_index × 65,536`** (the ciphertext stride, which includes each chunk's 16-byte tag), which the client decrypts and verifies. Mixing the two strides — serving from `chunk_index × 65,520` — corrupts every ranged read; the offset formula is normative. ### Re-keying on Rewrite diff --git a/capsule-docs/src/content/docs/design/cryptography/keys.md b/capsule-docs/src/content/docs/design/cryptography/keys.md index c67fd5b2..9fe0ec37 100644 --- a/capsule-docs/src/content/docs/design/cryptography/keys.md +++ b/capsule-docs/src/content/docs/design/cryptography/keys.md @@ -20,7 +20,8 @@ account_master_key (backed up — see Failure Modes) ├─ wraps device identity private keys (IK / DSK / DEK private halves) └─ anchors the encrypted backup that escrows: AMK_v{n} (random 32 bytes, per album, minted per MLS epoch) - └─ HKDF-SHA512(ikm=AMK_v{n}, salt=file_id||nonce_prefix, info="asset-file/v1") → 32-byte AES file key + └─ per-file key, derived fresh per encryption (formula owned by + Encryption — Asset Key Derivation) └─ AES-256-GCM-STREAM ``` @@ -31,7 +32,7 @@ Construction rules (consistent with the [KDF choice](/design/cryptography/primit - The 512-bit KDF output is truncated to 32 bytes for the AES-256 file key. - Each **encryption** gets a fresh derived key: the per-encryption `nonce_prefix` is folded into the `salt` (`file_id || nonce_prefix`; see [Encryption — Asset Key Derivation](/design/cryptography/encryption/#asset-key-derivation)), so even a same-`file_id` [`replace`](/design/authorization/#the-closed-action-set) under the same epoch re-rolls the key, never merely the nonce. The STREAM nonce can therefore safely start at zero per encryption, and no `(file_key, nonce_prefix)` pair is ever reused. -**Two file-key modes.** The derivation above is the **derived** mode — the default for every asset a client imports itself, where no per-file key is stored because the file key is recomputed from the AMK on demand. A second, **wrapped** mode exists for assets whose bytes a client did *not* author and whose key it therefore cannot re-derive: a [web-upload drop](/design/web-upload/) the user adopts in place arrives encrypted under a random key `K` the guest chose, so the adopting client *carries* `K` wrapped under the AMK rather than deriving it. The wrap key follows the same KDF discipline as the file key — `wrap_key = HKDF-SHA512(ikm=AMK_v{n}, salt=file_id || wrap_nonce, info="asset-keywrap/v1")`, with a fresh `wrap_nonce` folded into the salt so no `(wrap_key, wrap_nonce)` pair repeats (see [Encryption — Asset Key Derivation](/design/cryptography/encryption/#asset-key-derivation)). The mode is recorded in the signed [`key_mode`](/design/cryptography/provenance/#asset-manifest) manifest field; wrapped mode is set only at the adopting `create`, and any later `replace` re-encrypts under derived mode. Both modes feed the same [STREAM construction](/design/cryptography/encryption/#stream-construction) and the same [`verify_asset`](#write-authorization) authorization checks — the mode changes only how the decryption key is obtained, never who is authorized to write. +**Two file-key modes.** The derivation above is the **derived** mode — the default for every asset a client imports itself, where no per-file key is stored because the file key is recomputed from the AMK on demand. A second, **wrapped** mode exists for assets whose bytes a client did *not* author and whose key it therefore cannot re-derive: a [web-upload drop](/design/web-upload/) the user adopts in place arrives encrypted under a random key `K` the guest chose, so the adopting client *carries* `K` wrapped under the AMK rather than deriving it. The wrap key follows the same KDF discipline as the file key (the `asset-keywrap/v1` derivation, owned by [Encryption — Asset Key Derivation](/design/cryptography/encryption/#asset-key-derivation)), with a fresh `wrap_nonce` folded into the salt so no `(wrap_key, wrap_nonce)` pair repeats. The mode is recorded in the signed [`key_mode`](/design/cryptography/provenance/#asset-manifest) manifest field; wrapped mode is set only at the adopting `create`, and any later `replace` re-encrypts under derived mode. Both modes feed the same [STREAM construction](/design/cryptography/encryption/#stream-construction) and the same [`verify_asset`](#write-authorization) authorization checks — the mode changes only how the decryption key is obtained, never who is authorized to write. The master key also derives one **identifier** — the [default album](/design/organization/#the-default-album)'s `album_id`, via HKDF with a dedicated `info` label — so any device can recompute which album is the de facto default from the master key alone, even after recovery. This derives an *ID*, not a key: the default album is an ordinary album with a random per-epoch AMK like any other. @@ -94,10 +95,10 @@ This is **the** contract every consumer of `capsule-core::crypto` depends on. It - **Epoch ceiling is MLS-attested, not server-asserted.** The monotonic `amk_version` ceiling a client enforces is derived from the album's admin-signed [MLS commit chain](/design/cryptography/mls/#membership-operations) — the same authority that admits writers — never from the server's stored counter. A brand-new client learns the current epoch from the MLS group state delivered in its [Welcome](/design/cryptography/mls/#history-delivery-for-new-joiners), then rejects any manifest whose `amk_version` exceeds that MLS-attested epoch, so a server can neither fabricate a future epoch nor rewind to an old one and have a client honor it. The server's own no-key monotonicity check (invariant 18 in [Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants)) is a structural backstop, not the authority — it stops a *client* from skipping epochs, while MLS stops the *server* from fabricating them. - **What this accepts vs. rejects.** An asset signed by a writer who was *later* removed is still acknowledged — it was valid when written, and nothing after removal un-seeds it. An asset signed at an epoch where the signer lacked write capability is **rejected**: an attacker (or a buggy/colluding server) cannot produce a valid write-tier signature for an epoch they were not a writer in. - **Backdating buys nothing.** Ordering and authorization ride the provenance hash-chain and the `amk_version` epoch, never the self-asserted `timestamp` (see the timestamp note below). The "pre-sign backdated assets, then upload them after removal" attack therefore fails on the *epoch*, not the clock: a manifest must carry a `write_sig` for the epoch it claims, and a removed writer holds no write-tier key for any epoch past their removal. Anything they upload afterward either names an old epoch the chain has already advanced beyond (rejected by the monotonic `amk_version` + chain-head checks) or a new epoch they cannot sign for. -- **Single verification chokepoint.** All of this lives in one `verify_asset(manifest, ciphertext, authority)` function in `capsule-core::crypto`, where `authority` is the album's [`AlbumAuthority`](#write-authority-interface). It is the only path by which a client acknowledges an asset, and per [contract-driven development](/design/principles/) it ships with exhaustive negative test cases: reader-signed, removed-writer, wrong-epoch, forged certificate chain, replayed manifest. +- **Single verification chokepoint.** All of this lives in one `verify_asset(manifest, ciphertext_hash, authority)` function in `capsule-core::crypto`, where `authority` is the album's [`AlbumAuthority`](#write-authority-interface). It takes the **ciphertext hash**, never the bytes: hash confirmation is computed incrementally by the streaming path ([Provenance](/design/cryptography/provenance/#asset-manifest)), so verification never buffers a file. It is the only path by which a client acknowledges an asset, and per [contract-driven development](/design/principles/) it ships with exhaustive negative test cases: reader-signed, removed-writer, wrong-epoch, forged certificate chain, replayed manifest. - **Crypto validity, not durability.** `verify_asset` proves an asset is cryptographically valid and authorized; it says nothing about whether the *server* still durably holds the bytes. Confirming durable, indexed, retrievable server storage — the precondition for any destructive local cleanup — is a separate, key-free query, the [storage-verification endpoint](/design/import/storage-verification/). A safe local release requires **both** to pass. - **Defensive failure handling.** A verification failure is *never* silently dropped and *never* silently accepted. The asset is quarantined and surfaced in the [provenance/audit trail](/design/cryptography/provenance/#provenance-of-library-modifications) so an operator can distinguish a bug from an attack after the fact. This bounds the blast radius of an implementation bug. -- **Transient vs. terminal outcomes.** `verify_asset` returns one of three outcomes, not two: **accept**, **terminal-reject** (reader-signed, removed-writer, wrong-epoch, forged chain, suite-downgrade → quarantined as above), and **pending**. *Pending* is the narrow, recoverable case where the manifest's `amk_version` is within the MLS-attested epoch range but the corresponding AMK content key has not yet arrived over the in-band [AlbumKeyDistribution](/design/cryptography/encryption/#asset-key-derivation) message (an epoch bump whose key broadcast is still in flight). A pending asset is **held and retried** as MLS state catches up — never quarantined as forged and never accepted unverified — until the key arrives or a configurable timeout elapses, after which it escalates to a surfaced quarantine. This distinction stops an in-flight epoch bump from flagging honest concurrent uploads as attacks; see [Failure Modes](/design/cryptography/failure-modes/#failure-mode-catalog). +- **Transient vs. terminal outcomes.** `verify_asset` returns one of three outcomes, not two: **accept**, **terminal-reject** (reader-signed, removed-writer, wrong-epoch, forged chain, suite-downgrade → quarantined as above), and **pending**. *Pending* is the narrow, recoverable case where the manifest's `amk_version` is within the MLS-attested epoch range but the corresponding AMK content key has not yet arrived over the in-band [AlbumKeyDistribution](/design/cryptography/mls/#history-delivery-for-new-joiners) message (an epoch bump whose key broadcast is still in flight). A pending asset is **held and retried** as MLS state catches up — never quarantined as forged and never accepted unverified — until the key arrives or a configurable timeout elapses, after which it escalates to a surfaced quarantine. This distinction stops an in-flight epoch bump from flagging honest concurrent uploads as attacks; see [Failure Modes](/design/cryptography/failure-modes/#failure-mode-catalog). - **Downgrade-resistant.** Both signatures cover `crypto_suite_id`, `protocol_version`, and `prior_provenance_hash`. A manifest cannot be silently re-signed under a weaker suite or back-dated onto a different chain position without breaking either signature; an attempt is rejected at the same `verify_asset` chokepoint. - **Key delivery is authorization-neutral.** An adopted [web-upload drop](/design/web-upload/#why-adopt-in-place) carries its file key *wrapped* under the AMK ([`key_mode = wrapped`](/design/cryptography/provenance/#asset-manifest)) instead of deriving it, but it is authorized **identically** to any other write — `verify_asset` checks the adopter's `device_sig` and epoch `write_sig`, and the guest who supplied the bytes is never a signer. `key_mode` and `wrapped_file_key` are ordinary signed manifest fields covered by both signatures, so a tampered wrapped key, or a `key_mode` inconsistent with the manifest, is a terminal-reject like any other forged field. - **Timestamp is audit-only.** A manifest's `timestamp` is the client's self-asserted capture/write time and is **never** load-bearing for authorization or ordering — those ride the epoch and the chain above. The server stamps its own trusted `received_at` in the [server-visible envelope](/design/filesystem/server/#postgresql-what-the-server-knows) as the authoritative wall-clock for any time-based policy (retention, rate limits); the client `timestamp` is preserved verbatim for display and audit. A server-side *sanity* bound on `timestamp` is a gross-drift guard for honest clients, **not** an authorization control — it surfaces a wildly-wrong clock rather than silently distorting the audit trail. Grammar owned by [Threat Model — Schema Rules](/design/threat-model/schema-rules/) and mirrored in [Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). @@ -139,7 +140,7 @@ When Alice's device A1 adds Bob to an album, it fetches Bob's directory, verifie Concurrent edits (A1 and A2 trying to add different people simultaneously) are handled by MLS's proposal/commit ordering — one wins, the other re-proposes on top. OpenMLS exposes this. -**Monotonic version (anti-rollback).** The directory is the trust anchor every peer reads to learn which device keys are current, so a server that could silently serve a *stale* directory — one that still lists a revoked device, or omits a freshly-added one — would undo a revocation or hide a device. `directory_version` closes this: it is master-signed and **strictly monotonic**, and every reader (local client, [federated](/design/federation/) peer, [peering](/design/peering/) handshake) caches the highest version it has seen per user and **refuses a directory whose `directory_version` is below that high-water mark**, surfacing the regression rather than applying it. A reader with no cached version trusts-on-first-use and pins from there. This makes a revocation durable: once a peer has seen the post-revocation directory, the server cannot walk it back. The check is the directory-layer counterpart of the [stale-revival defense](/design/import/download-sync/#stale-revival-detection) for manifests, and is enforced as a [client-](/design/threat-model/validation/#client-side-validation-invariants) and [server-side](/design/threat-model/validation/#server-side-validation-invariants) invariant. +**Monotonic version (anti-rollback).** The directory is the trust anchor every peer reads to learn which device keys are current, so a server that could silently serve a *stale* directory — one that still lists a revoked device, or omits a freshly-added one — would undo a revocation or hide a device. `directory_version` closes this: it is master-signed and **strictly monotonic**, and every reader (local client, [federated](/design/federation/) peer, [peering](/design/peering/) handshake) caches the highest version it has seen per user and **refuses a directory whose `directory_version` is below that high-water mark**, surfacing the regression rather than applying it. A reader with no cached version trusts-on-first-use and pins from there. First contact is the one unprotected moment — a malicious server could serve an already-rolled-back directory to a brand-new reader — and it is corroborated out-of-band: a device enrolling into the *own* account receives the current directory over the verified [enrollment channel](/design/device-enrollment/), not from server TOFU, and cross-user first contact is backed by safety-number verification where the UI surfaces it. TOFU-then-pin therefore bounds the exposure to a first contact that skipped corroboration. This makes a revocation durable: once a peer has seen the post-revocation directory, the server cannot walk it back. The check is the directory-layer counterpart of the [stale-revival defense](/design/import/download-sync/#stale-revival-detection) for manifests, and is enforced as a [client-](/design/threat-model/validation/#client-side-validation-invariants) and [server-side](/design/threat-model/validation/#server-side-validation-invariants) invariant. The directory entry's `added_at` field is what blocks the damage scenario where a new device claims its key is older than the account itself: a server rejects an upload from a device whose `added_at` postdates the manifest's `timestamp`. See [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). @@ -165,6 +166,8 @@ A sponsored account is anchored under the sponsor's master key but holds its own The sponsor's *own* master key is untouched by any sponsoree revocation. The published revocation certificate is what clients and [federated](/design/federation/) peers check to refuse traffic from a revoked sponsoree. +**Sponsoree recovery matrix.** A sponsoree's keys are recoverable **only through its sponsor**: the sponsor unwraps the sponsoree KEK, which unwraps the sponsoree's private keys. A sponsoree has no recovery passphrase, no master-key escrow of its own, and none of the [seven recovery paths](/design/cryptography/failure-modes/#redundant-recovery-paths) independently — every path it has routes through the sponsor's master key, so **a sponsor who irrecoverably loses their master key loses every sponsoree's data with their own**. This is the same explicit, directional trust as the damage bound below, stated for recovery: a user who needs recovery that survives the sponsor must hold a registered account. + #### Damage bound under sponsor compromise A compromised sponsor holds every sponsoree's KEK, which wraps that sponsoree's private identity and device keys. It can therefore impersonate a sponsoree's device — forge its `device_sig`, append to or rewrite the sponsoree's [provenance history](/design/cryptography/provenance/#provenance-of-library-modifications), and write under the sponsoree's albums. Unlike a registered account — whose past records are protected even against a full current-key compromise because retired device keys are hardware-bound and non-recoverable — a sponsoree's history is **not** independent of its sponsor. This is inherent to delegation and is **bounded, not eliminated**: diff --git a/capsule-docs/src/content/docs/design/cryptography/mls.md b/capsule-docs/src/content/docs/design/cryptography/mls.md index f5e54738..e22b40b3 100644 --- a/capsule-docs/src/content/docs/design/cryptography/mls.md +++ b/capsule-docs/src/content/docs/design/cryptography/mls.md @@ -52,7 +52,7 @@ For first-device enrollment (a brand new account with no other device), see [Dev ## History Delivery for New Joiners -The one spot where the wrapper writes real custom code. Two patterns: +The one spot where the wrapper writes real custom code. AMK delivery — both the steady-state broadcast on an epoch bump and the batch form inside a `Welcome` — rides one application message whose shape this doc owns: `AlbumKeyDistribution { amk_version, amk_bytes }` ([Encryption](/design/cryptography/encryption/#asset-key-derivation) consumes it; the AMK itself is owned by [Keys](/design/cryptography/keys/#album-master-keys-amks)). Two delivery patterns: **Full history (recommended for shared albums):** Welcome message carries an encrypted blob of `[AMK_v1, AMK_v2, ..., AMK_current]`. The new joiner decrypts all, can now read every photo. diff --git a/capsule-docs/src/content/docs/design/cryptography/primitives.md b/capsule-docs/src/content/docs/design/cryptography/primitives.md index 2d088b51..e7520623 100644 --- a/capsule-docs/src/content/docs/design/cryptography/primitives.md +++ b/capsule-docs/src/content/docs/design/cryptography/primitives.md @@ -36,6 +36,8 @@ A faulty, malicious, or version-mismatched client could damage data by writing u `crypto_suite_id = 0x0001` denotes exactly the [Primitives Inventory](#primitives-inventory) above. Retiring any primitive (a broken SHA-256, a deprecated AEAD) **does not edit the row** — it adds a new row and a new suite id. An old AssetManifest carrying `0x0001` keeps verifying against the original row forever; new writes use the new suite id. This is the single-doc edit the inventory promises, generalized to the bundle. +`crypto_suite_id` values are a **Capsule-owned namespace**, unrelated to the MLS ciphersuite codepoint namespace: `0x0001` (the Capsule bundle) and `0x004D` (the OpenMLS id of the [MLS ciphersuite](#mls-ciphersuite) *inside* that bundle) are identifiers in two different registries, and neither implies the other. + The signatures on every manifest cover `crypto_suite_id` and `protocol_version`, so a downgrade-attempt (re-signing an existing manifest under a weaker suite) cannot be silently produced. ### Backward Compatibility diff --git a/capsule-docs/src/content/docs/design/cryptography/provenance.md b/capsule-docs/src/content/docs/design/cryptography/provenance.md index 7f5a8094..fc112866 100644 --- a/capsule-docs/src/content/docs/design/cryptography/provenance.md +++ b/capsule-docs/src/content/docs/design/cryptography/provenance.md @@ -22,9 +22,10 @@ AssetManifest { album_id: UUID, amk_version: u32, // identifies the AMK epoch + write-tier key ciphertext_hash: bytes, // content-address digest; algorithm fixed by crypto_suite_id; reused by upload protocol - metadata_blob_hash: bytes, // content-address of the asset's encrypted metadata blob (see Encryption); - // a ciphertext hash (server-visible, no plaintext leak); set on - // create | replace | metadata-update + metadata_blob_hash: Option, // content-address of the asset's encrypted metadata blob (see Encryption); + // a ciphertext hash (server-visible, no plaintext leak). PRESENT on + // create | replace | metadata-update; ABSENT (key omitted) on + // delete | derivative-add | derivative-replace | trash-restore plaintext_size: u64, chunk_size: u32, // plaintext bytes per chunk (65,520) nonce_prefix: [u8; 7], // STREAM nonce prefix, random per file @@ -49,18 +50,16 @@ AssetManifest { write_sig: Hybrid(Ed25519, ML-DSA-65), // under epoch write-tier key, over all fields above; both halves required } -AssetBlob { - manifest: AssetManifest, - chunks: [AES-256-GCM-STREAM encrypted chunks], -} ``` +**Physical placement.** The manifest is **not** embedded inside the ciphertext blob. Every ciphertext blob — original, derivative, metadata — is an independent, fully opaque content-addressed object ([the upload protocol](/design/import/upload-protocol/) uploads each under its own session), and the signed manifest travels beside them as the asset's **envelope object**: a small, deliberately server-visible signed CBOR object (it contains no plaintext secrets by construction) persisted twice on the server — in the PostgreSQL rows the hot path queries, and as a content-addressed object in the blob store that makes the index [reconstructible without PostgreSQL](/design/filesystem/server/#recovering-the-index-from-blobs-alone). The client persists the same manifests inside its local provenance chain file. Per blob role, the envelope's associations are by hash: the **original** and each **derivative** are named by `ciphertext_hash` in their (Asset/Derivative) manifest, and the **metadata blob** by the manifest's `metadata_blob_hash`. No ciphertext blob carries an embedded header. + The manifest carries **two signatures**, and a client acknowledges the asset only if **both** verify: 1. `device_sig` — hybrid Ed25519 + ML-DSA-65 by the uploading device's [DSK](/design/cryptography/keys/#device-keys). Provides provenance; the device certificate chains to the user IK via the [device directory](/design/cryptography/keys/#device-directory). 2. `write_sig` — a **hybrid Ed25519 + ML-DSA-65** signature under the epoch's [write-tier key](/design/cryptography/keys/#album-master-keys-amks); both halves must verify. Proves the signer held write authorization at `amk_version` (see [Write Authorization](/design/cryptography/keys/#write-authorization)). The signature being hybrid is what keeps its coverage of `crypto_suite_id` non-downgradable even if one algorithm is later broken. -The signed manifest is stored as the encrypted asset's header and is itself part of the [provenance record](#provenance-of-library-modifications). The same signing approach applies to other surfaces — [metadata blobs](/design/cryptography/encryption/#metadata-encryption) and the [device directory](/design/cryptography/keys/#device-directory) are each hybrid, device-signed, and versioned. +The signed manifest is stored as its own envelope object (see **Physical placement** above — never as a header inside the ciphertext) and is itself part of the [provenance record](#provenance-of-library-modifications). The same signing approach applies to other surfaces — [metadata blobs](/design/cryptography/encryption/#metadata-encryption) and the [device directory](/design/cryptography/keys/#device-directory) are each hybrid, device-signed, and versioned. **Streaming is preserved.** STREAM authentication tags verify every chunk *during* the stream. The manifest signature is a one-time provenance check. `ciphertext_hash` is computed incrementally as bytes arrive and confirmed at stream end — no separate pass, no buffering the whole file. @@ -70,6 +69,11 @@ The signed manifest is stored as the encrypted asset's header and is itself part The closed action enum is owned by [Authorization — The Closed Action Set](/design/authorization/#the-closed-action-set). +**Wire presence rules.** Presence is signature-visible in canonical CBOR (a present-`null` key and an absent key produce different signed bytes), so how each optional field encodes is normative, not a serializer detail: + +- The two options in v1's original schema — `prior_provenance_hash`, `retention_until` — encode as **present with `null`** when logically absent (the encoding existing signatures were produced over; kept for signature stability). +- Fields added within v1 after that — `metadata_blob_hash`, `key_mode`, `wrapped_file_key`, and the `DerivativeManifest`'s `amk_version`/`protocol_version` — encode as **absent keys** at their default/absent state, so manifests signed before the fields existed re-verify byte-identically. In particular an absent `key_mode` **means `derived`**; an implementation that writes `key_mode: "derived"` explicitly, or `metadata_blob_hash: null`, emits different signed bytes and breaks verification. + ## Provenance of Library Modifications Every modification of data or metadata produces a **provenance record** — timestamp, device, client version, action — anchored by the [signed manifest](#asset-manifest) above. The records form an **append-only, hash-chained log per asset**, which is the only structure that lets a key-holding attacker be detected after the fact. @@ -80,14 +84,16 @@ Every modification of data or metadata produces a **provenance record** — time ProvenanceRecord { asset_id: UUID, manifest: AssetManifest, // see Asset Manifest above - prior_provenance_hash: Option<[u8;32]>, // SHA-256 over the previous record; - // null only for `action = create` - // The manifest's own `prior_provenance_hash` mirrors this value, so signature - // coverage of the manifest is signature coverage of the chain link itself. + prior_provenance_hash: Option<[u8;32]>, // hash over the previous record; + // null only for `action = create`. + // MUST equal manifest.prior_provenance_hash — + // a checked invariant, not trusted redundancy } ``` -Each non-create record references its predecessor by hash; a rewrite of any past record breaks the chain at that point and is detectable by any client walking forward from `create`. +The record's `prior_provenance_hash` is a **derived mirror** of the manifest's own field, kept so signature coverage of the manifest is signature coverage of the chain link itself; `verify_asset` and the chain walker **reject a record whose two copies diverge** rather than preferring either. Each non-create record references its predecessor by hash; a rewrite of any past record breaks the chain at that point and is detectable by any client walking forward from `create`. + +**Verification cost is bounded by chain length**, and photo lifecycles keep chains short (a create, occasional edits, at most a delete/restore). Clients cache the verified `latest_provenance_hash` per asset, so steady-state verification is O(1) against the cached head — a full forward walk happens only on first fetch, restore, or suspected tamper. v1 defines no checkpoint structure; if edit-heavy assets ever make full walks material, a signed chain checkpoint is the designated v2 extension (a new record kind, not a schema break). ### What an Attacker With All Current Keys Still Cannot Do @@ -102,7 +108,7 @@ This bounds the blast radius of a credential compromise: history is read-only. ### Physical Storage - **Client.** An append-only CBOR file at `media/{YYYY}/{YYYY-MM}/{uuid}.provenance.cbor`, alongside the asset and its sidecar — a sequence of `ProvenanceRecord` entries; on hard-delete the log persists as a tombstone-with-history. This file is a **non-authoritative local cache**. A faulty or malicious client can corrupt or truncate *its own* copy, but cannot rewrite history: the chain is self-authenticating — each record is signed and carries the prior record's hash, so dropping or altering any record breaks the forward walk from `create` — and the authoritative copy is the server's append-only blob sequence plus the replicas every other album member holds, any of which re-detects the tamper on next sync as a chain-head mismatch. A client that finds its local cache inconsistent with the authoritative chain rebuilds it from the server. -- **Server.** A content-addressed encrypted blob, distinct from the [encrypted metadata blob](/design/cryptography/encryption/#metadata-encryption), so a metadata edit (which mints a new metadata blob) never rewrites history. The server's no-key envelope of every provenance write includes `prior_provenance_hash`, so the server can enforce monotonic chain advance without holding any key — see [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). +- **Server.** The append-only sequence of **envelope objects** (see Physical placement above) — deliberately server-visible, since every field of a manifest is server-visible by construction and the server *must* read `prior_provenance_hash`, `action`, and `retention_until` to enforce its no-key invariants and the purge floor. The chain is distinct from the [encrypted metadata blob](/design/cryptography/encryption/#metadata-encryption), so a metadata edit (which mints a new metadata blob) never rewrites history, and the server can enforce monotonic chain advance without holding any key — see [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). The server is **append-only** for provenance: there is no API path that overwrites or deletes an existing entry. An attempt is rejected at the [server's structural validation layer](/design/threat-model/validation/). @@ -114,6 +120,12 @@ Thumbnails, previews, and embeddings are generated client-side and uploaded as o DerivativeManifest { version: "derivative-manifest/v1", crypto_suite_id: u16, + protocol_version: Option, // YYYY-MM-DD; matches the album pin. Wire-optional (absent key) only for + // pre-binding fixtures; REQUIRED on every real write + amk_version: Option, // the epoch whose write-tier key produced write_sig — the verifier needs it + // to select the verification key. Wire-optional for pre-binding fixtures; + // REQUIRED on every real write: a derivative without it cannot be + // authorization-verified and is rejected source_asset_id: UUID, role: enum, // thumbnail | preview | embedding (LQIP lives in the signed sidecar, not here) format: String, // e.g. "image/avif", "embedding/mobileclip-b" diff --git a/capsule-docs/src/content/docs/design/filesystem/client.md b/capsule-docs/src/content/docs/design/filesystem/client.md index 91f9e4e6..1a1068e3 100644 --- a/capsule-docs/src/content/docs/design/filesystem/client.md +++ b/capsule-docs/src/content/docs/design/filesystem/client.md @@ -35,7 +35,7 @@ The directory layout below is itself a contract — the recovery-first rebuild a └── {uuid}.reason.json # parse error / signature failure / schema mismatch ``` -- **`media/`**: originals, their sidecars, and their provenance chains. Filenames are `{UUIDv7}.{extension}` (always lowercase), `{UUIDv7}.cbor`, and `{UUIDv7}.provenance.cbor` respectively. The CBOR sidecar is the client's canonical, self-describing metadata record (see [Metadata — Sidecar Schema v1](/design/metadata/#sidecar-schema-v1)) — the plaintext counterpart of the encrypted metadata blob the server stores. The `.provenance.cbor` file is an append-only signed log per asset (see [Cryptography — Provenance](/design/cryptography/provenance/#provenance-of-library-modifications)); the client never deletes it, so a hard-deleted asset leaves a tombstone-with-history. Per the recovery-first principle, the entire library is reconstructible from these three files alone. Files are date-bucketed by capture timestamp because the client, unlike the server, can read capture dates. +- **`media/`**: originals, their sidecars, and their provenance chains. Filenames are `{UUIDv7}.{extension}` (always lowercase), `{UUIDv7}.cbor`, and `{UUIDv7}.provenance.cbor` respectively. The CBOR sidecar is the client's canonical, self-describing metadata record (see [Metadata — Sidecar Schema v1](/design/metadata/#sidecar-schema-v1)) — the plaintext counterpart of the encrypted metadata blob the server stores. The `.provenance.cbor` file is an append-only signed log per asset (see [Cryptography — Provenance](/design/cryptography/provenance/#provenance-of-library-modifications)); the client never deletes it, so a hard-deleted asset leaves a tombstone-with-history. Per the recovery-first principle, the entire library is reconstructible from these three files alone. Files are date-bucketed by capture timestamp because the client, unlike the server, can read capture dates. The bucket is **fixed at import** from the capture timestamp known then: a later capture-date correction (a `metadata-update`) does not relocate the bundle — the sidecar is authoritative for date and the path is only a shard — so bucket-vs-timestamp drift after such an edit is expected, and [maintenance](/design/filesystem/maintenance/#self-validation) repairs it opportunistically rather than treating it as a fault. - **`cache/`**: purely derived and rebuildable — thumbnails and previews (formats declared in [Thumbnails — Thumbnail and Preview Formats](/design/thumbnails/#thumbnail-and-preview-formats)), verbose parsed-metadata caches, and transcodes. Sharded by UUID prefix to bound directory sizes. Deletable at any time; never a source of truth. - **`index/library.sqlite`**: a rebuildable query cache over the sidecars, and the local vector index backing AI features (`sqlite-vec` — see [AI/ML Integrations](/design/ai/)). It is also the substrate for [view albums](/design/organization/#system--smart-albums-views) — system aggregations like *All* and user-defined smart albums are materialized by querying this index entirely client-side, with no server involvement. On a schema change it may be dropped and rebuilt rather than migrated, since it is always reconstructible. - **`.library/`**: library-scoped state — schema version, user configuration, a process lock file that prevents two app instances from opening the same library, the trash (soft-delete retention area), and `quarantine/` (where irreplaceable bytes that failed structural or signature validation are preserved verbatim alongside a `.reason.json` recording the rejection). The quarantine area is the union surface listed in [Threat Model — Quarantine Surfaces](/design/threat-model/scenarios/#quarantine-surfaces). The `version` file pins the on-disk layout schema; a layout bump rebuilds derived structures (cache, index) and never touches the canonical original/sidecar/provenance files, so it cannot lose data. diff --git a/capsule-docs/src/content/docs/design/filesystem/maintenance.md b/capsule-docs/src/content/docs/design/filesystem/maintenance.md index edde36cf..fe37e414 100644 --- a/capsule-docs/src/content/docs/design/filesystem/maintenance.md +++ b/capsule-docs/src/content/docs/design/filesystem/maintenance.md @@ -22,14 +22,14 @@ Validation answers a stronger question than scrubbing: *is the library still a f A directory walk that checks the invariants of the [client layout](/design/filesystem/client/#desktop-library-layout): - Every `{uuid}.{ext}` original has a matching `{uuid}.cbor` sidecar and `{uuid}.provenance.cbor` chain. Every sidecar parses as valid CBOR with its required fields present, has a `sidecar_schema` ≤ the client's max known (per the [tightened Postel's Law](/design/principles/#postels-law-asymmetric)), and bears a valid signature from a device in the user's directory. -- A sidecar's `uuid` field matches its filename, and its date bucket matches its capture timestamp. +- A sidecar's `uuid` field matches its filename, and its date bucket matches its capture timestamp. (Bucket-vs-timestamp drift is *expected* after a capture-date correction — the bucket is [fixed at import](/design/filesystem/client/#desktop-library-layout) — and is repaired opportunistically as a rename bundle; it is a fault only when no correction explains it.) - Every `cache/` entry (thumbnail, transcode, parsed-metadata cache) and every `.library/trash/` file refers to an asset the library still knows. - The provenance chain for each asset is walkable from `create` to head, with each record's `prior_provenance_hash` matching the preceding record's content hash. A break — a missing record or a non-matching `prior_provenance_hash` — is a quarantine surface, not a silent skip. - Index rows reference files that exist — this subsumes the [local index staleness](/design/filesystem/client/#local-index-staleness) check. ### Content Validation (Expensive, Scheduled) -Recomputes the [content hash](/design/cryptography/primitives/) of each locally present original and compares it against the sidecar's `hash` field (the algorithm-tagged form declared in [Metadata — Sidecar Schema v1](/design/metadata/#sidecar-schema-v1); the algorithm itself follows whatever `crypto_suite_id` the sidecar carries). The original is the only irreplaceable thing on a client, so silent bit rot is the worst failure a client can suffer and nothing else detects it. +Recomputes the [content hash](/design/cryptography/primitives/) of each locally present original and compares it against the sidecar's `hash` field (the algorithm-tagged form declared in [Metadata — Sidecar Schema v1](/design/metadata/#sidecar-schema-v1); the algorithm itself follows whatever `crypto_suite_id` the sidecar carries — one suite id governs *both* digests Capsule computes, this plaintext `hash` and the ciphertext content hash, as a single algorithm choice that changes in lockstep, never independently). The original is the only irreplaceable thing on a client, so silent bit rot is the worst failure a client can suffer and nothing else detects it. Because hashing every original is heavy I/O, content validation is **not** run at startup: it is scheduled opportunistically (device idle, on power, unmetered) and throttled, can be triggered on demand, and re-verifies each original on a slow rolling cadence rather than all at once. @@ -71,7 +71,7 @@ Resolution is conservative and never silent. The client presents each duplicate Every write that must not tear uses temp-file + atomic rename, staged on the same filesystem as its destination. The atomicity rule is enforced at three granularities — the single file, the per-asset bundle, and the multi-asset edit. These are also the canonical statement of the rule; [Threat Model — Atomicity Invariants](/design/threat-model/validation/#atomicity-invariants) cross-references them and is where the cross-doc invariant lives. - **Client — single-file writes.** Sidecar and provenance appends stage to `{uuid}.cbor.tmp` and `{uuid}.provenance.cbor.tmp` in the destination directory, then rename into place. A direct overwrite is never used. -- **Client — per-asset bundle.** An asset import or update is a *bundle*: original (when present locally), sidecar, and a new provenance record. All `.tmp` files stage first; only after every staged file is on disk do the renames execute, and only in a fixed order (original → sidecar → provenance). A failure at any rename discards every remaining `.tmp` and rolls back the renames already done by deleting the just-renamed targets, so the on-disk state never reflects a partial bundle. The `.provenance.cbor` is the last to be renamed, so the existence of a new provenance record implies the rest of the bundle is committed. +- **Client — per-asset bundle.** An asset import or update is a *bundle*: original (when present locally), sidecar, and a new provenance record. All `.tmp` files stage first; only after every staged file is on disk do the renames execute, and only in a fixed order (original → sidecar → provenance). A **rewrite** bundle — a `metadata-update` or `replace` whose rename lands over an existing file — first stages the displaced prior version aside (`{uuid}.cbor.prev`, same directory), so rollback can *restore* it rather than merely delete the new target; the `.prev` staging is removed only after the bundle's final rename commits. A failure at any rename then discards every remaining `.tmp` and rolls back the renames already done — deleting freshly-created targets, restoring `.prev` files over rewritten ones — so the on-disk state never reflects a partial bundle and an interrupted rewrite can never lose the pre-edit state. The `.provenance.cbor` is the last to be renamed, so the existence of a new provenance record implies the rest of the bundle is committed. - **Client — stack edit.** A stack edit touches multiple sidecars and writes a single provenance record per affected asset. All `.tmp` files (one per sidecar plus one per provenance file) stage first and rename together; any rename failure discards the entire batch. There is no partial stack. - **Server — chunk assembly.** Chunks stage as `{upload_id}_{n}.part`; the assembled blob is `{upload_id}.bin`. The blob is renamed into its content-addressed location under `blobs/` only after the ciphertext hash is recomputed and matches the declared value (see [Upload Protocol — Finalization and Integrity](/design/import/upload-protocol/#finalization-and-integrity)). - **Server — finalization transaction.** The blob rename into its content-addressed `blobs/` location is a filesystem operation and so necessarily happens *before* the Postgres commit; the manifest-envelope insert, metadata-blob insert, provenance-blob insert, and asset-row `uploaded` flip then commit in a **single PostgreSQL transaction**. That ordering is what makes every crash point safe: a crash *before* the rename leaves only `incoming/` debris (scrubbed below); a crash *after* the rename but *before* the commit leaves a finalized blob in `blobs/` that **no committed row references** — an orphan the [reference-count GC](/design/filesystem/server/#deletion-and-garbage-collection) reclaims, while the idempotent retry re-finalizes against the already-present blob (re-placing a content-addressed hash is a no-op). The "single transaction" guarantee is over the **index rows**; blob *placement* is idempotent and GC-safe precisely because it is content-addressed. The server never exposes an asset whose index bundle is partially persisted — the session stays in `WaitingForProcessing` until a finalization attempt commits the whole bundle or fails it cleanly. diff --git a/capsule-docs/src/content/docs/design/filesystem/server.md b/capsule-docs/src/content/docs/design/filesystem/server.md index 0f495a42..121d6646 100644 --- a/capsule-docs/src/content/docs/design/filesystem/server.md +++ b/capsule-docs/src/content/docs/design/filesystem/server.md @@ -46,22 +46,22 @@ Switching profiles is operationally invisible to clients — the [upload protoco ## Uniform, Opaque Blobs -A single asset produces a **bundle** of blobs (see [Import — Upload Protocol: What Gets Uploaded](/design/import/upload-protocol/#what-gets-uploaded)): the encrypted original, encrypted derivatives (thumbnails, previews), the encrypted CBOR metadata blob (which carries the LQIP), and the encrypted provenance blob (see [Cryptography — Provenance](/design/cryptography/provenance/)). The blob store does not distinguish them — every blob is just content-addressed ciphertext. The mapping from an asset to its constituent blobs, and the role of each blob, lives entirely in PostgreSQL. +A single asset produces a **bundle** of blobs (see [Import — Upload Protocol: What Gets Uploaded](/design/import/upload-protocol/#what-gets-uploaded)): the encrypted original, encrypted derivatives (thumbnails, previews), and the encrypted CBOR metadata blob (which carries the LQIP) — every one of them fully opaque, content-addressed ciphertext the store does not distinguish. Beside them, each write persists its signed **manifest envelope object** (see [Provenance — Physical placement](/design/cryptography/provenance/#asset-manifest)): a small, deliberately server-visible signed CBOR object, stored content-addressed like any blob, whose append-only sequence is the asset's provenance chain. The hot-path mapping from an asset to its blobs and their roles lives in PostgreSQL, with the envelope objects as its durable, key-free backup. ## Recovering the Index from Blobs Alone -The PostgreSQL index is authoritative but **not the only copy** of what the server knows. Every blob carries enough server-visible structural metadata — the [unencrypted portion](/design/cryptography/provenance/#asset-manifest) of the asset manifest — to rebuild the index row that referenced it. This is the server-side counterpart of the recovery-first principle that lets a client rebuild its index from CBOR sidecars. +The PostgreSQL index is authoritative but **not the only copy** of what the server knows. Every write persists its signed [manifest envelope object](/design/cryptography/provenance/#asset-manifest) in the blob store beside the opaque ciphertext blobs it names, and those envelopes carry everything needed to rebuild the index rows. This is the server-side counterpart of the recovery-first principle that lets a client rebuild its index from CBOR sidecars. -The server-visible portion of a blob includes: +The server-visible envelope includes: - `crypto_suite_id`, `protocol_version`, `amk_version` — what bundle of primitives encrypted this asset and which album epoch - the ciphertext hash and declared size — content address and storage attribution - `created_by_user`, `created_by_device`, `album_id`, `file_id`, `prior_provenance_hash`, `action` — owner, provenance chain link, and lifecycle action - the device's hybrid signature — provenance attribution; verifiable against the public device directory even without any key the server holds -A rebuild walks `blobs/`, reads the manifest envelope of each blob, verifies the device signature against the cached device directory, and writes an index row. The rebuild is idempotent: re-running it against an existing index produces no changes. The full envelope check list a server runs at recovery is the same list it runs at write time — see [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). +A rebuild walks the **envelope objects** under `blobs/`, verifies each device signature against the cached device directory, and writes index rows for the blobs each envelope names (original and derivatives by `ciphertext_hash` + role, the metadata blob by `metadata_blob_hash`). A ciphertext blob referenced by no envelope surfaces as an orphan for [GC](#deletion-and-garbage-collection); an envelope naming a missing blob surfaces as a dangling reference. The rebuild is idempotent: re-running it against an existing index produces no changes. The full envelope check list a server runs at recovery is the same list it runs at write time — see [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). -A blob whose manifest envelope fails structural validation during rebuild is **quarantined**, not silently dropped — moved to `{blob_root}/quarantine/` with a sibling `.reason.json` recording the rejection code. This guarantees that an unrecoverable byte sequence is preserved for forensic inspection rather than vanishing on rebuild. +An envelope object that fails structural validation during rebuild is **quarantined**, not silently dropped — moved to `{blob_root}/quarantine/` with a sibling `.reason.json` recording the rejection code. This guarantees that an unrecoverable byte sequence is preserved for forensic inspection rather than vanishing on rebuild. Operationally the rebuild is invoked when a PostgreSQL restore is incomplete or a logical-corruption event is detected; it is **never** the hot path. The hot path runs through the authoritative PG index. The recovery path's job is to make the index reconstructible if PG is lost, not to substitute for it. diff --git a/capsule-docs/src/content/docs/design/metadata.md b/capsule-docs/src/content/docs/design/metadata.md index 4d970ad1..8d6aee38 100644 --- a/capsule-docs/src/content/docs/design/metadata.md +++ b/capsule-docs/src/content/docs/design/metadata.md @@ -28,7 +28,7 @@ SidecarV1 { // collaborative metadata (see Collaborative Metadata below) tags_user: OR_set<(tag: String, add_id)>, tags_ai: OR_set<(tag: String, add_id, model_id: String, model_version: String)>, - caption_lww: Option<{ value: String, ts: RFC3339, by: device_id }>, + caption_lww: Option<{ value: String, ts: RFC3339, by: device_id }>, // value bounded ≤ 4096 bytes superseded_captions: Vec<{ value: String, written_by: device_id, ts: RFC3339 }>, // bounded ≤ 16 rating_lww: Option<{ value: u8, ts: RFC3339, by: device_id }>, @@ -43,8 +43,10 @@ SidecarV1 { // geolocation (see Geolocation below) gps: Option<{ lat: f64, lon: f64, source: GpsSource }>, - // provenance binding - provenance_chain_hash: [u8; 32], // hash of the latest ProvenanceRecord for this asset + // provenance binding — the PRIOR chain head; see Provenance Binding and Sealing Order below + provenance_chain_hash: Option<[u8; 32]>, // hash of the provenance record PRECEDING the write that seals this + // sidecar — always equal to that write's manifest + // prior_provenance_hash; absent only on the initial create // forward-compat _unknown: Map, // unknown CBOR keys preserved verbatim, never executed @@ -61,6 +63,15 @@ SidecarV1 { - The signature covers every byte including `_unknown`, so stripping unknown fields invalidates the signature and is detectable. - A schema bump is a coordinated change; per [Versioning — Album Protocol Version Pinning](/design/versioning/#album-protocol-version-pinning), an album's pinned protocol version constrains which sidecar schemas may be written into it. +### Closed Enum Value Sets + +Two sidecar fields are closed enums whose authoritative value sets live here (the blanket closed-enum rule is [Threat Model — Schema Rules](/design/threat-model/schema-rules/); the code mirror is a closed Rust enum in `capsule-core::domain`, and adding a value bumps `protocol_version`): + +- **`content_type`** — MIME syntax, exactly **one canonical value per format** (never an alias like `image/jpg`). The v1 set: + - images: `image/jpeg`, `image/png`, `image/webp`, `image/gif`, `image/tiff`, `image/heic`, `image/avif`, `image/jxl`, `image/x-adobe-dng` + - video: `video/mp4`, `video/quicktime`, `video/x-matroska`, `video/webm` +- **`gps.source` (`GpsSource`)** — `exif` (written by the capturing device), `manual` (set by the user), `inferred` (client-derived, e.g. from a paired device's location or an ML suggestion). An `inferred` value is written to the canonical `gps` field only on **explicit user confirmation** — the same promotion rule as `tags_ai` → `tags_user`, so an automated guess can never silently overwrite capture truth. + ### Canonical CBOR Encoding The sidecar — and the [encrypted metadata blob](/design/cryptography/encryption/#metadata-encryption) whose plaintext is this same CBOR document — must serialize **byte-identically across every implementation and language**: the bytes are what the [signed manifest](/design/cryptography/provenance/#asset-manifest) and content hash commit to, so one divergent byte makes an honest sidecar look forged to another platform or [federated](/design/federation/) peer. The canonical rules are RFC 8949 §4.2 deterministic encoding, normative here: @@ -82,14 +93,29 @@ The plaintext of the server's [encrypted metadata blob](/design/cryptography/enc A client therefore never persists a sidecar that does not round-trip to the committed `metadata_blob_hash`, and a server can expose only the exact metadata bytes the originating client encrypted. The matching client-side check is a [client-side validation invariant](/design/threat-model/validation/#client-side-validation-invariants); the no-key server enforces the blob-hash match structurally as [invariant 25](/design/threat-model/validation/#server-side-validation-invariants). +### Provenance Binding and Sealing Order + +`provenance_chain_hash` binds the sidecar to a specific point in the asset's [provenance chain](/design/cryptography/provenance/#provenance-of-library-modifications). It references the **prior** chain head — the record *preceding* the write that seals this sidecar — never "the latest record": the latest record *is* the write being produced, and its manifest commits to `metadata_blob_hash`, so a sidecar referencing it would have to contain a hash of a structure that contains a hash of the sidecar itself. Referencing the prior head keeps the binding well-founded, and makes the sidecar and manifest mutually checkable: **`sidecar.provenance_chain_hash` MUST equal the sealing manifest's `prior_provenance_hash`** (both absent exactly on `create`), a divergence being quarantined like any round-trip failure. + +The sealing order every writer follows: + +1. **Fix the prior head** `H` — the current chain head for this asset (`None` on `create`). +2. **Author and sign the sidecar** with `provenance_chain_hash = H`. +3. **Seal** the sidecar into the [metadata blob](/design/cryptography/encryption/#metadata-blob-wire-format); compute its content hash. +4. **Build and sign the manifest** with `prior_provenance_hash = H` and `metadata_blob_hash` from step 3; append it as the new chain head. + ### Add-id Binding `add_id` is the tuple `(device_id: UUIDv4, monotonic_counter: u64)`, where `monotonic_counter` is incremented per-device per-(asset, OR-set) pair. Every OR-set add carries an `add_id`; every OR-set remove targets a specific `add_id`. A remove that names an `add_id` the receiver has never observed an add for is **rejected**, not silently no-op — preventing the "remove an element you never added" attack noted in the [Threat Model](/design/threat-model/scenarios/). **Counter durability across restarts.** A `monotonic_counter` must never repeat for a given `(device_id, asset, OR-set)`: a reused `add_id` would alias two distinct adds, so removing one would silently delete the other and break OR-set convergence. The counter is persisted in the local [index](/design/filesystem/client/#desktop-library-layout), and on client restart or reinstall it is **reseeded to one past the maximum `add_id.counter` this device has ever issued**, recovered from the signed sidecars themselves (a device's own past `add_id`s are durably recorded in the sidecars it wrote). An add lost to a crash *before* its sidecar was persisted was never observed by any peer, so its counter may be safely reused — correctness depends only on never reusing a counter that ever reached a written sidecar. A counter is reset to zero only when the device can prove it has issued nothing — i.e. no sidecar bears its `device_id`. This makes the counter monotonic over the lifetime of a `device_id`, not merely within one process. +When the device's own sidecars are not held locally (a metadata-only sync scope, or local loss repaired from the server), the reseed source is the same sidecars fetched back from the server — the durable record is the signed sidecar wherever it is held, so the rule is unchanged. And in practice a *reinstalled* device re-enrolls with a **new** `device_id` (device keys are hardware-bound and non-exportable, so the old identity cannot be resumed), which is why the reset-to-zero case is safe: it applies only to genuinely fresh device identities. + ## Identifiers +**One canonical asset identity.** The sidecar's `uuid`, the manifest's `file_id`, the provenance chain's and server index's `asset_id`, and the metadata-key-salt's `blob_id` are **the same UUIDv7**, minted once at import and never re-minted for the asset's lifetime. The per-schema names survive for local readability only; they never diverge, and every equality between them may be assumed (and is asserted) by validators. UUID versions across the system: the asset id, `session_id`, `stack_id`, and `import_id` are UUIDv7 (time-ordered); `device_id` is UUIDv4 (unordered — a device id must not leak creation ordering). + The three identifying fields defined inside the sidecar schema are subject to the [Privacy on Export](#privacy-on-export) rules below when an asset crosses a trust boundary. - **Camera identifier (`camera_id`).** Model ID of the device plus a unique identifier for the specific device (e.g. serial number). Useful for grouping shots from the same physical camera across libraries. @@ -138,7 +164,7 @@ This converts a silent-data-loss damage vector (a buggy client clobbering anothe ### How Operations Travel -We encrypt the **operations**, not the resulting state. Merges are then commutative and associative, so order of arrival does not matter and a peer replaying a stale operation cannot corrupt current state. The operation log reconciles into the canonical CBOR sidecar, which remains the source of truth (see [Core Principles](/design/principles/) — recovery-first). +We encrypt the **operations**, not the resulting state. Merges are then commutative and associative, so order of arrival does not matter and a peer replaying a stale operation cannot corrupt current state. The operation log reconciles into the canonical CBOR sidecar, which remains the source of truth (see [Core Principles](/design/principles/) — recovery-first). Operations name **known fields only** — an operation targeting a field the receiver does not know is from a newer schema and is version-gated like any forward sidecar — so reconciliation rewrites only the CRDT fields it understands and re-emits `_unknown` verbatim from the stored sidecar bytes; the byte-fidelity of unknown fields survives the op path exactly as it survives a whole-sidecar rewrite. Each operation carries the same `prior_provenance_hash` chain link as any [lifecycle action](/design/authorization/#the-closed-action-set), so a metadata-update is provenance-tracked exactly like a create or delete. diff --git a/capsule-docs/src/content/docs/design/versioning.md b/capsule-docs/src/content/docs/design/versioning.md index 1398846b..377fbeff 100644 --- a/capsule-docs/src/content/docs/design/versioning.md +++ b/capsule-docs/src/content/docs/design/versioning.md @@ -59,14 +59,14 @@ A version-pinned album is upgraded by a **tombstone-plus-fork** ceremony: the ol ### Steps -1. **Freeze proposal.** An album admin issues an MLS application message `UpgradeIntent { from_version, to_version, intent_id, proposer_device, deadline }`, hybrid-signed by the admin's [DSK](/design/cryptography/keys/#device-keys). The proposal carries a deadline (default 7 days). Any member's client receiving an `UpgradeIntent` for an album that is already in upgrade quiescence under a *different* `intent_id` rejects the new proposal — only one upgrade can be in flight per album. +1. **Freeze proposal.** An album admin issues an MLS application message `UpgradeIntent { from_version, to_version, intent_id, proposer_device, deadline }`, hybrid-signed by the admin's [DSK](/design/cryptography/keys/#device-keys). `deadline` is a **duration** (default 7 days); the effective expiry is `received_at + deadline` on the **server's trusted clock** ([Filesystem — Server](/design/filesystem/server/#postgresql-what-the-server-knows)), and the abort-on-expiry in step 3 is evaluated against that server-attested time — a skewed member clock can neither extend nor shorten the window. Any member's client receiving an `UpgradeIntent` for an album that is already in upgrade quiescence under a *different* `intent_id` rejects the new proposal — only one upgrade can be in flight per album. 2. **Quiesce writes.** Members enter upgrade quiescence on receipt of `UpgradeIntent`: - In-flight uploads against the album are allowed to reach a terminal state. - New writes are queued **locally** with a `pending_until_upgrade` flag and the `intent_id`; they are not sent to the server. - The server augments the album row with `upgrade_pending_to = to_version, intent_id`. New upload sessions for this album whose `manifest.intent_id` does **not** match are rejected with `409 Conflict` — preventing a stale v_old client from writing past the freeze. 3. **Drain.** The upgrade cannot proceed while any session for this album is in `Uploading` or `WaitingForProcessing`. The server exposes the in-flight count to the proposer's client. The deadline from step 1 bounds the wait; on deadline expiry the upgrade aborts cleanly (state returns to v_old normal; queued local writes are flushed back to v_old). -4. **Tombstone.** Once drained, the proposing admin issues an MLS commit `AlbumTombstone { intent_id, frozen_state_hash }`. `frozen_state_hash` is a SHA-256 over the canonical CBOR of the album's full state: the sorted member list, every accepted manifest's hash, and the head of the album's provenance log. Every receiving member's client recomputes the hash against its own state; on mismatch the upgrade aborts (each member independently — the album returns to normal operation). Hash mismatch means at least one member's view of the album diverges and must be resolved before any upgrade. -5. **Fork.** A new album group is created at `to_version`, MLS-named `parent_id_v{n}`, with the manifest field `upgraded_from: { old_album_id, intent_id, frozen_state_hash }`. Assets are **not** re-encrypted: the new album references the existing ciphertext blobs by content hash. Members are added to the new MLS group via standard `Add` proposals; fresh `AMK_v1` and a fresh write-tier key are minted. +4. **Tombstone.** Once drained, the proposing admin issues an MLS commit `AlbumTombstone { intent_id, frozen_state_hash }`. `frozen_state_hash` is the [content hash fixed by `crypto_suite_id`](/design/cryptography/primitives/#cryptographic-hash) over the canonical CBOR of the album's full state: the sorted member list, every accepted manifest's hash, and the head of the album's provenance log. Every receiving member's client recomputes the hash against its own state; on mismatch the upgrade aborts (each member independently — the album returns to normal operation). Hash mismatch means at least one member's view of the album diverges and must be resolved before any upgrade. +5. **Fork.** A new album group is created at `to_version` (its MLS group naming is an MLS-layer detail owned by [Cryptography — MLS](/design/cryptography/mls/); the normative link between old and new album is the field below, never the group name), with the manifest field `upgraded_from: { old_album_id, intent_id, frozen_state_hash }`. Assets are **not** re-encrypted: the new album references the existing ciphertext blobs by content hash. Members are added to the new MLS group via standard `Add` proposals; fresh `AMK_v1` and a fresh write-tier key are minted. 6. **Apply queued writes.** Each member's locally queued `pending_until_upgrade` writes are re-encoded against `to_version` (the album pin and `crypto_suite_id` may have changed) and replayed into the new album. 7. **Resumption (partial-failure recovery).** A client that crashes between step 2 and step 6 reads its local `upgrade_pending_to` on restart, queries the server for the upgrade's current phase via the album row, and resumes from there. The `intent_id` is the idempotency key — the same `UpgradeIntent` never produces two forks, and a duplicate `AlbumTombstone` commit is a no-op at the MLS layer. 8. **Atomicity guarantee.** The cutover is the single MLS commit in step 4. Until that commit is applied by a member's client, the client is operating in v_old; after, in v_new. There is no in-between state visible to one client. Cross-member, the cutover is observed as each member processes the commit; until the slowest member processes it, that member is still in v_old (and its `pending_until_upgrade` writes remain queued locally, never lost). From b50d4fd60e9b1f623938dd2856e3d8dce1d5c3c3 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:22:48 -0400 Subject: [PATCH 04/58] docs(design): fix networked-plane contract defects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Security-contract and protocol defects across the networked docs, each either self-contradictory or exploitable as written: - backup-recovery: the 48-bit recovery-code floor wrapping the master key (offline-brute-forceable once the escrow blob leaks) is raised to >=128 bits; sub-floor codes are permitted only inside an attested rate-limiting enclave (SVR pattern) - share-links: the POST .../passphrase endpoint contradicted the contract's own the-server-never-receives-the-passphrase rule — it becomes a GET of the wrapped material with client-side unwrap; the 60s revocation cache is motivated (replicas of one home server) and stays fail-closed - web-upload: the write-gating passphrase gets a real possession proof (stored Argon2id verifier, KDF-derived proof, passphrase never transmitted); adoption's quota wording corrected to bulk-bytes-only; invariant-14-at-drop-finalization made explicit; drop-invariant anchor precision - quota: Grace-expired now exempts the provenance writes a delete or trash-restore itself produces (a user must be able to delete their way back under quota) and states exactly what it adds over Hard-exceeded - federation: capability token gains the referenced-but-undeclared iat claim; min_protocol_version disambiguated (the album pin); the nonstandard aud noted for verifier authors; revoked-jti pruning bounded; invariant 25 applies on pulls; read-derivative-only is now structurally enforceable via the per-role envelope objects; perspective check requires >=2 unanimous vantage points - moderation: 410-vs-404 divergence made an explicit per-surface rule; suspended users keep master-key-gated revoke_all_sessions (the DoS it defended against is already closed upstream, and a compromised suspended user needs it most); no-content-scanner scoped server-side with the client-side AI cross-link - mls-resilience: the lost-commit and local-ahead paths overlapped — a lost commit routes to re-submission and only budget exhaustion or a provable fork routes to discard-and-rebootstrap - device-enrollment + keys: hardware key generation claims corrected to what secure elements ship (classical half hardware-bound and P-256, PQ half software-sealed; the P-256 hybrid variant is the planned composition); the enrollment-code entropy/transcribability tension resolved (full-entropy QR, shorter rate-limited text fallback backstopped by the safety code) - peering: cross-device add/recovery mislink fixed to the right owners; device-key-as-TLS-certificate specified (classical half in the handshake, hybrid IK chain at the application layer); mDNS rotation interval bounded; X-Capsule-Protocol declared its own version space - authentication: the moved certificate gets a path and schema; access tokens are issued-against not derived; WebFinger opt-in record set bounded - ai: the vector-search section described pgvector (HNSW, <#>) for a client-local sqlite-vec index — rewritten for the actual engine; insert refusal targets unknown models, not superseded ones - download-sync: 403 treated as authorization change (re-sync membership) rather than durability loss; idle-notification interaction with session expiry noted - storage-verification: durable scoped to the home server; deep rate-limited and coalesced; the verify-to-release TOCTOU window bounded by re-verify + the GC grace period; endpoint-planned vs predicate-seam status split - pipeline: streaming mode's single-asset headroom floor and the default-album pointer's last-writer snapshot semantics stated - clients: the stale closed-enum list replaced by the blanket rule + owner-doc pointers; the forbidden-behavior tripwire scoped to capsule-core-mapped items; per-platform sandbox primitives named --- capsule-docs/src/content/docs/design/ai.md | 4 ++-- .../src/content/docs/design/authentication.md | 8 ++++---- .../src/content/docs/design/backup-recovery.md | 4 ++-- capsule-docs/src/content/docs/design/clients.md | 6 +++--- .../src/content/docs/design/cryptography/keys.md | 2 +- .../src/content/docs/design/device-enrollment.md | 6 +++--- capsule-docs/src/content/docs/design/federation.md | 13 +++++++------ .../src/content/docs/design/import/download-sync.md | 4 ++-- .../src/content/docs/design/import/pipeline.md | 4 +++- .../docs/design/import/storage-verification.md | 8 ++++++-- .../src/content/docs/design/mls-resilience.md | 2 +- capsule-docs/src/content/docs/design/moderation.md | 8 ++++---- capsule-docs/src/content/docs/design/peering.md | 8 ++++---- capsule-docs/src/content/docs/design/quota.md | 6 +++--- capsule-docs/src/content/docs/design/share-links.md | 10 ++++++---- capsule-docs/src/content/docs/design/web-upload.md | 6 +++--- 16 files changed, 54 insertions(+), 45 deletions(-) diff --git a/capsule-docs/src/content/docs/design/ai.md b/capsule-docs/src/content/docs/design/ai.md index 5cddc713..af9a342d 100644 --- a/capsule-docs/src/content/docs/design/ai.md +++ b/capsule-docs/src/content/docs/design/ai.md @@ -55,7 +55,7 @@ Embeddings share a common vector space and are stored locally in **SQLite + `sql Every embedding Capsule stores — in the local SQLite vector index, in an encrypted backup, or inside a [`DerivativeManifest`](/design/cryptography/provenance/#derivative-provenance) for an embedding-class derivative — carries the tuple `(model_id, model_version)` identifying which [inventory](#models-and-algorithms) row produced it. Vector spaces differ across pairs, so embeddings are not comparable across `(model_id, model_version)`. Every `model_id` is declared in exactly one inventory row ([SSoT](/design/principles/#single-source-of-truth)); a swap is a one-row edit that propagates by `model_id` to every consumer. The invariant: -- The vector index **refuses inserts** whose `model_id` is not the current canonical row for its task. A buggy or new client uploading embeddings from an unrecognized model is rejected at the insert API, never silently mixed in. +- The vector index **refuses inserts** whose `(model_id, model_version)` is **unknown to the inventory** — a buggy or new client producing embeddings from an unrecognized model is rejected at the insert API, never silently mixed in. Entries from a *superseded-but-known* version are admitted as stale-flagged rows (they are the regeneration queue and are excluded from queries) — the refusal targets unknown models, not known-but-old ones. - A model swap increments `model_version` for that task. Old embeddings are **flagged stale** and excluded from queries until regenerated from the originals. Cross-version comparison is forbidden — see [Threat Model — Client-Side Validation Invariants](/design/threat-model/validation/#client-side-validation-invariants). - Regeneration is a background task that walks the library producing fresh embeddings at the new version; old entries are removed only after new ones persist (per-asset replace, not a global truncate-and-rebuild). @@ -116,7 +116,7 @@ Identifies individuals even when they turn away from the camera during an event: #### High-Dimensional Vector Search -Exact KNN is too slow at millions of rows: use **HNSW** indexes on the vector columns, and the inner-product operator (`<#>`) for normalized embeddings (cheaper than $L_2$ or cosine at scale). +Vectors live in `sqlite-vec` `vec0` virtual tables (the [engine declared above](#database-indexing-and-view-generation)) and are queried by **inner-product distance** over normalized embeddings — the cheapest of the metrics. `sqlite-vec`'s SIMD brute-force scan covers libraries into the hundreds of thousands of rows; at millions of rows the designated escalation is its ANN indexing as it matures, or a partitioned scan (by album/date bucket) until then. Server-side vector-database idioms (pgvector's HNSW indexes, the `<#>` operator) do not apply — the index is client-local SQLite by design. ## Validation diff --git a/capsule-docs/src/content/docs/design/authentication.md b/capsule-docs/src/content/docs/design/authentication.md index c68df011..e76c2ee1 100644 --- a/capsule-docs/src/content/docs/design/authentication.md +++ b/capsule-docs/src/content/docs/design/authentication.md @@ -27,7 +27,7 @@ Patterns borrowed from Matrix 2.0, with one critical departure: **`.well-known/` - **User lookup is authenticated.** A client or peer server must present credentials to resolve `user@server.tld`: - **Local client lookup** (resolving another user on the same server, e.g. for sharing): authenticated by the looker's session token. - **Federated peer lookup** (resolving a user across servers): authenticated by a federation capability token (see [Federation — Federation Capabilities](/design/federation/#federation-capabilities)) and rate-limited per peer. - - **Anonymous WebFinger**: returns only records the target user has explicitly opted into making public. The default is opt-out: no anonymous record. This is deliberately stricter than Matrix's default and follows the [deny-by-default rule](/design/threat-model/schema-rules/#schema-evolution-and-field-grammar) from the threat model. + - **Anonymous WebFinger**: returns only records the target user has explicitly opted into making public. The default is opt-out: no anonymous record. The opt-in-able record set is deliberately tiny — handle and display name only, never keys, device lists, or album hints; anything richer requires authenticated lookup. This is deliberately stricter than Matrix's default and follows the [deny-by-default rule](/design/threat-model/schema-rules/#schema-evolution-and-field-grammar) from the threat model. ## Account Portability @@ -36,8 +36,8 @@ A user must be able to move servers without losing their identity. Capsule does Migration re-homes the handle while keeping the same IK: - The new server registers the account under the same IK; nothing in the [key hierarchy](/design/cryptography/keys/) changes. -- The old server publishes an IK-signed **"moved" certificate** at its `.well-known/` path, naming the new handle. This is the one well-known record that names a specific user — opted-into (the user actively migrates) and carrying the user's own signature, so it does not constitute the kind of enumeration leak we forbid. -- Clients and [federated](/design/federation/) peers that resolve the old handle fetch this certificate, verify its IK signature, and re-resolve to the new handle it names. +- The old server publishes an IK-signed **moved certificate** at `.well-known/capsule/moved/{user}` — a small signed record `{ old_handle, new_handle, moved_at, ik_sig }`, cacheable for 24 hours. This is the one well-known record that names a specific user — opted-into (the user actively migrates) and carrying the user's own signature, so it does not constitute the kind of enumeration leak we forbid. +- Clients and [federated](/design/federation/) peers that resolve the old handle fetch this certificate, verify `ik_sig` against the IK they already trust for the user (from the signed [device directory](/design/cryptography/keys/#device-directory)), and re-resolve to the new handle it names. An unverifiable certificate is ignored — the old handle keeps resolving as before, and the failure is surfaced. Because the IK signs the move and every device cross-signs to that IK, no server — old or new — can forge a migration or hijack the handle. @@ -55,7 +55,7 @@ A long-lived **128-bit secret** generated by the server upon successful authenti ### Access Token -Short-lived tokens derived from the session token, used to authenticate API requests. They have a limited lifespan and are refreshed using the session token without re-authenticating the user. +Short-lived tokens issued against the session token (presented, not cryptographically derived), used to authenticate API requests. They have a limited lifespan and are refreshed using the session token without re-authenticating the user. Capsule uses **EdDSA JWTs** as access tokens, signed under the server's Ed25519 signing key — classical only, per the [operational-signature carve-out](/design/cryptography/primitives/#signature-scheme) (access tokens are short-lived, so PQ hybridization buys no margin). diff --git a/capsule-docs/src/content/docs/design/backup-recovery.md b/capsule-docs/src/content/docs/design/backup-recovery.md index f86f577b..2ee6b905 100644 --- a/capsule-docs/src/content/docs/design/backup-recovery.md +++ b/capsule-docs/src/content/docs/design/backup-recovery.md @@ -56,9 +56,9 @@ ZIP was considered and rejected: its central-directory-at-end makes streaming wr The account master key is the single backed-up root of the key hierarchy (see [Cryptography — Keys](/design/cryptography/keys/)). It is escrowed server-side so a user holding only their recovery secret can reconstruct it: -- Wrap the account master key with a user-chosen high-entropy passphrase or a randomly generated 48+ bit recovery code. +- Wrap the account master key with a user-chosen high-entropy passphrase or a randomly generated recovery code of **≥128 bits** (e.g. a BIP39-style phrase) — the same entropy floor as every other unguessable identifier in the system. The escrow blob is offline-attackable once exfiltrated, and [Argon2id](/design/cryptography/primitives/#password-based-kdf) raises brute-force cost only linearly, so the secret itself must carry the security: **no low-entropy code is permitted without an enclave** (below). - Derive the wrapping key with the [password-based KDF](/design/cryptography/primitives/#password-based-kdf). Store the wrapped blob server-side. -- If you can run enclaves (SGX/Nitro/SEV-SNP), do Signal's SVR trick: rate-limit PIN attempts inside the enclave so a weak PIN is still safe. Without enclaves, require a real passphrase or recovery code — don't let users pick 4-digit PINs. +- If you can run enclaves (SGX/Nitro/SEV-SNP), Signal's SVR pattern is the *only* sanctioned way to soften that floor: rate-limit unwrap attempts inside the enclave so a shorter, human-friendly PIN is still safe. Without an attested enclave, the ≥128-bit floor is mandatory — never a 4-digit PIN, never a short numeric code. ## Recovery Mechanisms diff --git a/capsule-docs/src/content/docs/design/clients.md b/capsule-docs/src/content/docs/design/clients.md index 7dd995b4..72e7334f 100644 --- a/capsule-docs/src/content/docs/design/clients.md +++ b/capsule-docs/src/content/docs/design/clients.md @@ -24,7 +24,7 @@ Clients are not trusted to enforce their own correctness — but they **are** re - **Refuse forward-version writes.** Reject any incoming `sidecar_schema`, `crypto_suite_id`, or `protocol_version` above the client's max known. Reading is allowed only in read-only mode if explicitly opted into. - **Enforce the protocol handshake.** Send `X-Capsule-Protocol` on every request; honor `426 Upgrade Required` by stopping the request, never by silently downgrading. - **Check the provenance chain.** Maintain a local `latest_provenance_hash` per asset; refuse to apply a manifest whose `prior_provenance_hash` is behind it. See [Import — Stale-Revival Detection](/design/import/download-sync/#stale-revival-detection). -- **Reject unknown closed-enum values.** `action`, `content_type`, `DerivativeManifest.role`, and `gps.source` are closed per protocol version; unknown values are structural errors, not "future to ignore." +- **Reject unknown closed-enum values.** *Every* closed enum rejects unknown values as structural errors, never "future to ignore" — the blanket per-`protocol_version` rule is owned by [Threat Model — Schema Rules](/design/threat-model/schema-rules/#schema-evolution-and-field-grammar), and each value set by its owner doc (among them `action`, `content_type`, `gps.source`, `key_mode`, `DerivativeManifest.role`/`format`, `StackMembership.stack_type`/`role`). This list is illustrative; the rule is total. - **Preserve unknown CBOR keys within a known schema** (Postel's Law) but never act on them. - **Decode remote-origin asset bytes only in the [Sandboxed Decoder](#sandboxed-decoder).** - **Honor the [forbidden behaviors checklist](/design/threat-model/schema-rules/#forbidden-client-behaviors).** A client that backdates timestamps, strips unknown sidecar fields, overwrites provenance, signs for an epoch it does not hold, or invokes `revoke_all_sessions` without master-key proof is *buggy by definition*. @@ -47,7 +47,7 @@ Capsule's server never holds plaintext, so server-side image/video decoding is i The defense is structural isolation: -- **Every remote-origin asset is decoded in a separate OS process or a WASM sandbox** that has no filesystem write access, no network access, and no shared memory with the host app process. +- **Every remote-origin asset is decoded in a separate OS process or a WASM sandbox** that has no filesystem write access, no network access, and no shared memory with the host app process. The isolation primitive differs per platform — an XPC service / app extension on Apple platforms, an `isolatedProcess` service on Android, a privilege-dropped subprocess on desktop, a Worker-hosted WASM sandbox in the browser — and these invariants are the contract each mechanism must meet; a platform that cannot meet one documents the deviation in its client rather than silently weakening the boundary. - The sandbox communicates with the host via a narrow IPC channel that exchanges only the produced pixel buffer (or an error code) — not arbitrary structured data. - **The sandbox is allowed to crash.** A decoder CVE that triggers a segfault kills the sandbox, not the app. The host process logs the crash, surfaces "asset failed to decode," and continues. The sandbox is restarted on the next decode request. - **Local-origin assets** (this device was the uploader and the bytes have never left local storage) bypass the sandbox at the user's option — they have not crossed a trust boundary. By default the sandbox is still used uniformly, because the modest perf cost is worth the categorical guarantee. @@ -64,6 +64,6 @@ The validation duties above translate directly to test surface. Most live in `ca - **Forward-state read surface (unit).** Present a sidecar with unknown CBOR keys and (opt-in) a higher `sidecar_schema`; assert known fields render, the non-destructive "newer version" indicator shows, editing is disabled, and any write-back attempt is refused *without* stripping the unknown keys. - **Sandbox crash isolation (smoke per platform).** Feed the sandbox a known-CVE corpus; assert the host process survives every crash; assert the asset is surfaced as "unreadable on this device" and not removed from the library. - **Sandbox boundary (smoke per platform).** Assert the sandbox cannot read the parent process's filesystem, open network sockets, or write outside its scratch area. Per-platform fixtures verify each restriction. -- **Forbidden-behavior tripwire (unit).** For each item in the [forbidden-behaviors checklist](/design/threat-model/schema-rules/#forbidden-client-behaviors), a unit test confirms that calling the corresponding `capsule-core` API in the forbidden way panics or returns a structural error (so a buggy client cannot accidentally do the wrong thing). +- **Forbidden-behavior tripwire (unit).** For each item in the [forbidden-behaviors checklist](/design/threat-model/schema-rules/#forbidden-client-behaviors) **that maps to a `capsule-core` API** — backdating into signed structures, stripping unknown sidecar fields, overwriting provenance, signing for an epoch the client does not hold — a unit test confirms the API panics or returns a structural error (so a buggy client cannot accidentally do the wrong thing). The auth-surface items (notably `revoke_all_sessions` without master-key proof) are server-enforced and tested in `capsule-api-auth`, not in core. There is no client-only E2E case; the closest cross-module test is the upload-and-display round-trip used by the [Import](/design/import/) pipeline, which is bounded E2E in [Module Map](/design/module-map/#e2e-test-surface). diff --git a/capsule-docs/src/content/docs/design/cryptography/keys.md b/capsule-docs/src/content/docs/design/cryptography/keys.md index 9fe0ec37..6429deeb 100644 --- a/capsule-docs/src/content/docs/design/cryptography/keys.md +++ b/capsule-docs/src/content/docs/design/cryptography/keys.md @@ -57,7 +57,7 @@ Each device's keys are cross-signed into the [device directory](#device-director 1. **DSK** (Device Signing Key): hybrid **Ed25519 + ML-DSA-65**. 2. **DEK** (Device Encryption Key): hybrid **X25519 + ML-KEM-768**. -Both are signed by the IK (hybrid signature). Device private keys are **generated inside and never leave hardware** — Secure Enclave (iOS), StrongBox/Keystore (Android), TPM (desktop) — and are non-exportable. Because they cannot be backed up, devices are treated as disposable: a lost device is removed and a new one re-bootstrapped from the master key. +Both are signed by the IK (hybrid signature). The **classical half** of each device key is **generated inside and never leaves hardware** — Secure Enclave (iOS), StrongBox/Keystore (Android), TPM (desktop) — and is non-exportable; the **PQ half** (ML-DSA-65 / ML-KEM-768) is software-sealed under hardware-protected material, because no shipping secure element implements the PQ algorithms. Shipping elements also expose **P-256**, not Ed25519/X25519, so the hardware-backed composition is the planned **P-256 hybrid variant**; the software composition (both halves in software) is what integrates end-to-end today. Because hardware halves cannot be backed up, devices are treated as disposable either way: a lost device is removed and a new one re-bootstrapped from the master key. A device key can be revoked without affecting the user's identity or other devices. Revocation is done by signing a revocation statement with the IK and publishing it to a well-known location. The server then refuses to deliver new key wraps to that device, and remaining devices rotate any group keys the revoked device had access to. The revoked device's directory entry is **retained** — marked with `revoked_at` (RFC3339), never deleted — so the manifests it signed *before* revocation stay verifiable forever (provenance is append-only; see [Provenance](/design/cryptography/provenance/#what-an-attacker-with-all-current-keys-still-cannot-do)). diff --git a/capsule-docs/src/content/docs/design/device-enrollment.md b/capsule-docs/src/content/docs/design/device-enrollment.md index f4a5c1fa..af584528 100644 --- a/capsule-docs/src/content/docs/design/device-enrollment.md +++ b/capsule-docs/src/content/docs/design/device-enrollment.md @@ -18,7 +18,7 @@ When a user creates a brand-new Capsule account, the very first device runs the 1. **Generate the master key.** A 32-byte CSPRNG draw becomes the account master key. It is wrapped under a recovery passphrase via [Argon2id](/design/cryptography/primitives/#password-based-kdf); the wrapped blob is uploaded to the server-side [master-key escrow](/design/backup-recovery/#master-key-escrow). The plaintext recovery passphrase is shown to the user and never persisted. 2. **Generate the User IK.** A hybrid Ed25519 + ML-DSA-65 keypair (the [User Identity Keys](/design/cryptography/keys/#user-identity-keys-user-iks)). The private halves are wrapped under the master key; the public halves go into the (initial, single-member) device directory. -3. **Generate this device's keys.** A DSK (hybrid Ed25519 + ML-DSA-65) and a DEK (hybrid X25519 + ML-KEM-768), both generated inside the hardware secure element and non-exportable. Both are signed by the IK and added to the device directory. +3. **Generate this device's keys.** A DSK and a DEK per the [device-key composition](/design/cryptography/keys/#device-keys). The **classical half** of each is generated inside the hardware secure element and is non-exportable; shipping secure elements (Secure Enclave, StrongBox, TPM) expose **ECDSA/ECDH-P256**, not Ed25519/X25519, so the hardware-backed composition is the planned **P-256 hybrid variant** ([Keys](/design/cryptography/keys/)), with the PQ half software-sealed under hardware-protected material — no secure element holds PQ keys. Until the P-256 variant lands, the software composition generates both halves in software. Both keys are signed by the IK and added to the device directory. 4. **Publish the device directory.** The IK-signed directory is uploaded to the server. 5. **Create the default album.** Establish the owner's [default album](/design/organization/#the-default-album) — a new MLS group at the `album_id` derived from the master key (see [Keys — Key Chain](/design/cryptography/keys/#key-chain)), with this device as the sole admin/writer — and set the owner's `default_album_id` pointer ([Filesystem — Server](/design/filesystem/server/#ownership-partitioning-and-quota)) to it. This guarantees a writable import destination from the first moment the account exists. 6. **Show the recovery passphrase.** This is the only path back into the account if every device is lost, so saving it is **gated, not advisory**: the user must type back a short slice of the passphrase before setup completes, forcing them to actually record it rather than dismiss the screen. The plaintext passphrase is never persisted. @@ -32,7 +32,7 @@ Two design points: When an existing signed-in device adds a new device to the same account: -1. **Initiate from the existing device.** The user opens "Add another device" on device A (already signed in). Initiating an add requires a **fresh local device authorization** on A (biometric or device passcode) — a valid session token alone is **not** sufficient, so an attacker holding only a stolen session token cannot enroll a rogue device without physical control of A. Device A then generates a one-time **enrollment code** — **single-use, ≥64 bits of entropy, valid 10 minutes**, scoped to this one ceremony, collision-checked at generation, and deleted by the server on redemption or expiry — and displays it as a QR code (with a text fallback). +1. **Initiate from the existing device.** The user opens "Add another device" on device A (already signed in). Initiating an add requires a **fresh local device authorization** on A (biometric or device passcode) — a valid session token alone is **not** sufficient, so an attacker holding only a stolen session token cannot enroll a rogue device without physical control of A. Device A then generates a one-time **enrollment code** — **single-use, valid 10 minutes**, scoped to this one ceremony, collision-checked at generation, rate-limited at redemption, and deleted by the server on redemption or expiry; the QR payload carries **≥64 bits of entropy** (the text fallback may be shorter — see the presentation note below) — and displays it as a QR code (with a text fallback). 2. **Scan or enter on the new device.** Device B scans the QR (or types the code). 3. **Establish a short-lived channel.** Devices A and B perform an ephemeral X25519 ECDH to derive a one-time channel key, carried over a **server relay by default, or a direct LAN connection when both devices are on the same network** (discovered via mDNS; LAN preferred — fewer moving parts, no relay trust). The channel is mutually authenticated by the enrollment code plus the ephemeral DH. 4. **Verify the channel.** A short safety code derived from the channel transcript is displayed on both devices, **alongside each device's identity (model + a short key fingerprint)**; the user confirms both that the codes match and that the device being added is the one physically in front of them. Binding the code to device identity defends against a MITM on the relay channel and against a relay that swaps in a different device. @@ -42,7 +42,7 @@ When an existing signed-in device adds a new device to the same account: Two presentation choices: -- **Enrollment code.** Presented as a QR code with a **friendly numeric** text fallback — the channel is independently authenticated by the safety code, so the code itself only needs to be conveniently transcribable, not dense. Entropy (≥64-bit), single-use, and 10-minute expiry are fixed in step 1. +- **Enrollment code.** Presented as a QR code carrying the full ≥64-bit payload, with a **deliberately shorter, friendly numeric text fallback** (8–10 digits) that trades entropy for transcribability. The shorter fallback is safe to offer because it never stands alone: redemption is single-use, expires in 10 minutes, and is rate-limited per pending enrollment (so it cannot be brute-forced within its lifetime), and channel integrity never rests on the code — the safety-code check in step 4 is the MITM defense. Single-use and expiry for both forms are fixed in step 1. - **Safety-code check.** Step 4 binds the code to each device's identity (model + key fingerprint). To make the human comparison failure-resistant, both devices show the code in the same chunked, fixed-length format, and confirming requires an explicit match-and-identity acknowledgement on **both** devices — a mismatch is the abort path, not a missed default. ## Relationship to Cross-Device Recovery diff --git a/capsule-docs/src/content/docs/design/federation.md b/capsule-docs/src/content/docs/design/federation.md index a5546707..679e0d4e 100644 --- a/capsule-docs/src/content/docs/design/federation.md +++ b/capsule-docs/src/content/docs/design/federation.md @@ -57,12 +57,13 @@ A federation capability token is an EdDSA-JWT with the following claims: | ---------------------- | -------- | ---------------------------------------------------------------------------------------- | | `iss` | string | The issuing home server (`home.tld`). | | `sub` | string | The peer server identity (`other.tld`). | -| `aud` | string | The album id this capability scopes to (`urn:capsule:album:UUID`). | -| `scope` | enum | `read` (full) or `read-derivative-only` (thumbnails and previews only, never originals). | +| `aud` | string | The album id this capability scopes to (`urn:capsule:album:UUID`). Deviates from RFC 7519's recipient-oriented `aud` (the recipient is `sub`); verifiers MUST be configured to match `aud` against the *album*, never against themselves. | +| `scope` | enum | `read` (full) or `read-derivative-only` (thumbnails and previews only, never originals). Enforced structurally: every blob's **role** is server-visible (recorded on its index row and named by its signed [envelope object](/design/cryptography/provenance/#asset-manifest)), so a `/blob/{hash}` fetch under a derivative-only capability is refused when the hash's role is `original`. | +| `iat` | RFC 3339 | Issued-at; the anchor `exp` is bounded against. | | `exp` | RFC 3339 | Expiry; never more than **24 h** after `iat`. | | `nbf` | RFC 3339 | Not-before; clock-skew tolerance against the peer's wall-clock. | | `jti` | UUIDv7 | Unique token identifier; the revocation key. | -| `min_protocol_version` | string | Lowest `protocol_version` the issuing server still serves; matches the album's pin. | +| `min_protocol_version` | string | The **album's pinned** `protocol_version` — every event in the album conforms to it, so the peer selects its parser from this claim. (The issuing server's own supported window is discovered via `.well-known/capsule/server-info`, never from this claim.) | Signed under the home server's signing key — classical Ed25519 only, per the [operational-signature carve-out](/design/cryptography/primitives/#signature-scheme). @@ -71,7 +72,7 @@ Signed under the home server's signing key — classical Ed25519 only, per the [ 1. **Issuance.** A user on `home.tld` shares an album with `alice@other.tld`. `home.tld` mints a capability token for `other.tld` and delivers it as part of the share-invite message to Alice's client. Alice's client posts the token to `other.tld`; `other.tld` caches it server-side and uses it on every subsequent pull. 2. **Verification.** Capsule (the verifier, `home.tld` in this case) verifies the token offline against its own published signing key — no third-party PKI, no network call to a notary except for key rotation (see [Server Identity and Key Rotation](#server-identity-and-key-rotation)). 3. **Refresh.** A token nearing `exp` is replaced by `other.tld` requesting a new one on Alice's behalf; the request is itself authenticated by the previous token. Idempotency keyed by `(peer_id, jti)` per [Threat Model — Idempotency Invariants](/design/threat-model/validation/#idempotency-invariants). -4. **Revocation.** Revocation is a short TTL (`exp ≤ 24h`) plus a published **revocation list** at `/.well-known/capsule/revoked-jti`. Peers fetch and cache the list with a **maximum staleness of 15 minutes**. A peer holding a revoked-but-not-yet-expired token will still be honored for up to 15 minutes after revocation — this is the deliberate trade-off between revocation latency and revocation-list polling overhead. **List unavailability fails closed:** a verifier that relies on a *cached* copy of an issuer's revocation list and cannot refresh it must reject, past the 15-minute bound, any token whose `jti` it can no longer confirm against a current list — it never honors tokens indefinitely on a stale list. The `exp ≤ 24h` ceiling caps the worst case regardless, but the explicit rule means revocation cannot be outlived by making the list unreachable. (A server verifying its *own* tokens checks its own always-fresh list and is never stale.) +4. **Revocation.** Revocation is a short TTL (`exp ≤ 24h`) plus a published **revocation list** at `/.well-known/capsule/revoked-jti`. Peers fetch and cache the list with a **maximum staleness of 15 minutes**. A peer holding a revoked-but-not-yet-expired token will still be honored for up to 15 minutes after revocation — this is the deliberate trade-off between revocation latency and revocation-list polling overhead. **List unavailability fails closed:** a verifier that relies on a *cached* copy of an issuer's revocation list and cannot refresh it must reject, past the 15-minute bound, any token whose `jti` it can no longer confirm against a current list — it never honors tokens indefinitely on a stale list. The `exp ≤ 24h` ceiling caps the worst case regardless, but the explicit rule means revocation cannot be outlived by making the list unreachable. (A server verifying its *own* tokens checks its own always-fresh list and is never stale.) Revoked `jti`s are **pruned** from the published list once their `exp` passes — an expired token is rejected unconditionally anyway — so the list stays bounded by at most 24 hours of revocations. 5. **Expiry.** A token past `exp` is rejected unconditionally; the verifier returns `401` and the peer must obtain a fresh token before continuing. This capability is a **transport-scoped control, not a confidentiality control**: it gates *who may fetch at all* (rate-limiting, anti-enumeration, clean revocation of a sharing relationship), nothing more. Confidentiality is already enforced by [MLS album membership](/design/cryptography/mls/) — without the album master key, fetched bytes are unreadable. @@ -84,7 +85,7 @@ Every byte from a peer crosses a hard boundary before it is trusted. The exhaust - **Closed enums.** `action`, `content_type`, `DerivativeManifest.role`, and `gps.source` are closed per protocol version. An unknown value is a structural error, not a "future to ignore." - **Hard caps.** Size caps on every field, depth caps on nested structures, length caps on bounded collections (e.g. `superseded_captions ≤ 16`), rate caps per peer. No unbounded input reaches a parser. - **Unknown fields within a known schema preserved, never executed.** Top-level unknown fields are rejected; field-level unknown CBOR keys within a known schema are preserved verbatim for forward compatibility but are never interpreted. -- **Manifest envelope checks.** All items 1–18 of [Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants) apply — `protocol_version` in range, `crypto_suite_id` in inventory, hash length matches the suite's digest size, declared size against received bytes, `created_by_device` in the user's device directory, `timestamp` within the sanity bound, monotonic `amk_version`, and the [stale-revival check](/design/import/download-sync/#stale-revival-detection) on `prior_provenance_hash`. +- **Manifest envelope checks.** All items 1–18 of [Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants) apply — `protocol_version` in range, `crypto_suite_id` in inventory, hash length matches the suite's digest size, declared size against received bytes, `created_by_device` in the user's device directory, `timestamp` within the sanity bound, monotonic `amk_version`, and the [stale-revival check](/design/import/download-sync/#stale-revival-detection) on `prior_provenance_hash`. Where a pulled bundle carries a metadata blob, **invariant 25** (the `metadata_blob_hash` match) applies too — federation never unlocks looser rules. (Invariants 26–32 are drop-path-only and cannot arise on a pull.) - **Capability token.** Items 19–21 of the same list: token verifies under the home server's signing key, `exp` in future, `jti` not in the revocation list, per-peer rate budgets unbroken. - **The parser is a security boundary.** Capsule's decoders for federated input are written in memory-safe Rust against audited libraries (`ciborium`, `serde_cbor`); we explicitly assume the host language and decoder are memory-safe (the same assumption [Security Against Malicious Files](#security-against-malicious-files) makes at the client edge). Decoder CVEs in client decode paths for *opaque media bytes* are handled by the [sandboxed decoder](/design/clients/#sandboxed-decoder), not by re-implementing the decoder. The federation CBOR decode path is additionally fuzzed. @@ -146,7 +147,7 @@ Federation introduces moderation hooks for handling abuse across servers; the fu ## Server Identity and Key Rotation - Server-to-server requests are signed under the server's signing key (classical Ed25519 only, per the [operational-signature carve-out](/design/cryptography/primitives/#signature-scheme)), published at a well-known path. Matrix, ActivityPub (HTTP Signatures), and AT Protocol all converge on this pattern. -- Servers cache each other's public keys (TOFU-pinned on first contact). A rotation is confirmed by a **perspective check**: before accepting a rotated key, a peer corroborates it against one or more independent vantage points (other servers, or a configured notary) and accepts only on agreement — so a single compromised network path cannot substitute a forged key. A rotation that fails corroboration is surfaced, not silently accepted. This is the mechanism behind [Threat Model — scenario #26](/design/threat-model/scenarios/#damage-scenario--invariant-map). +- Servers cache each other's public keys (TOFU-pinned on first contact). A rotation is confirmed by a **perspective check**: before accepting a rotated key, a peer corroborates it against **at least two** independent vantage points — federated servers it already trusts, or a deployment-configured notary — and accepts only on **unanimous** agreement, so a single compromised network path (or a single colluding vantage) cannot substitute a forged key. A rotation that fails corroboration is surfaced, not silently accepted. This is the mechanism behind [Threat Model — scenario #26](/design/threat-model/scenarios/#damage-scenario--invariant-map). - Album protocol versions are pinned per album — see [Album Protocol Version Pinning](/design/versioning/#album-protocol-version-pinning). ## Validation diff --git a/capsule-docs/src/content/docs/design/import/download-sync.md b/capsule-docs/src/content/docs/design/import/download-sync.md index a213a688..ba5001b8 100644 --- a/capsule-docs/src/content/docs/design/import/download-sync.md +++ b/capsule-docs/src/content/docs/design/import/download-sync.md @@ -44,7 +44,7 @@ The default policy follows the per-library setting in [Synchronization Scope](#s Because every blob is content-addressed, a fetch is skipped entirely when the blob is already in the local cache — the client looks up its cache by hash before issuing any request, so a representation shared between assets (an identical thumbnail, a merged original) is only ever fetched once. -**When an above-tier fetch cannot succeed.** A lazily-fetched representation may be temporarily or permanently unavailable. The client distinguishes the two: a **transient** failure (network drop, `5xx`) retries with backoff and resumes via `Range`; a **permanent** failure (`410 Gone`, `403`, a purged origin, or an unreachable [federated home server](/design/federation/#robustness-against-connectivity-loss)) **degrades gracefully** to the best representation already in hand — preview → thumbnail → LQIP, down to the always-present LQIP — and surfaces a non-destructive "full resolution unavailable" state on the asset. It never thrashes the fetch, and it never removes the asset's metadata or local index entry over a missing derivative. The asset stays listed and re-fetches automatically once the representation becomes reachable again. +**When an above-tier fetch cannot succeed.** A lazily-fetched representation may be temporarily or permanently unavailable. The client distinguishes the two: a **transient** failure (network drop, `5xx`) retries with backoff and resumes via `Range`; a **permanent** failure (`410 Gone`, a purged origin, or an unreachable [federated home server](/design/federation/#robustness-against-connectivity-loss)) **degrades gracefully** to the best representation already in hand. A **`403`** is neither: it signals an *authorization change*, not a durability loss — the client re-syncs its membership/capability state for the album before retrying, and only then degrades (the asset may have been unshared), so a revocation event is surfaced as such rather than masked as a missing file — preview → thumbnail → LQIP, down to the always-present LQIP — and surfaces a non-destructive "full resolution unavailable" state on the asset. It never thrashes the fetch, and it never removes the asset's metadata or local index entry over a missing derivative. The asset stays listed and re-fetches automatically once the representation becomes reachable again. ## Resumption and Verification @@ -79,7 +79,7 @@ Auto sync is implemented **only** if it can be guaranteed to behave appropriatel When the auto sync criteria have not been met for a prolonged period — **two weeks** specifically — the library falls silently out of date, which defeats the purpose of a backup. The client surfaces this rather than letting it pass unnoticed: -- After two weeks without a completed sync *while changes remain un-synced*, the user is notified that the library is behind and offered a one-tap **force sync now**, which proceeds regardless of the metered/Wi-Fi criteria with their explicit consent. +- After two weeks without a completed sync *while changes remain un-synced*, the user is notified that the library is behind and offered a one-tap **force sync now**, which proceeds regardless of the metered/Wi-Fi criteria with their explicit consent. (A device idle long enough to trigger this may also be approaching the session's [sliding inactivity expiry](/design/authentication/#sliding-inactivity-expiry); the force-sync flow routes through re-authentication when the session has lapsed rather than failing the sync.) - The notification can be **snoozed** until a later date (e.g. another two weeks) or **disabled** outright. Snoozing only suppresses the warning; disabling opts out of the warning entirely and does not affect auto sync itself. ## Synchronization Scope diff --git a/capsule-docs/src/content/docs/design/import/pipeline.md b/capsule-docs/src/content/docs/design/import/pipeline.md index 9d23dfc8..75961bd8 100644 --- a/capsule-docs/src/content/docs/design/import/pipeline.md +++ b/capsule-docs/src/content/docs/design/import/pipeline.md @@ -35,7 +35,7 @@ The planner is **pure**: given the scanned files and their extracted metadata, i - If an asset is already uploaded *locally* in the library, import refuses it — no merge needed. - If an asset already exists *remotely* under a different ciphertext (e.g. re-encrypted under a newer album key), import still admits it; the [upload protocol](/design/import/upload-protocol/#deduplication-and-merge) then resolves it as a merge (the existing blob is linked rather than re-uploaded). -**Destination resolution.** Each added asset is assigned a destination [container album](/design/organization/#container-albums). If the user picks one explicitly the planner uses it; otherwise it calls `resolve_default_album(context)` — the active scope's override, else the owner [default-album](/design/organization/#the-default-album) pointer, else the derived de facto album. To keep the planner pure, the active context and a snapshot of the pointer/overrides are planner *inputs*, so the resolved `album_id` is deterministic and recorded in the `ImportPlan` rather than discovered later at upload time. +**Destination resolution.** Each added asset is assigned a destination [container album](/design/organization/#container-albums). If the user picks one explicitly the planner uses it; otherwise it calls `resolve_default_album(context)` — the active scope's override, else the owner [default-album](/design/organization/#the-default-album) pointer, else the derived de facto album. To keep the planner pure, the active context and a snapshot of the pointer/overrides are planner *inputs*, so the resolved `album_id` is deterministic and recorded in the `ImportPlan` rather than discovered later at upload time. The snapshot is last-writer-wins by design: a concurrent re-point of the default album after the user confirms does **not** retarget the confirmed plan — the recorded `album_id` stands, exactly as if the user had picked that album explicitly. The planner's purity is what lets it be unit-tested exhaustively without filesystem fixtures: every edge case (overlapping selections, mixed formats, sidecar pairing, partial state from a prior interrupted import) becomes a table of `(scan_input → expected_plan)` pairs. @@ -68,6 +68,8 @@ It is **auto-detected**: when the [free-space probe](#plan--confirm) reports `to Peak local disk is therefore bounded to the in-flight window, not the whole import, so an import far larger than free space can complete. The verify-before-release step *is* the [verify-before-destroy](/design/import/storage-verification/#verify-before-destroy) rule applied per asset — the device never drops the only copy of bytes the server has not confirmed it holds. +**Minimum headroom.** The window must fully materialize at least one asset locally (original + derivatives + metadata) before that asset's upload and release complete, so streaming mode requires headroom for the **largest single asset in the plan**. If even one asset exceeds free space, the plan surfaces a hard "cannot import *X* without freeing *Y*" error at confirmation rather than stalling mid-stream — streaming bounds peak usage to the window; it cannot make a single file smaller than the disk. + **Halt on lost connectivity.** Streaming mode depends on the server: releasing local copies is safe only because the server is confirmed to hold them. If the connection to the server drops, the pipeline **pauses** — it stops admitting new source files into the library (continuing would refill the very disk the mode exists to spare) and waits, rather than importing ahead into space it does not have. In-flight uploads resume via the protocol's [`HEAD` resumption](/design/import/upload-protocol/#session-lifecycle) once connectivity returns; if the source media itself is removed mid-import, the deterministic planner re-derives the remaining work on resume. Bulk streaming import is a *large reconciliation* and obeys the same metered/Wi-Fi [connection rules](/design/import/download-sync/#synchronization-criteria) as auto-sync. **Quota.** Streaming creates one upload session per asset, so server [quota](/design/quota/#enforcement-points) is enforced exactly as for any upload — at session creation, no new enforcement point. If quota is exhausted mid-stream the next session creation is refused; the pipeline pauses and surfaces the remediable state rather than failing the whole import, and no local original is released for an asset that did not upload. diff --git a/capsule-docs/src/content/docs/design/import/storage-verification.md b/capsule-docs/src/content/docs/design/import/storage-verification.md index 89893527..e38767bf 100644 --- a/capsule-docs/src/content/docs/design/import/storage-verification.md +++ b/capsule-docs/src/content/docs/design/import/storage-verification.md @@ -5,7 +5,7 @@ description: The endpoint a client calls to confirm an asset is durably stored, [Upload finalization](/design/import/upload-protocol/#finalization-and-integrity) confirms, **once**, that the bytes the server assembled match the declared hash. That `Completed` acknowledgement is a transfer receipt, not a standing durability guarantee a client can re-check later — and [`verify_asset`](/design/cryptography/keys/#write-authorization) proves *cryptographic* validity and authorization, never that the server still holds the bytes. A client that is about to discard local data therefore has no way, today, to ask the server: *do you actually still have this, indexed and retrievable?* -This doc defines that missing query and the rule that every client follows before any destructive local action. The endpoint lives in `capsule-api-media` (it answers from the blob store plus the Postgres index); the client-side gate is a pure predicate in `capsule-core` invoked from `capsule-sdk`. +This doc defines that missing query and the rule that every client follows before any destructive local action. The endpoint lives in `capsule-api-media` (planned; it answers from the blob store plus the Postgres index); the client-side gate is a pure predicate in `capsule-core` invoked from `capsule-sdk` (planned — the offline core's `verify_asset` already covers the complementary crypto-validity half of the gate). ## What "Safely Stored" Means @@ -17,6 +17,8 @@ A blob being content-addressed and hash-matching is necessary but not sufficient A `durable` verdict requires all three to hold for every **required** blob of the asset — its original and metadata blobs, plus any derivative the client declares it is relying on. +`durable` attests **this home server's** storage only. It says nothing about replicas, peers, or off-server backups — multi-location redundancy is the [backup artifact](/design/backup-recovery/)'s job — and a deployment running its own replication layer still attests only what this server can itself `stat` and serve. + ## Endpoint | Method | Path | Purpose | @@ -55,7 +57,7 @@ StorageVerification { ``` - A hash the client lists that the server does not associate with the asset comes back `stored=false, indexed=false` — surfaced, never silently omitted. -- **`deep`** (default `false`) asks the server to re-read and re-hash the blob bytes rather than trusting the `stat` + index, catching silent bit-rot at the cost of I/O. It is opt-in because the structural check answers the common case (did finalization actually durably land?) cheaply, while a deep scan is for periodic integrity audits. +- **`deep`** (default `false`) asks the server to re-read and re-hash the blob bytes rather than trusting the `stat` + index, catching silent bit-rot at the cost of I/O. It is opt-in because the structural check answers the common case (did finalization actually durably land?) cheaply, while a deep scan is for periodic integrity audits. `deep` is a server-priced operation: rate-limited per user and coalesced server-side (a repeated `deep` request against the same blob within a window returns the cached result), so a client cannot turn it into an I/O-amplification attack. The endpoint is deliberately cheap and idempotent so clients can call it freely on the destructive path below. @@ -78,6 +80,8 @@ The gate **does not** apply to: A non-`durable` verdict never triggers a destructive action: the client retains the local copy, retries verification with backoff, and surfaces the asset as "not yet confirmed on server" rather than silently dropping it. +**The verdict is a point-in-time fact**, so the verify→release window is kept tight on both sides: the client re-verifies if more than a bounded interval (default 60 s) elapses between verdict and release, and on the server a blob that just answered `durable` cannot reach byte deletion faster than the standing [GC grace window](/design/filesystem/server/#deletion-and-garbage-collection) — the verdict plus that grace period is what makes the gate sound without a lease protocol. + ## Relationship to Other Checks - **vs. upload finalization.** Finalization's `Completed` is a one-time receipt for one transfer; `/storage/verify` is the re-checkable, after-the-fact confirmation that the receipt still holds — across app restarts, across devices, and after server-side GC or migration could have changed state. diff --git a/capsule-docs/src/content/docs/design/mls-resilience.md b/capsule-docs/src/content/docs/design/mls-resilience.md index 06fae6d8..6e339a5e 100644 --- a/capsule-docs/src/content/docs/design/mls-resilience.md +++ b/capsule-docs/src/content/docs/design/mls-resilience.md @@ -21,7 +21,7 @@ A device sends an MLS commit (e.g. an `Add` or AMK rotation) and the server neve Two devices' local MLS state has diverged — different views of the group's current epoch, different write-tier key, different member list. This can happen after a buggy commit, an incomplete sync, or a long offline period. -**Recovery direction:** the device with the older epoch reconciles by replaying every commit it missed from the server's chain. A device whose local state is *ahead* of the server — it holds a commit whose hash is **absent from the server's authoritative chain** (a local state-mutation bug, or a commit the server never persisted) — declares itself unreconcilable, discards its local group state, and **re-bootstraps in full**. Partial re-bootstrap is deliberately not attempted: MLS group state is small, so a clean full re-fetch is simpler to reason about than splicing suspect local state, and is the only path taken. +**Recovery direction:** the device with the older epoch reconciles by replaying every commit it missed from the server's chain. A device whose local state is *ahead* of the server — it holds a commit whose hash is **absent from the server's authoritative chain** — first treats that as a [lost commit](#lost-commit) and re-submits within that path's backoff budget; **only after the budget is exhausted**, or when the absence is a *provable fork* (the server's chain has advanced past the local commit's parent with a different commit, so re-submission can never land), does it declare itself unreconcilable, discard its local group state, and **re-bootstrap in full**. The discriminator matters: without it, an honestly-lost commit (e.g. an AMK rotation the server never persisted) would route to discard and be silently dropped instead of retried. Partial re-bootstrap is deliberately not attempted: MLS group state is small, so a clean full re-fetch is simpler to reason about than splicing suspect local state, and is the only path taken. ### Concurrent commits with the wrong ordering diff --git a/capsule-docs/src/content/docs/design/moderation.md b/capsule-docs/src/content/docs/design/moderation.md index e1785b0e..4e99fd0b 100644 --- a/capsule-docs/src/content/docs/design/moderation.md +++ b/capsule-docs/src/content/docs/design/moderation.md @@ -3,7 +3,7 @@ title: Moderation description: Server moderation policy — reports, suspensions, takedowns, blocklists, federated reporting --- -Capsule is end-to-end encrypted, so a server **cannot** scan content it holds — server-side content or CSAM scanning is impossible by design, and no content scanner will be built. Moderation operates entirely on what *is* available: user reports, account-level signals, and federated peer reputation. +Capsule is end-to-end encrypted, so a server **cannot** scan content it holds — server-side content or CSAM scanning is impossible by design, and no server-side content scanner will be built. (Client-side, opt-in ML over a user's *own* library — [AI/ML](/design/ai/) — is a different thing entirely: it runs where the keys are, under the user's control; its candidate shared-album-flagging classifiers are client-side and user-initiated, never a server scanner.) Moderation operates entirely on what *is* available: user reports, account-level signals, and federated peer reputation. Implementation will live in `capsule-api::moderation` (a new sub-crate or service inside `capsule-api`). The boundary surfaces — report submission, federated report exchange, blocklist publication — are the eventual contract; this doc captures what they will need to do. @@ -44,8 +44,8 @@ Server-level blocklists, plus per-user blocks that federate: A server admin can suspend a user account on their home server. Suspended accounts: - Cannot upload — `POST /upload` session creation is refused with a structured `403 AccountSuspended` code (distinct from quota and permission rejections, so the client surfaces the right remediation). -- Cannot share new albums (existing shares remain valid for the share-link TTL; revocation lists can revoke them). -- Cannot revoke other devices' sessions (a suspended account's `revoke_all_sessions` is refused — defends against compromised-account-as-DoS). +- Cannot share new albums or create new share/upload links (existing share links keep serving until they expire or are revoked — [share links](/design/share-links/) have optional expiry plus revocation, no implicit TTL). +- **Can still secure the account**: `revoke_all_sessions` remains available to a suspended user — it is gated by [master-key proof](/design/authentication/#explicit-revocation), not by account standing, so it cannot be abused by a session-token thief, and a suspended user whose account may be compromised needs it most. Suspension removes upload and sharing capability, never the ability to evict sessions. The user's *data* is untouched — suspension is an access-level action, not a data-level one. Reversibility (a suspension can be lifted) is the default; permanent termination is a separate policy. @@ -54,7 +54,7 @@ The user's *data* is untouched — suspension is an access-level action, not a d When a moderation action requires the *home server* to stop serving a specific asset (e.g. legal request, CSAM report verified by admin viewing in their album): - The asset is marked unservable on the home server (`served = false` in the index). -- Federated peers fetching the asset receive `410 Gone`. +- Federated peers fetching the asset receive `410 Gone`. (This deliberately diverges from the [share-link and drop serve paths](/design/share-links/#security-contract), which return an indistinguishable `404` — those must not confirm a capability URL ever existed, while a takedown *intends* to signal removal of content whose existence the peer already knows. The per-surface rule: capability-URL serving → `404`; takedown of known content → `410`.) - The asset's underlying blob is **not** deleted — the user owns the data, and a takedown is a serving constraint, not a destruction; the user can still restore from their own backup. A takedown is therefore **reversible by default** (an admin can lift it). A **legal-hold** variant marks the asset indefinitely unservable where law requires it — lifted only when the legal obligation ends, not at admin discretion — but even then never destroys the user's bytes: the constraint is on the *home server's serving*, not on the data the user holds. - The takedown emits a **server-visible moderation provenance record** the user sees in their audit log — what was taken down, when, and (where policy permits) why — honoring the "[No silent operations](#what-moderation-cannot-do-structural)" rule. A user whose asset stops serving is never left to guess why, and the moderation action is itself auditable after the fact. diff --git a/capsule-docs/src/content/docs/design/peering.md b/capsule-docs/src/content/docs/design/peering.md index b7657412..92e6f1fd 100644 --- a/capsule-docs/src/content/docs/design/peering.md +++ b/capsule-docs/src/content/docs/design/peering.md @@ -37,15 +37,15 @@ Even two of the same user's devices are separate failure-containment boundaries Discovery is the one genuinely new mechanism. Devices advertise a peering service over **mDNS** on the local network and accept connections over **TCP**. -Discovery is **LAN-only** — there is no relay, no internet-wide rendezvous. mDNS broadcasts are visible to every host on the segment, so the advertisement must not leak identity: a device advertises an **opaque, rotating service instance**, not `user@server.tld` or a device name. Whether two advertisements belong to the same user is established *inside* the encrypted channel (below), never from the broadcast itself. +Discovery is **LAN-only** — there is no relay, no internet-wide rendezvous. mDNS broadcasts are visible to every host on the segment, so the advertisement must not leak identity: a device advertises an **opaque service instance** — rotated at least per boot and at most every 24 hours (deployment-tunable within that band) — never `user@server.tld` or a device name. Whether two advertisements belong to the same user is established *inside* the encrypted channel (below), never from the broadcast itself. If no peer answers, discovery fails silently and the device proceeds with ordinary server sync. ## Establishing the Channel -A peer connection is HTTP over a **mutually authenticated TLS 1.3** channel. The certificates presented are the **device keys themselves** — there is no CA. Each side verifies that the other's device certificate carries a valid hybrid signature chaining to the shared User IK, exactly as published in the [device directory](/design/cryptography/keys/#device-directory). The directory *is* the trust anchor; a device not in it cannot complete the handshake. +A peer connection is HTTP over a **mutually authenticated TLS 1.3** channel. The certificates presented are the **device keys themselves** — there is no CA. Concretely: the TLS handshake authenticates with the device key's **classical half** (as a raw public key or self-signed certificate — TLS 1.3 has no ML-DSA certificate path), and the **hybrid** check — that the presented key carries a valid hybrid signature chaining to the shared User IK, covering both halves per the [signature scheme](/design/cryptography/primitives/#signature-scheme), exactly as published in the [device directory](/design/cryptography/keys/#device-directory) — runs at the application layer over the established channel, before any payload byte. The directory *is* the trust anchor; a device not in it cannot complete the handshake. -This doc covers sync between devices that are **already provisioned** — both already hold the account master key. Bootstrapping a brand-new device (handing it the master key for the first time) is **cross-device recovery** and is specified in [Device Enrollment](/design/device-enrollment/); peering does not re-document it. +This doc covers sync between devices that are **already provisioned** — both already hold the account master key. Bootstrapping a brand-new device (handing it the master key for the first time) is **cross-device add** — or, when every device was lost, **cross-device recovery** — owned by [Device Enrollment](/design/device-enrollment/) and [Backup and Recovery](/design/backup-recovery/#default-mechanisms) respectively; peering re-documents neither. ## Determining the Delta @@ -84,7 +84,7 @@ Peering does not fork a device's state away from the server. A peering-received Peering has two independently versioned surfaces, both checked **once, up front**, crashing early on mismatch per [Principles](/design/principles/) and the universal [protocol handshake](/design/threat-model/validation/#protocol-and-capability-negotiation): -- The peering **transport protocol** — date-based (`YYYY-MM-DD`), exchanged via `X-Capsule-Protocol` at channel establishment. Mismatch terminates the TLS connection **before any payload byte is sent** — `426 Upgrade Required` in the channel's framing layer. There is no degraded-mode fallback; peering simply fails and the device proceeds to ordinary server sync. +- The peering **transport protocol** — date-based (`YYYY-MM-DD`), exchanged via `X-Capsule-Protocol` at channel establishment. It shares the header name and date format with the client-server wire protocol but is its **own version space**: peering-protocol values compare only against peering-protocol values, and the peering transport revs independently of the server protocol. Mismatch terminates the TLS connection **before any payload byte is sent** — `426 Upgrade Required` in the channel's framing layer. There is no degraded-mode fallback; peering simply fails and the device proceeds to ordinary server sync. - The **artifact format** — versioned by [Backup and Recovery](/design/backup-recovery/#backup-artifact), so a newer device can still ingest an artifact built by an older one. The artifact's `crypto_suite_id` and album `protocol_version` are validated against the receiver's max known on ingest; a forward-jumping value is rejected (refuse-by-default), never best-effort-parsed. These two surfaces are independent: a device with up-to-date transport protocol may still receive an artifact format it does not implement (and vice versa). Both checks must pass before any bytes are applied to local state. diff --git a/capsule-docs/src/content/docs/design/quota.md b/capsule-docs/src/content/docs/design/quota.md index 6a961ebd..853655fa 100644 --- a/capsule-docs/src/content/docs/design/quota.md +++ b/capsule-docs/src/content/docs/design/quota.md @@ -32,8 +32,8 @@ A user account exists in one of these quota states: | ----------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **OK** | quota_used < soft_limit | All uploads succeed normally. | | **Soft warning** | soft_limit ≤ quota_used < hard_limit | Uploads succeed, but the UI surfaces a warning. | -| **Hard exceeded** | quota_used ≥ hard_limit | New uploads rejected at session creation with a structured error. Existing assets remain accessible. | -| **Grace expired** | quota_used ≥ hard_limit for > `grace_window` (default 14 days) | Read-only mode: reads, deletes, and restore-from-trash still work; only new uploads and metadata-growth writes are refused. Freeing space (emptying trash) lifts it. | +| **Hard exceeded** | quota_used ≥ hard_limit | New uploads rejected at session creation with a structured error. Metadata edits and every other write still work; existing assets remain accessible. | +| **Grace expired** | quota_used ≥ hard_limit for > `grace_window` (default 14 days) | Adds to Hard-exceeded: **all other metadata-growth writes** (caption/tag edits, new share or upload links) are now refused too. Reads, deletes, and restore-from-trash still work — the provenance/metadata writes a `delete`, `trash-restore`, or trash-empty itself produces are **always admitted** (a user must be able to delete their way back under quota). Freeing space lifts it. | | **Suspended** | (admin or billing action — see [Moderation](/design/moderation/)) | Server-defined; possibly upload refusal, possibly full lockout. | Defaults for `soft_limit`, `hard_limit`, and `grace_window` are deployment-configurable. Self-hosted servers might run with no quota (`hard_limit = ∞`); hosted services set per-tier limits. @@ -50,7 +50,7 @@ Where the quota check actually runs: ## Scope Decisions - **Sponsored-account attribution.** A sponsoree's uploads count against the **sponsor's** quota — the sponsoree's `upload_user_id` derives from the sponsor ([Keys — Delegated/Sponsored](/design/cryptography/keys/#delegatedsponsored-accounts)), so storage rolls up to the sponsoring (billing) account. There is no separate sponsoree quota. -- **Web-upload drops.** A pending [web-upload drop](/design/web-upload/) counts against the **provisioning user's** quota — the link owner is the `upload_user_id`, charged at [drop-session creation](/design/web-upload/#drop-and-adoption-lifecycle) like any other session, so a leaked link cannot push storage past the owner's hard limit. [Adoption in place](/design/web-upload/#why-adopt-in-place) only reclassifies the already-stored blob from inbox to album asset, so it incurs no new quota; a discarded or expired drop frees its bytes on the next GC. There is no separate inbox quota. +- **Web-upload drops.** A pending [web-upload drop](/design/web-upload/) counts against the **provisioning user's** quota — the link owner is the `upload_user_id`, charged at [drop-session creation](/design/web-upload/#drop-and-adoption-lifecycle) like any other session, so a leaked link cannot push storage past the owner's hard limit. [Adoption in place](/design/web-upload/#why-adopt-in-place) only reclassifies the already-stored blob from inbox to album asset, so the **bulk bytes** incur no new quota — the adoption's small metadata and provenance blobs are charged like any metadata write; a discarded or expired drop frees its bytes on the next GC. There is no separate inbox quota. - **Per-album quotas.** Out of scope for v1 — quota is per `upload_user_id` only. A deployment that later wants per-album caps adds them as a second, independent check at the same enforcement point; the accounting model above does not change. - **Streaming import.** A storage-constrained [streaming import](/design/import/pipeline/#import-upload-streaming-mode) creates one upload session per asset, so it is bounded by the same per-session check at creation — there is no new enforcement point. Quota exhaustion mid-stream refuses the next session; the pipeline pauses, and no local original is released for an asset that did not upload. - **Grace-window UX.** The structural rule is "upload session creation refused" in read-only mode; the client surfaces this as a discoverable, remediable state (what is full, what to delete) rather than an opaque mid-import error. Concrete copy is a client-UX detail. diff --git a/capsule-docs/src/content/docs/design/share-links.md b/capsule-docs/src/content/docs/design/share-links.md index 6f9a28c1..7e1cb770 100644 --- a/capsule-docs/src/content/docs/design/share-links.md +++ b/capsule-docs/src/content/docs/design/share-links.md @@ -32,7 +32,7 @@ These are **normative** — the security-relevant decisions are committed; only - **Passphrase unwrap is client-side.** When a passphrase protects a link, the server stores only the **wrapped** secret and never receives the passphrase: the client fetches the wrapped material and unwraps it locally via the [password-based KDF](/design/cryptography/primitives/#password-based-kdf). The server is never in the password-trust path, so a server compromise cannot brute-force passphrases beyond the [Argon2id](/design/cryptography/primitives/#password-based-kdf) cost already imposed. Because unwrap is client-side the server cannot observe a *failed* attempt, so the endpoint that returns the wrapped material is rate-limited per source IP and per `{opaque-id}` (the same limiter as the serve path); the Argon2id cost is the real brute-force backstop. - **Privacy strip on serve is mandatory.** The serve path **always** applies the boundary-crossing strip from [Metadata — Privacy on Export](/design/metadata/#privacy-on-export) (camera serial, device/session ids, GPS truncated to city level, contact tags). There is **no per-share opt-out** that could leak fingerprinting fields — a public share is, by definition, a boundary crossing. - **Home-server-only serving.** A share link is served **only by the album's [home server](/design/federation/#album-ownership-v1-single-home-server)**. A federated peer never serves a share; a share-scoped request at a peer returns a **structured `{ home_server }` JSON pointer** the client resolves — explicitly *not* an HTTP redirect, to avoid an open-redirect surface — never content. This keeps revocation and rate-limiting at a single authoritative point. -- **Revocation cache.** Per-link revocation is checked against a **short-TTL cache (default 60 s)** with the same fail-closed posture as the [federation revocation list](/design/federation/#token-lifecycle-and-chain-of-trust): a serve path that cannot confirm a link is still live past the TTL refuses rather than serving on stale-allowed state. +- **Revocation cache.** Home-server-only serving still leaves *intra-server* staleness: the serve path may run on several processes or replicas of the one home server, which consult revocation state through a **short-TTL cache (default 60 s)** rather than an authoritative read per request. The posture is fail-closed, matching the [federation revocation list](/design/federation/#token-lifecycle-and-chain-of-trust): a serving process that cannot confirm a link is still live past the TTL refuses rather than serving on stale-allowed state. (A single-process deployment reads revocation state directly and the cache is a no-op.) ## Contract Skeleton @@ -46,9 +46,11 @@ trait ShareLinkIssuer { } // in capsule-api-media::shares -// GET /s/{opaque-id} → metadata blob + LQIP (mandatory server-side strip — see Security Contract) -// GET /s/{opaque-id}/blob/{hash} → ciphertext blob; client decrypts using link-derived key -// POST /s/{opaque-id}/passphrase → if passphrase-wrapped, exchange passphrase for unwrap material +// GET /s/{opaque-id} → metadata blob + LQIP (mandatory server-side strip — see Security Contract) +// GET /s/{opaque-id}/blob/{hash} → ciphertext blob; client decrypts using link-derived key +// GET /s/{opaque-id}/wrapped-secret → the passphrase-WRAPPED link material, when the link is +// passphrase-protected; unwrap is client-side via the password-based +// KDF. The passphrase itself is NEVER transmitted (Security Contract). ``` Concrete error variants are an implementation detail; the rate-limit, opaque-id entropy, privacy-strip, and revocation policies are fixed by the [Security Contract](#security-contract) above. diff --git a/capsule-docs/src/content/docs/design/web-upload.md b/capsule-docs/src/content/docs/design/web-upload.md index d7142138..66d771a2 100644 --- a/capsule-docs/src/content/docs/design/web-upload.md +++ b/capsule-docs/src/content/docs/design/web-upload.md @@ -44,7 +44,7 @@ These are **normative** — the security-relevant decisions are committed; only - **Per-link caps are enforced server-side at the no-key layer.** Expiry, cumulative-byte cap, file-count cap, and per-file size cap are checked on every drop-session creation; an over-cap or expired/revoked link is refused. These bound a leaked link to *wasted quota and inbox space*, never to library corruption. - **Quota is charged to the provisioning user at drop-session creation.** A drop debits the link owner's quota at session creation — the single hard [enforcement point](/design/quota/#enforcement-points) — using the [`upload_user_id = owner_id` attribution](/design/quota/#accounting-model). A link cannot be used to push the owner past their hard limit. - **Serving-endpoint rate limits.** Drop-session creation is rate-limited **per source IP and per `{opaque-id}`** (two independent limiters), and a not-found, revoked, or expired link returns an **indistinguishable `404`** — never `410 Gone` — exactly as the [share-link serve path](/design/share-links/#security-contract), so probing reveals nothing. -- **Optional passphrase is an abuse gate, not a confidentiality layer.** A link may carry an optional passphrase (wrapped via the [password-based KDF](/design/cryptography/primitives/#password-based-kdf)); the guest must supply it to open a drop session. It limits *who may spend the owner's quota*; it adds no confidentiality, because the guest already encrypts every asset. The passphrase is verified the same client-out-of-band way share links use, never transmitted in the clear. +- **Optional passphrase is an abuse gate, not a confidentiality layer.** A link may carry an optional passphrase. Unlike a share-link passphrase — which wraps a *read* secret the client unwraps locally — this one gates a **write**, so the server must verify possession: the link record stores an [Argon2id](/design/cryptography/primitives/#password-based-kdf) **verifier** (salt + derived hash), and the guest proves possession at drop-session creation by submitting the Argon2id-derived proof. The passphrase itself is never transmitted, and the KDF cost rate-limits guessing on top of the per-IP/per-link limiters below. It limits *who may spend the owner's quota*; it adds no confidentiality, because the guest already encrypts every asset. - **Home-server-only.** Like a [share link](/design/share-links/#security-contract), an upload link is served **only by the album owner's [home server](/design/federation/#album-ownership-v1-single-home-server)**; a federated peer never accepts a drop and returns a structured `{ home_server }` pointer the client resolves. This keeps revocation, rate-limiting, and quota at one authoritative point. - **Adopted bytes are external-origin.** No device authored a drop's plaintext, so an adopted asset is **never** "local-origin": every client (including the adopter, on preview) decodes its bytes only in the [sandboxed decoder](/design/clients/#sandboxed-decoder). A hostile or malformed guest file can at worst crash a sandbox; it cannot reach the host or be silently admitted. @@ -79,7 +79,7 @@ A `DropDescriptor` is deliberately **not** an [`AssetManifest`](/design/cryptogr ### 3. Stage (server) -The server validates the drop session against the link record and the no-key drop invariants ([Threat Model — On `POST /drop`](/design/threat-model/validation/#server-side-validation-invariants)), debits the provisioning user's quota, and stores the ciphertext as a content-addressed blob in the [blob store](/design/filesystem/server/) referenced by a **drop-inbox row** (not an album asset row), with the `DropDescriptor` attached. The drop now appears in the provisioning user's inbox and on their native clients as "awaiting your review" — a [quarantine surface](/design/threat-model/scenarios/#quarantine-surfaces), never silently applied. +The server validates the drop session against the link record and the no-key drop invariants ([Threat Model — On `POST /drop`](/design/threat-model/validation/#on-post-drop-upload-link-drop-session-and-adoption)), debits the provisioning user's quota, and stores the ciphertext as a content-addressed blob in the [blob store](/design/filesystem/server/) referenced by a **drop-inbox row** (not an album asset row), with the `DropDescriptor` attached. The drop now appears in the provisioning user's inbox and on their native clients as "awaiting your review" — a [quarantine surface](/design/threat-model/scenarios/#quarantine-surfaces), never silently applied. ### 4. Review and adopt-in-place (native client, provisioning user) @@ -88,7 +88,7 @@ The user fetches a pending drop, decapsulates `kem_ct` with the Drop Key private 1. Assign a `file_id` and author the [signed sidecar](/design/metadata/#sidecar-schema-v1), including an **unverified, self-asserted guest-origin note** (`received via link {opaque-id} on {date}`, optional `suggested_filename`). This note is descriptive provenance only; the guest is **never** a signer. 2. **Rewrap `K` under the destination album's AMK** with the [`asset-keywrap/v1`](/design/cryptography/encryption/#asset-key-derivation) derivation, producing `wrapped_file_key`. Because `K` was chosen by an external party it cannot be re-derived from the AMK — it is *carried* wrapped instead. 3. Build an `AssetManifest` with `action = create`, `ciphertext_hash = drop.ciphertext_hash`, `nonce_prefix = drop.nonce_prefix`, the freshly authored `metadata_blob_hash`, `key_mode = wrapped`, and `wrapped_file_key`; set `created_by_user`/`created_by_device` to the **adopter** (the cryptographic author); sign `device_sig` + `write_sig` and append the `create` provenance record. See [Provenance — Asset Manifest](/design/cryptography/provenance/#asset-manifest). -4. Submit the `create` write. Its `ciphertext_hash` references the **already-stored drop blob**, so only the small metadata blob is uploaded. The server validates the manifest envelope (invariants 1–8, 16–18, 25) **and** that the referenced blob is a drop in the caller's own inbox, then **atomically promotes** the blob from inbox to album asset — writing the asset row, the provenance record, and the refcount — and deletes the inbox row, in one transaction. The bulk bytes never move; the quota is unchanged (same user). +4. Submit the `create` write. Its `ciphertext_hash` references the **already-stored drop blob**, so only the small metadata blob is uploaded. The server validates the manifest envelope (invariants 1–8, 16–18, 25 — invariant 14, the ciphertext-hash recomputation, already ran at the drop's own finalization, which is why adoption need not re-hash the referenced blob) **and** that the referenced blob is a drop in the caller's own inbox, then **atomically promotes** the blob from inbox to album asset — writing the asset row, the provenance record, and the refcount — and deletes the inbox row, in one transaction. The bulk bytes never move and their charge is unchanged (same user); only the small metadata and provenance blobs are new quota, as for any metadata write. From this point the asset is an ordinary library asset: it syncs, it `verify_asset`-accepts on every other album member's device, and any later edit (`replace`, `metadata-update`) follows the standard **derived-key** path. `key_mode = wrapped` is set only by this adopting `create`. From 32132a70297ae07b6e3bb5751ee89c9ec6234d9c Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:24:57 -0400 Subject: [PATCH 05/58] docs(design): complete governance tables and registries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The threat model's governing tables lagged the docs they govern: - validation: idempotency rows for the five stateful write surfaces the docs introduced without one (device enrollment, MLS re-keying, share/upload-link create + revoke, drop adoption) — the table's own rule says an unlisted surface must be designed before it ships; the universal-headers table becomes the census of the whole X-Capsule-* namespace (upload headers registered by pointer); every rejection now names its error.* catalog code, wiring the orphaned i18n error-code contract into the refuse-by-default surface - threat-model index: cross-reference rows for Share Links, Quota, Moderation, Device Enrollment, and MLS Resilience, which all carry scenario/invariant ties the navigation table omitted - scenarios: rows 40-42 (upload-link passphrase brute-force, capability min_protocol_version doctoring, share-link passphrase offline brute-force) - authentication: the .well-known/capsule/* namespace gets a single registry (server-info, moved certificate, revoked-jti, deprecation) instead of four docs declaring paths piecemeal - i18n: the error-code contract explicitly covers the validation invariants, with the flagship codes named - organization: smart-album definitions and scope overrides bound to the library-settings document metadata already declares (schema is its own slice); StackType's normative value set pinned to the capsule-core::domain enum --- .../src/content/docs/design/authentication.md | 11 +++++++++++ capsule-docs/src/content/docs/design/i18n.md | 10 ++++++++++ capsule-docs/src/content/docs/design/organization.md | 4 ++-- .../src/content/docs/design/threat-model/index.md | 5 +++++ .../src/content/docs/design/threat-model/scenarios.md | 3 +++ .../content/docs/design/threat-model/validation.md | 9 ++++++++- 6 files changed, 39 insertions(+), 3 deletions(-) diff --git a/capsule-docs/src/content/docs/design/authentication.md b/capsule-docs/src/content/docs/design/authentication.md index e76c2ee1..ac9ae11c 100644 --- a/capsule-docs/src/content/docs/design/authentication.md +++ b/capsule-docs/src/content/docs/design/authentication.md @@ -29,6 +29,17 @@ Patterns borrowed from Matrix 2.0, with one critical departure: **`.well-known/` - **Federated peer lookup** (resolving a user across servers): authenticated by a federation capability token (see [Federation — Federation Capabilities](/design/federation/#federation-capabilities)) and rate-limited per peer. - **Anonymous WebFinger**: returns only records the target user has explicitly opted into making public. The default is opt-out: no anonymous record. The opt-in-able record set is deliberately tiny — handle and display name only, never keys, device lists, or album hints; anything richer requires authenticated lookup. This is deliberately stricter than Matrix's default and follows the [deny-by-default rule](/design/threat-model/schema-rules/#schema-evolution-and-field-grammar) from the threat model. +### The `.well-known/capsule/*` Registry + +Every well-known path Capsule serves, in one census. Each path's record format is owned by the linked doc; a new path MUST add a row here when introduced. + +| Path | Contents | Owner | +| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| `.well-known/capsule/server-info` | Public server-scoped facts: API base URL, auth + federation endpoints, server signing key, supported `protocol_version` range, deprecation cutoffs. Never a user list. | this doc ([Identity and Discovery](#identity-and-discovery)) | +| `.well-known/capsule/moved/{user}` | The IK-signed moved certificate for a migrated account. | this doc ([Account Portability](#account-portability)) | +| `.well-known/capsule/revoked-jti` | Federation capability revocation list (bounded to ≤ 24 h of revocations). | [Federation](/design/federation/#token-lifecycle-and-chain-of-trust) | +| `.well-known/capsule/deprecation` | Min-supported-client deprecation announcements. | [Threat Model — Schema Rules](/design/threat-model/schema-rules/#min-supported-client-deprecation-policy) | + ## Account Portability A user must be able to move servers without losing their identity. Capsule does **not** need a separate DID system: the user identity key (User IK — see [Cryptography — Keys](/design/cryptography/keys/#user-identity-keys-user-iks)) is *already* a server-independent root of trust. Only the `user@server.tld` handle is host-bound. diff --git a/capsule-docs/src/content/docs/design/i18n.md b/capsule-docs/src/content/docs/design/i18n.md index 6276e962..f15e6602 100644 --- a/capsule-docs/src/content/docs/design/i18n.md +++ b/capsule-docs/src/content/docs/design/i18n.md @@ -116,6 +116,16 @@ The server does **not** translate by `Accept-Language`; localization happens client-side off the code, which keeps it working offline and avoids coupling the client's language to the server. +This contract covers the refuse-by-default surface too: every structured +rejection in [Threat Model — Validation](/design/threat-model/validation/) +carries an `error.*` code alongside its transport status (HTTP or gRPC — the +cross-transport carriage is owned by +[API Surfaces](/design/api-surfaces/#rejection-mapping), and the code, not the +status, is what clients switch on). Flagship codes referenced across the design +docs: `error.protocol.version_unsupported` (the `426` class), +`error.quota.exceeded`, `error.moderation.account_suspended`, +`error.auth.invalid_credentials`. + ## Contributing translations Translators edit the JSON catalogs in `locales/` and open a pull request — no code diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index 9b796377..262719e5 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -39,7 +39,7 @@ A container album must be explicitly created, but a brand-new account has none View albums are organizational surfaces computed entirely client-side over the assets the user can already decrypt (the union of their container-album memberships), materialized by querying the [local index](/design/filesystem/client/#local-index-staleness). A view is **not** an MLS group, holds **no** AMK, **owns no assets**, and is **not** a sharing or access-control boundary — sharing happens only at the container tier. Two kinds: - **System albums** — built-in and implicit. The canonical one is **All** — every asset the user can see; because that is the union over their containers, every asset appears in it (which is exactly why the [default album](#the-default-album) matters: an import always enters *some* container and so shows up in All). [Trash](#recycling) is another system view, over lifecycle state. -- **Smart / dynamic albums** — user-defined filtered views whose membership is a predicate over sidecar fields and AI-derived attributes ([Metadata](/design/metadata/#sidecar-schema-v1), [AI](/design/ai/)). Membership is **computed**, never stored: editing a smart album, or an asset's attributes, never moves or re-encrypts an asset. A definition (predicate + display name) is user content — stored in a client-side, E2E-encrypted document synced across the user's devices with the same [CRDT semantics](/design/metadata/#collaborative-metadata) as other collaborative metadata, so the server never learns it. +- **Smart / dynamic albums** — user-defined filtered views whose membership is a predicate over sidecar fields and AI-derived attributes ([Metadata](/design/metadata/#sidecar-schema-v1), [AI](/design/ai/)). Membership is **computed**, never stored: editing a smart album, or an asset's attributes, never moves or re-encrypts an asset. A definition (predicate + display name) is user content — stored in the per-owner, E2E-encrypted **library-settings document** declared in [Metadata — How Operations Travel](/design/metadata/#how-operations-travel), synced across the user's devices with the same [CRDT semantics](/design/metadata/#collaborative-metadata) as other collaborative metadata, so the server never learns it. (That document's concrete schema is a design follow-up, tracked as its own slice in the repo-root `SLICES.md`; this doc owns only what it must carry — smart-album definitions and the scope-override map.) ## Asset Stacking @@ -60,7 +60,7 @@ StackMembership { } ``` -`stack_type` is a closed enum per `protocol_version` — adding a new stack type bumps the version. Old albums never see the new type. +`stack_type` is a closed enum per `protocol_version` — adding a new stack type bumps the version. Old albums never see the new type. The authoritative value set is the closed Rust enum `capsule-core::domain::stack_type`, one variant per type below; the taxonomy prose is descriptive, the enum is normative. ### Stack Types diff --git a/capsule-docs/src/content/docs/design/threat-model/index.md b/capsule-docs/src/content/docs/design/threat-model/index.md index 8d843039..b57d8307 100644 --- a/capsule-docs/src/content/docs/design/threat-model/index.md +++ b/capsule-docs/src/content/docs/design/threat-model/index.md @@ -72,3 +72,8 @@ Each owner doc gains a short section linking back to the relevant threat-model i | [Organization](/design/organization/) | [Atomicity](/design/threat-model/validation/#atomicity-invariants), [Forbidden Behaviors](/design/threat-model/schema-rules/#forbidden-client-behaviors) | | [Clients](/design/clients/) | [Client Validation](/design/threat-model/validation/#client-side-validation-invariants), [Deprecation Policy](/design/threat-model/schema-rules/#min-supported-client-deprecation-policy) | | [Web Upload](/design/web-upload/) | [Server Validation](/design/threat-model/validation/#server-side-validation-invariants), [Scenario Map](/design/threat-model/scenarios/#damage-scenario--invariant-map), [Quarantine Surfaces](/design/threat-model/scenarios/#quarantine-surfaces) | +| [Share Links](/design/share-links/) | [Scenario Map](/design/threat-model/scenarios/#damage-scenario--invariant-map) (enumeration + passphrase rows) | +| [Quota](/design/quota/) | [Scenario Map](/design/threat-model/scenarios/#damage-scenario--invariant-map) (federated-receive row), [Server Validation](/design/threat-model/validation/#server-side-validation-invariants) (session-creation checks) | +| [Moderation](/design/moderation/) | [Server Validation](/design/threat-model/validation/#server-side-validation-invariants) (federated-report invariant), [Scenario Map](/design/threat-model/scenarios/#damage-scenario--invariant-map) (mass-report row) | +| [Device Enrollment](/design/device-enrollment/) | [Idempotency](/design/threat-model/validation/#idempotency-invariants) (enrollment-code row), [Forbidden Behaviors](/design/threat-model/schema-rules/#forbidden-client-behaviors) | +| [MLS Resilience](/design/mls-resilience/) | [Idempotency](/design/threat-model/validation/#idempotency-invariants) (re-keying row), [Quarantine Surfaces](/design/threat-model/scenarios/#quarantine-surfaces) | diff --git a/capsule-docs/src/content/docs/design/threat-model/scenarios.md b/capsule-docs/src/content/docs/design/threat-model/scenarios.md index 8e3bb7f1..0ae41750 100644 --- a/capsule-docs/src/content/docs/design/threat-model/scenarios.md +++ b/capsule-docs/src/content/docs/design/threat-model/scenarios.md @@ -48,6 +48,9 @@ The lookup table for "what damage X is prevented by which invariant Y in which d | 37 | A guest uploads a hostile or malformed file through an upload link | Drops never enter the library; the adopter decodes external-origin bytes in the sandboxed decoder and reviews before adoption; a rejected drop is discarded | [Web Upload](/design/web-upload/#drop-and-adoption-lifecycle), [Clients — Sandboxed Decoder](/design/clients/#sandboxed-decoder) | | 38 | A server substitutes its own encapsulation key to read drop contents | The Drop Key public half is delivered in the URL fragment and never reaches the server, so it cannot splice in a key it controls; the server holds only a `kem_ct` it cannot open | [Web Upload — Two Confidentiality Properties](/design/web-upload/#two-confidentiality-properties) | | 39 | A guest drop is silently injected into the library as if user-authored | A drop carries no manifest, no signatures, and no provenance, and never flows through `verify_asset`; adoption produces a fresh adopter-signed `create` manifest and records the guest as unverified descriptive origin only | [Web Upload — Drop and Adoption Lifecycle](/design/web-upload/#drop-and-adoption-lifecycle) | +| 40 | A guest brute-forces an upload link's optional passphrase to spend the owner's quota | The stored Argon2id verifier makes every guess pay the KDF cost; per-IP + per-`{opaque-id}` rate limits throttle attempts; per-link caps and quota-at-creation bound the damage even on success | [Web Upload — Security Contract](/design/web-upload/#security-contract) | +| 41 | A peer doctors a capability's `min_protocol_version` to coax a downgraded parse of album events | The claim is inside the Ed25519-signed token — tampering breaks the signature — and every pulled event still passes the receiver's own protocol handshake and the album's pin regardless of what the token claims | [Federation — Federation Capabilities](/design/federation/#federation-capabilities) | +| 42 | A share link's passphrase is brute-forced offline from the fetched wrapped secret | Argon2id cost per guess on the client-side unwrap; the wrapped-secret endpoint is rate-limited per IP + per link; the fragment-delivered link secret itself is never server-held, so the server cannot shortcut the search | [Share Links — Security Contract](/design/share-links/#security-contract) | When a scenario surfaces during implementation that does not match any of the above, the rule is: add a row here, then declare the defense in exactly one owner doc. Never restate a defense in multiple docs. diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index f3d14be4..b9464752 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -75,7 +75,7 @@ A [web-upload](/design/web-upload/) drop carries **no `AssetManifest`** — no s Drop **chunks** reuse the `PATCH` chunk rules (9–12) and **finalization** reuses the integrity checks (13–14) unchanged; only drop-session creation (26–31) and adoption (32) differ from the album upload path. -Every rejection is logged with a structured reason code; the rejected hash is remembered (bounded, see [Federation — Soft-Fail Semantics](/design/federation/#soft-fail-semantics)) so divergence between Capsule's view and a permissive peer's view is detectable. +Every rejection carries a machine-readable [`error.*` code](/design/i18n/#server-error-codes) alongside its transport status — the code, never the bare status, is what clients switch on and localize (see [API Surfaces — Rejection Mapping](/design/api-surfaces/#rejection-mapping)) — and is logged with it; the rejected hash is remembered (bounded, see [Federation — Soft-Fail Semantics](/design/federation/#soft-fail-semantics)) so divergence between Capsule's view and a permissive peer's view is detectable. ## Client-Side Validation Invariants @@ -110,6 +110,8 @@ The rules are stated once, in REST terms (headers + HTTP statuses). On the gRPC | `X-Capsule-Protocol-Max` | server on every response | the highest protocol version this server accepts | | `X-Capsule-Min-Client-Build` | server on responses | semver deprecation cutoff; advisory unless the path is hard-deprecated | +This table is also the **census of the `X-Capsule-*` header namespace**. Surface-specific headers register here by pointer: the upload protocol's `X-Capsule-Offset`, `X-Capsule-Content-Length`, `X-Capsule-Checksum`, and `X-Capsule-Suggested-Chunk-Size` (semantics owned by [Import — Upload Protocol](/design/import/upload-protocol/#endpoints)). A new `X-Capsule-*` header MUST be registered here when introduced — two homes for the namespace is how headers drift. + ### Fail-Closed Rules - `X-Capsule-Protocol` outside `[Min, Max]` on a **write**: `426 Upgrade Required`. No session created, no row written. @@ -134,6 +136,11 @@ Every write surface has a single idempotency key. Duplicates are no-ops; conflic | Federation pull | `(peer_id, sync_cursor)` — the sync cursor itself is the key | Re-pull returns the same page | | MLS commit | Handled by OpenMLS; commits are ordered by the group's commit chain | OpenMLS rejects duplicates | | Album upgrade ceremony | `intent_id` (UUIDv7); see [Versioning](/design/versioning/#album-upgrade-ceremony) | Same intent never produces two forks | +| MLS group re-keying ceremony | `intent_id` (UUIDv7); same machinery as the album upgrade ([MLS Resilience](/design/mls-resilience/#group-re-keying-ceremony)) | Same intent never re-keys twice | +| Device enrollment (code redeem / cross-device add) | The [enrollment code](/design/device-enrollment/#cross-device-add) — single-use, deleted on redemption or expiry | Re-redemption is rejected (the code is consumed); a restarted ceremony mints a fresh code | +| Share-link / upload-link creation | Client-supplied operation id (UUIDv7) | Retried create returns the already-minted link | +| Share-link / upload-link revoke | `link_id` | Second revoke is a no-op | +| Drop adoption (`POST /drops/{id}/adopt`) | `drop_id` — the atomic inbox→album promotion (invariant 32) | A retry after success finds the inbox row gone and returns the already-promoted asset | A write surface that does not appear here is, by default, **not** idempotent and must be designed before it ships. From 56addec05762f0328d5c5663446f7c0fd9528a20 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:27:55 -0400 Subject: [PATCH 06/58] docs(design): normalize terminology and prune SSoT restatements Mechanical consistency pass over the whole design set: - one term for the session store (Valkey, matching its owner doc) - ciphertext_sha256 -> the suite-governed ciphertext_hash (the stray name also hard-coded an algorithm against the crypto-agility rule) - versioning's duplicated negotiation-header table replaced by a pointer to the validation registry (one census, per the SSoT rule) - protocol_version consistently framed as date-based (no more 'bumps' / ordinal v_k phrasing, no LaTeX math in peering) - capability-token vs link-capability terminology note in federation - restatements pruned to owner-doc references: web-upload's Drop Key composition, STREAM constants, and opaque-id rule; device-enrollment's master-key and IK compositions - authorization's suite-pin claim aligned with what invariants 2 and 6 actually check; schema-rules' enum census extended to the newly declared value-set owners; failure-modes' sponsoree path count made honest and its AlbumKeyDistribution link re-homed to MLS - RFC 3339 spelling normalized in prose --- capsule-docs/src/content/docs/design/authorization.md | 4 ++-- .../content/docs/design/cryptography/failure-modes.md | 6 +++--- .../src/content/docs/design/cryptography/keys.md | 2 +- .../src/content/docs/design/device-enrollment.md | 4 ++-- capsule-docs/src/content/docs/design/federation.md | 2 +- capsule-docs/src/content/docs/design/metadata.md | 2 +- capsule-docs/src/content/docs/design/module-map.md | 2 +- capsule-docs/src/content/docs/design/organization.md | 2 +- capsule-docs/src/content/docs/design/peering.md | 2 +- .../content/docs/design/threat-model/schema-rules.md | 4 ++-- capsule-docs/src/content/docs/design/versioning.md | 11 +---------- capsule-docs/src/content/docs/design/web-upload.md | 8 ++++---- 12 files changed, 20 insertions(+), 29 deletions(-) diff --git a/capsule-docs/src/content/docs/design/authorization.md b/capsule-docs/src/content/docs/design/authorization.md index f014d7df..dc5259ae 100644 --- a/capsule-docs/src/content/docs/design/authorization.md +++ b/capsule-docs/src/content/docs/design/authorization.md @@ -21,7 +21,7 @@ Every lifecycle operation's `action` field is one of the following **closed enum | `derivative-replace` | Replace an existing derivative — the only authorized path; a silent overwrite is rejected. | | `trash-restore` | Recover a soft-deleted asset from trash within its retention window. | -Adding a value to this enum bumps `protocol_version` and old albums remain pinned to their original set — a faulty or new client cannot inject an unknown action into a v_k album. +Adding a value to this enum requires a new (later-dated) `protocol_version`, and old albums remain pinned to their original set — a faulty or new client cannot inject an unknown action into an older-pinned album. ## Authorizing a Lifecycle Operation @@ -38,7 +38,7 @@ A `delete` or `replace` is therefore authorized by the same proof as the origina Per the principle of [trusting the server for storage, never for authorization](/design/cryptography/), the server **carries out** a remote delete or replace but is **never** the authority that permits it. A server-asserted lifecycle change with no valid write-tier signature is rejected by every client. This bounds the damage a compromised or buggy server can do: it can refuse to store data, but it cannot forge its destruction. -That said, the server is not *passive*. Even without keys, it enforces the structural envelope of every manifest before persisting it — `action` is in the closed enum, `prior_provenance_hash` matches the stored chain head, `created_by_device` is in the user's published device directory, the device's hybrid signature is structurally well-formed (correct curve, correct key lengths), `crypto_suite_id` and `protocol_version` match the album's pin, and the `timestamp` passes the [sanity bound](/design/threat-model/schema-rules/#timestamp-grammar). The full checklist is owned by [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). A rejection here means no row is written and no provenance record is appended; the rejection itself is logged. +That said, the server is not *passive*. Even without keys, it enforces the structural envelope of every manifest before persisting it — `action` is in the closed enum, `prior_provenance_hash` matches the stored chain head, `created_by_device` is in the user's published device directory, the device's hybrid signature is structurally well-formed (correct curve, correct key lengths), `crypto_suite_id` is in the [inventory](/design/cryptography/primitives/#primitives-inventory) and `protocol_version` matches the album's pin (invariants 2 and 6), and the `timestamp` passes the [sanity bound](/design/threat-model/schema-rules/#timestamp-grammar). The full checklist is owned by [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). A rejection here means no row is written and no provenance record is appended; the rejection itself is logged. ## Deletes Are Soft First diff --git a/capsule-docs/src/content/docs/design/cryptography/failure-modes.md b/capsule-docs/src/content/docs/design/cryptography/failure-modes.md index 0b11192a..a7b88b04 100644 --- a/capsule-docs/src/content/docs/design/cryptography/failure-modes.md +++ b/capsule-docs/src/content/docs/design/cryptography/failure-modes.md @@ -21,9 +21,9 @@ It is a cross-cutting doc by nature: the failure-mode logic lives in many module | **Server compromise** | Server is never trusted for authorization or plaintext | Authorization is verified against MLS history; data is E2E-encrypted at rest | | **Classical primitive broken** (Ed25519, X25519) | Hybrid construction | The PQ half (ML-DSA-65 / ML-KEM-768) still holds — confidentiality and authentication survive | | **PQ primitive broken** (ML-DSA, ML-KEM) | Hybrid construction | The classical half still holds | -| **Ciphertext corruption; chunk truncation, reorder, or deletion** | AES-256-GCM-STREAM per-chunk tags + `ciphertext_sha256` | Re-fetch the blob from a content-addressed copy (path 6) | +| **Ciphertext corruption; chunk truncation, reorder, or deletion** | AES-256-GCM-STREAM per-chunk tags + the manifest's `ciphertext_hash` (algorithm fixed by `crypto_suite_id`) | Re-fetch the blob from a content-addressed copy (path 6) | | **Reader-signed / removed-writer / wrong-epoch / forged-chain / replayed manifest** | The single [`verify_asset`](/design/cryptography/keys/#write-authorization) chokepoint | Asset is quarantined and surfaced in the [audit trail](/design/cryptography/provenance/#provenance-of-library-modifications) | -| **AMK distribution lag** (manifest cites an in-flight epoch whose key has not yet arrived) | `verify_asset` *pending* outcome — `amk_version` is within the [MLS-attested range](/design/cryptography/keys/#write-authorization) but the [AlbumKeyDistribution](/design/cryptography/encryption/#asset-key-derivation) message is still in transit | Asset is **held and retried** as MLS state catches up; escalated to quarantine only if the key never arrives within the timeout — never misread as a forgery | +| **AMK distribution lag** (manifest cites an in-flight epoch whose key has not yet arrived) | `verify_asset` *pending* outcome — `amk_version` is within the [MLS-attested range](/design/cryptography/keys/#write-authorization) but the [AlbumKeyDistribution](/design/cryptography/mls/#history-delivery-for-new-joiners) message is still in transit | Asset is **held and retried** as MLS state catches up; escalated to quarantine only if the key never arrives within the timeout — never misread as a forgery | | **MLS ratchet corruption or loss** | — | The recovery path is independent of ratchet state (paths 1, 3, 4). State-divergence repair owned by [MLS Resilience](/design/mls-resilience/) | | **Backup incompleteness** (a referenced `amk_version` missing from the escrow) | Backup verification's AMK-completeness check | Caught before the backup is relied on; re-export | | **Nonce reuse** | Structurally prevented | STREAM derives per-chunk nonces; metadata blobs draw fresh random nonces; a fresh per-file key lets the STREAM counter start at zero | @@ -47,7 +47,7 @@ Restoring a complete asset collection does not depend on any single mechanism. T 6. **Content-addressed durability redundancy.** Ciphertext is addressed by the SHA-256 of its bytes, so any byte-identical copy — on another device or a [federated](/design/federation/) peer — is independently verifiable. This is a *durability* path: it restores ciphertext, not keys. *Survives: single-server data loss.* 7. **Trash soft-delete window.** Deletes are soft first — `soft_delete()` / `purge_expired_trash()` (`capsule-core::library::trash`) give a reversal window before a hard purge. *Survives: erroneous deletes by a bug or user.* -**Account-type coverage.** Registered accounts have all seven paths. [Delegated/sponsored accounts](/design/authentication/#account-types) are recovered via the sponsoring account's master key, since their keys derive from it. Non-registered ([share-link](/design/share-links/)) accounts hold no collection of their own — recovery is not applicable. +**Account-type coverage.** Registered accounts have all seven paths. [Delegated/sponsored accounts](/design/authentication/#account-types) are recovered via the sponsoring account's master key, since their keys derive from it — effectively **one** path class, routed entirely through the sponsor; the [sponsoree recovery matrix](/design/cryptography/keys/#delegatedsponsored-accounts) states the consequence (a sponsor who irrecoverably loses their master key loses every sponsoree's data with it). Non-registered ([share-link](/design/share-links/)) accounts hold no collection of their own — recovery is not applicable. ## Bug-Resistance Invariants diff --git a/capsule-docs/src/content/docs/design/cryptography/keys.md b/capsule-docs/src/content/docs/design/cryptography/keys.md index 6429deeb..744b0bc4 100644 --- a/capsule-docs/src/content/docs/design/cryptography/keys.md +++ b/capsule-docs/src/content/docs/design/cryptography/keys.md @@ -59,7 +59,7 @@ Each device's keys are cross-signed into the [device directory](#device-director Both are signed by the IK (hybrid signature). The **classical half** of each device key is **generated inside and never leaves hardware** — Secure Enclave (iOS), StrongBox/Keystore (Android), TPM (desktop) — and is non-exportable; the **PQ half** (ML-DSA-65 / ML-KEM-768) is software-sealed under hardware-protected material, because no shipping secure element implements the PQ algorithms. Shipping elements also expose **P-256**, not Ed25519/X25519, so the hardware-backed composition is the planned **P-256 hybrid variant**; the software composition (both halves in software) is what integrates end-to-end today. Because hardware halves cannot be backed up, devices are treated as disposable either way: a lost device is removed and a new one re-bootstrapped from the master key. -A device key can be revoked without affecting the user's identity or other devices. Revocation is done by signing a revocation statement with the IK and publishing it to a well-known location. The server then refuses to deliver new key wraps to that device, and remaining devices rotate any group keys the revoked device had access to. The revoked device's directory entry is **retained** — marked with `revoked_at` (RFC3339), never deleted — so the manifests it signed *before* revocation stay verifiable forever (provenance is append-only; see [Provenance](/design/cryptography/provenance/#what-an-attacker-with-all-current-keys-still-cannot-do)). +A device key can be revoked without affecting the user's identity or other devices. Revocation is done by signing a revocation statement with the IK and publishing it to a well-known location. The server then refuses to deliver new key wraps to that device, and remaining devices rotate any group keys the revoked device had access to. The revoked device's directory entry is **retained** — marked with `revoked_at` (RFC 3339), never deleted — so the manifests it signed *before* revocation stay verifiable forever (provenance is append-only; see [Provenance](/design/cryptography/provenance/#what-an-attacker-with-all-current-keys-still-cannot-do)). ### Owner Group Keys (OGKs) diff --git a/capsule-docs/src/content/docs/design/device-enrollment.md b/capsule-docs/src/content/docs/design/device-enrollment.md index af584528..aad20e07 100644 --- a/capsule-docs/src/content/docs/design/device-enrollment.md +++ b/capsule-docs/src/content/docs/design/device-enrollment.md @@ -16,8 +16,8 @@ Implementation lives in `capsule-core::crypto::keys` (key generation and wrappin When a user creates a brand-new Capsule account, the very first device runs the full setup ceremony: -1. **Generate the master key.** A 32-byte CSPRNG draw becomes the account master key. It is wrapped under a recovery passphrase via [Argon2id](/design/cryptography/primitives/#password-based-kdf); the wrapped blob is uploaded to the server-side [master-key escrow](/design/backup-recovery/#master-key-escrow). The plaintext recovery passphrase is shown to the user and never persisted. -2. **Generate the User IK.** A hybrid Ed25519 + ML-DSA-65 keypair (the [User Identity Keys](/design/cryptography/keys/#user-identity-keys-user-iks)). The private halves are wrapped under the master key; the public halves go into the (initial, single-member) device directory. +1. **Generate the master key.** A fresh account master key is drawn from the [OS CSPRNG](/design/cryptography/primitives/#randomness) ([Keys — Registered accounts](/design/cryptography/keys/#registered-accounts) owns its shape). It is wrapped under a recovery passphrase via [Argon2id](/design/cryptography/primitives/#password-based-kdf); the wrapped blob is uploaded to the server-side [master-key escrow](/design/backup-recovery/#master-key-escrow). The plaintext recovery passphrase is shown to the user and never persisted. +2. **Generate the User IK.** The [User Identity Key](/design/cryptography/keys/#user-identity-keys-user-iks) pair (composition owned there). The private halves are wrapped under the master key; the public halves go into the (initial, single-member) device directory. 3. **Generate this device's keys.** A DSK and a DEK per the [device-key composition](/design/cryptography/keys/#device-keys). The **classical half** of each is generated inside the hardware secure element and is non-exportable; shipping secure elements (Secure Enclave, StrongBox, TPM) expose **ECDSA/ECDH-P256**, not Ed25519/X25519, so the hardware-backed composition is the planned **P-256 hybrid variant** ([Keys](/design/cryptography/keys/)), with the PQ half software-sealed under hardware-protected material — no secure element holds PQ keys. Until the P-256 variant lands, the software composition generates both halves in software. Both keys are signed by the IK and added to the device directory. 4. **Publish the device directory.** The IK-signed directory is uploaded to the server. 5. **Create the default album.** Establish the owner's [default album](/design/organization/#the-default-album) — a new MLS group at the `album_id` derived from the master key (see [Keys — Key Chain](/design/cryptography/keys/#key-chain)), with this device as the sole admin/writer — and set the owner's `default_album_id` pointer ([Filesystem — Server](/design/filesystem/server/#ownership-partitioning-and-quota)) to it. This guarantees a writable import destination from the first moment the account exists. diff --git a/capsule-docs/src/content/docs/design/federation.md b/capsule-docs/src/content/docs/design/federation.md index 679e0d4e..b4d09e43 100644 --- a/capsule-docs/src/content/docs/design/federation.md +++ b/capsule-docs/src/content/docs/design/federation.md @@ -45,7 +45,7 @@ Cross-server replication of a *single* album (where two users on different home ## Federation Capabilities -Sharing an album with `alice@other.tld` requires her server to be *able* to fetch that album's blobs. Capsule issues her server an **album-scoped capability token**: a signed, expiring, revocable grant naming the album, the scope, and an expiry, reusing the [EdDSA-JWT machinery](/design/authentication/#access-token) already built for access tokens — no separate macaroon or ZCAP format is introduced. +Sharing an album with `alice@other.tld` requires her server to be *able* to fetch that album's blobs. Capsule issues her server an **album-scoped capability token**: a signed, expiring, revocable grant naming the album, the scope, and an expiry, reusing the [EdDSA-JWT machinery](/design/authentication/#access-token) already built for access tokens — no separate macaroon or ZCAP format is introduced. (Terminology note: this server-to-server JWT and the user-facing **link capabilities** — [share links](/design/share-links/) and [upload links](/design/web-upload/), opaque-id + fragment secret — share the *concept* of unforgeable possession granting scoped access, not a format; "capability token" always means this JWT.) The capability token format is the contract every federated peer parses and that this server signs. Its shape and lifecycle below are normative. diff --git a/capsule-docs/src/content/docs/design/metadata.md b/capsule-docs/src/content/docs/design/metadata.md index 8d6aee38..f429cdce 100644 --- a/capsule-docs/src/content/docs/design/metadata.md +++ b/capsule-docs/src/content/docs/design/metadata.md @@ -65,7 +65,7 @@ SidecarV1 { ### Closed Enum Value Sets -Two sidecar fields are closed enums whose authoritative value sets live here (the blanket closed-enum rule is [Threat Model — Schema Rules](/design/threat-model/schema-rules/); the code mirror is a closed Rust enum in `capsule-core::domain`, and adding a value bumps `protocol_version`): +Two sidecar fields are closed enums whose authoritative value sets live here (the blanket closed-enum rule is [Threat Model — Schema Rules](/design/threat-model/schema-rules/); the code mirror is a closed Rust enum in `capsule-core::domain`, and adding a value requires a new, later-dated `protocol_version`): - **`content_type`** — MIME syntax, exactly **one canonical value per format** (never an alias like `image/jpg`). The v1 set: - images: `image/jpeg`, `image/png`, `image/webp`, `image/gif`, `image/tiff`, `image/heic`, `image/avif`, `image/jxl`, `image/x-adobe-dng` diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 1b32cca2..77696301 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -78,7 +78,7 @@ Hardware-key adapters do **not** live in `capsule-sdk`: the `Signer`/`HardwareSi | Module | Owning design doc | Validation tier | | ---------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------- | | `capsule-api` (routing) | [Filesystem — Server](/design/filesystem/server/) | Smoke | -| `capsule-api-auth::{oidc,session,claims,roles}` | [Authentication](/design/authentication/), [Authorization](/design/authorization/) | Unit + Smoke (testcontainer Postgres/Redis) | +| `capsule-api-auth::{oidc,session,claims,roles}` | [Authentication](/design/authentication/), [Authorization](/design/authorization/) | Unit + Smoke (testcontainer Postgres/Valkey) | | `capsule-api-auth::devices` (planned for enrollment) | [Device Enrollment](/design/device-enrollment/) | Smoke | | `capsule-api-library::schema::*` (legacy, retiring) | [API Surfaces](/design/api-surfaces/#legacy-graphql-retiring) | frozen (no new surface) | | `capsule-api-library::loaders` (legacy, retiring) | [API Surfaces](/design/api-surfaces/#legacy-graphql-retiring) | frozen (no new surface) | diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index 262719e5..a8ce9df4 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -60,7 +60,7 @@ StackMembership { } ``` -`stack_type` is a closed enum per `protocol_version` — adding a new stack type bumps the version. Old albums never see the new type. The authoritative value set is the closed Rust enum `capsule-core::domain::stack_type`, one variant per type below; the taxonomy prose is descriptive, the enum is normative. +`stack_type` is a closed enum per `protocol_version` — adding a new stack type requires a new (later-dated) version. Old albums never see the new type. The authoritative value set is the closed Rust enum `capsule-core::domain::stack_type`, one variant per type below; the taxonomy prose is descriptive, the enum is normative. ### Stack Types diff --git a/capsule-docs/src/content/docs/design/peering.md b/capsule-docs/src/content/docs/design/peering.md index 92e6f1fd..1d5161f4 100644 --- a/capsule-docs/src/content/docs/design/peering.md +++ b/capsule-docs/src/content/docs/design/peering.md @@ -27,7 +27,7 @@ Identity-trusted is **not** content-trusted, however. A device can still be bugg ### Peer-Class Containment -Even two of the same user's devices are separate failure-containment boundaries ([Threat Model — Damage Containment Layers](/design/threat-model/#damage-containment-layers)). A buggy $v_k$ device cannot overwrite a $v_{k+1}$ device's state via a stale-but-valid backup artifact, and a v_{k+1} device's writes are not retroactively applied to a v_k device's view of an older album. Specifically: +Even two of the same user's devices are separate failure-containment boundaries ([Threat Model — Damage Containment Layers](/design/threat-model/#damage-containment-layers)). A buggy older-version device cannot overwrite a newer device's state via a stale-but-valid backup artifact, and a newer device's writes are not retroactively applied to an older device's view of an older-pinned album. Specifically: - Every received manifest is checked against the receiver's local `latest_provenance_hash` for that asset (see [Applying Received Data](#applying-received-data)) — a stale manifest is quarantined, not silently applied. - Every received structure that announces a `sidecar_schema`, `crypto_suite_id`, or `protocol_version` above the receiver's max known is rejected at decode — the receiver refuses to interpret bytes it cannot validate. This is the client-side counterpart of the [server-side schema lockdown](/design/threat-model/schema-rules/#schema-evolution-and-field-grammar). diff --git a/capsule-docs/src/content/docs/design/threat-model/schema-rules.md b/capsule-docs/src/content/docs/design/threat-model/schema-rules.md index 53d31c56..f0c52cbe 100644 --- a/capsule-docs/src/content/docs/design/threat-model/schema-rules.md +++ b/capsule-docs/src/content/docs/design/threat-model/schema-rules.md @@ -16,9 +16,9 @@ Capsule schemas evolve over time, but the rules of evolution are fixed — what ### Closed Enums -**Every enum in a signed or validated structure is closed per `protocol_version`** — a value outside the set known at that version is a structural error, never a "future value to ignore." This is a blanket rule, not a curated list, so it cannot rot: adding a value to *any* such enum bumps `protocol_version` (see [Versioning — Album Protocol Version Pinning](/design/versioning/#album-protocol-version-pinning)), and a pinned old album never sees the new value. It is enforced on **both sides** — the server's structural envelope check (invariant 16) and the client's `verify_asset`/decode path (see [Validation](/design/threat-model/validation/)). +**Every enum in a signed or validated structure is closed per `protocol_version`** — a value outside the set known at that version is a structural error, never a "future value to ignore." This is a blanket rule, not a curated list, so it cannot rot: adding a value to *any* such enum requires a new (later-dated) `protocol_version` (see [Versioning — Album Protocol Version Pinning](/design/versioning/#album-protocol-version-pinning)), and a pinned old album never sees the new value. It is enforced on **both sides** — the server's structural envelope check (invariant 16) and the client's `verify_asset`/decode path (see [Validation](/design/threat-model/validation/)). -The authoritative value set for each enum lives in its owner doc — `AssetManifest.action` in [Authorization](/design/authorization/#the-closed-action-set), `content_type` and `gps.source` in [Metadata](/design/metadata/#sidecar-schema-v1), `DerivativeManifest.role` in [Provenance](/design/cryptography/provenance/#derivative-provenance) — never duplicated here. +The authoritative value set for each enum lives in its owner doc — `AssetManifest.action` in [Authorization](/design/authorization/#the-closed-action-set), `content_type` and `gps.source` in [Metadata — Closed Enum Value Sets](/design/metadata/#closed-enum-value-sets), `key_mode` and `DerivativeManifest.role`/`format` in [Provenance](/design/cryptography/provenance/#derivative-provenance), `stack_type`/`role` in [Organization](/design/organization/#stack-membership-schema) — never duplicated here. ### Timestamp Grammar diff --git a/capsule-docs/src/content/docs/design/versioning.md b/capsule-docs/src/content/docs/design/versioning.md index 377fbeff..e15b7fbf 100644 --- a/capsule-docs/src/content/docs/design/versioning.md +++ b/capsule-docs/src/content/docs/design/versioning.md @@ -19,16 +19,7 @@ Versioning happens on multiple layers, each owned by the doc that defines it: ## Negotiation Headers -The contract for version compatibility — every API request and response carries these. The full fail-closed rule set is owned by [Threat Model — Protocol and Capability Negotiation](/design/threat-model/validation/#protocol-and-capability-negotiation). - -| Header | Sent by | Meaning | -| ---------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------- | -| `X-Capsule-Protocol` | client / peer | `YYYY-MM-DD` protocol version the request is written against | -| `X-Capsule-Crypto-Suite` | client / peer on writes | `u16` suite id from the [Primitives Inventory](/design/cryptography/primitives/#primitives-inventory) | -| `X-Capsule-Sidecar-Schema` | client on metadata-update | `u16` schema version declared at `sidecar_schema` field 0 | -| `X-Capsule-Protocol-Min` | server on every response | the lowest protocol version this server accepts | -| `X-Capsule-Protocol-Max` | server on every response | the highest protocol version this server accepts | -| `X-Capsule-Min-Client-Build` | server on responses | semver deprecation cutoff; advisory unless the path is hard-deprecated | +The negotiation-header set — `X-Capsule-Protocol`, `X-Capsule-Crypto-Suite`, `X-Capsule-Sidecar-Schema`, and the server's `X-Capsule-Protocol-Min`/`-Max` and `X-Capsule-Min-Client-Build` responses — is declared **once**, in the registry at [Threat Model — Universal Headers](/design/threat-model/validation/#universal-headers), together with the fail-closed rules; the cross-transport carriage is [API Surfaces](/design/api-surfaces/#negotiation-across-transports). This doc adds only the versioning semantics: `protocol_version` is **date-based** (`YYYY-MM-DD`, ordered lexicographically = chronologically), a request is written against exactly one version, and the server advertises the closed `[Min, Max]` window it accepts on every response. ## Compatibility Verification diff --git a/capsule-docs/src/content/docs/design/web-upload.md b/capsule-docs/src/content/docs/design/web-upload.md index 66d771a2..e22e9b31 100644 --- a/capsule-docs/src/content/docs/design/web-upload.md +++ b/capsule-docs/src/content/docs/design/web-upload.md @@ -39,7 +39,7 @@ Out of scope for v1 (deliberate non-goals): These are **normative** — the security-relevant decisions are committed; only UX presentation remains open. -- **Upload-link URL format.** `https://server.tld/u/{opaque-id}#{drop_pubkey}`. `{opaque-id}` is a **random 128-bit value** from the CSPRNG — full 128-bit entropy, *not* a UUIDv7 or other structured id (identical rule to the [share-link opaque-id](/design/share-links/#security-contract)); it is fully opaque and carries no scope. `{drop_pubkey}` is the Drop Key public half, carried in the **fragment** so the server never receives it (see [Server-blind](#two-confidentiality-properties)). +- **Upload-link URL format.** `https://server.tld/u/{opaque-id}#{drop_pubkey}`. `{opaque-id}` follows the [share-link opaque-id rule](/design/share-links/#security-contract) exactly — a random ≥128-bit CSPRNG value, never a structured id — and is fully opaque, carrying no scope. `{drop_pubkey}` is the Drop Key public half, carried in the **fragment** so the server never receives it (see [Server-blind](#two-confidentiality-properties)). - **Drops never enter the library.** A drop is written only to the provisioning user's **inbox**; it is never an album asset, never appears in any album member's [sync feed](/design/import/download-sync/#discovering-what-changed), and is served only to the provisioning user's own authenticated devices. A drop carries **no `AssetManifest`** — no `device_sig`, no `write_sig`, no `album_id`, no provenance — and therefore never flows through [`verify_asset`](/design/cryptography/keys/#write-authorization). Library state is reachable only through adoption by a trusted client. - **Per-link caps are enforced server-side at the no-key layer.** Expiry, cumulative-byte cap, file-count cap, and per-file size cap are checked on every drop-session creation; an over-cap or expired/revoked link is refused. These bound a leaked link to *wasted quota and inbox space*, never to library corruption. - **Quota is charged to the provisioning user at drop-session creation.** A drop debits the link owner's quota at session creation — the single hard [enforcement point](/design/quota/#enforcement-points) — using the [`upload_user_id = owner_id` attribution](/design/quota/#accounting-model). A link cannot be used to push the owner past their hard limit. @@ -52,14 +52,14 @@ These are **normative** — the security-relevant decisions are committed; only ### 1. Provision (native client, registered user) -The user's client mints a [Drop Key](/design/cryptography/keys/#non-registered-accounts) (a hybrid X25519 + ML-KEM-768 KEM keypair), wraps the private half under the [account master key](/design/cryptography/keys/#registered-accounts) and re-wraps it to the user's [OGK](/design/cryptography/keys/#owner-group-keys-ogks) so any of the user's enrolled devices can later decapsulate, and registers an upload-link record with the server: `{opaque-id}`, the per-link caps, the pinned `protocol_version` + `crypto_suite_id`, and an optional passphrase wrap. The server stores the record under `{opaque-id}` and never sees `{drop_pubkey}`. The user shares the full URL (including the fragment) with the guest over any out-of-band channel. +The user's client mints a [Drop Key](/design/cryptography/keys/#non-registered-accounts) (composition owned there), wraps the private half under the [account master key](/design/cryptography/keys/#registered-accounts) and re-wraps it to the user's [OGK](/design/cryptography/keys/#owner-group-keys-ogks) so any of the user's enrolled devices can later decapsulate, and registers an upload-link record with the server: `{opaque-id}`, the per-link caps, the pinned `protocol_version` + `crypto_suite_id`, and an optional passphrase wrap. The server stores the record under `{opaque-id}` and never sees `{drop_pubkey}`. The user shares the full URL (including the fragment) with the guest over any out-of-band channel. ### 2. Seal and upload (web client, guest) For each selected asset the web client: 1. Draws a random 32-byte asset key **`K`** from the browser CSPRNG. -2. Encrypts the asset with **AES-256-GCM-STREAM** under `K`, using the *unchanged* [STREAM construction](/design/cryptography/encryption/#stream-construction) (65,520-byte plaintext chunks, a fresh 7-byte `nonce_prefix`), and computes `ciphertext_hash` incrementally. +2. Encrypts the asset under `K` with the *unchanged* [STREAM construction](/design/cryptography/encryption/#stream-construction) (chunking and nonce shape owned there), and computes `ciphertext_hash` incrementally. 3. **Encapsulates `K`** to `{drop_pubkey}` with the link's KEM, producing `kem_ct`. 4. Emits an unsigned **`DropDescriptor`** and uploads it alongside the ciphertext via the drop upload protocol — the [upload protocol](/design/import/upload-protocol/)'s chunk and finalization mechanics under link-capability auth, with the drop endpoints in the [Contract Skeleton](#contract-skeleton): @@ -67,7 +67,7 @@ For each selected asset the web client: DropDescriptor { content_type: enum, // closed enum for the link's protocol_version (same set as a manifest's) plaintext_size: u64, - chunk_size: u32, // 65,520 + chunk_size: u32, // the STREAM plaintext chunk size (owned by Encryption) nonce_prefix: [u8; 7], // the STREAM nonce prefix used above ciphertext_hash: bytes, // content-address digest of the STREAM ciphertext kem_ct: bytes, // K encapsulated to {drop_pubkey}; length fixed by crypto_suite_id From 6747a818d0d7adc852efec5c2c88a31e8af078ab Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:38:09 -0400 Subject: [PATCH 07/58] feat(core): add wrapped file-key and P-256 DSK contract types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The type-level half of the web-upload wrapped-key contract and the P-256 hardware seam, binding the slices that implement them (S-A1/S-A6 and S-A4 in the upcoming SLICES.md). Additive only — no behavior change, and the canonical CBOR of every existing signed structure is byte-identical. - ManifestCore gains key_mode (closed enum, Default = Derived), wrapped_file_key (byte-string newtype, following the Hash32 serde discipline so ciborium never emits an integer array), and metadata_blob_hash — all #[serde(default, skip_serializing_if)] so their defaults are ABSENT map keys: signing_bytes() re-serializes the struct for verification, so present-null defaults would have silently invalidated every previously signed manifest with no test catching it. A regression test pins the wire-absence contract and the byte-string encoding; the wire-presence rules are normative in the provenance design doc - DerivativeCore gains the protocol_version and amk_version a verifier needs to select the epoch write-tier key (wire-optional for pre-binding fixtures, required on real writes per the doc) - the asset-keywrap/v1 HKDF label lands beside its siblings in crypto::primitives::info with domain-separation assertions - crypto::keys::p256 is the hardware-composition seam: the ClassicalAlgorithm discriminator, the P256HybridSigningKey skeleton (todo!() until slice S-A4), and an #[ignore]d contract test that names the acceptance criteria (algorithm-tagged hybrid round-trip, Ed25519-verifier rejection, non-exportability) verify_asset behavior and structural_ok are deliberately untouched; presence-by-action enforcement lands with the wrapped-key slice. mise run check-rust passes end to end; all 291 existing core tests pass unchanged plus the two new wire-contract tests. --- capsule-core/src/backup/artifact.rs | 7 +- capsule-core/src/crypto/keys/mod.rs | 2 + capsule-core/src/crypto/keys/p256.rs | 73 +++++++++ capsule-core/src/crypto/primitives.rs | 8 + .../src/crypto/provenance/manifest.rs | 140 ++++++++++++++++++ capsule-core/src/crypto/provenance/mod.rs | 2 +- capsule-core/src/crypto/provenance/record.rs | 5 +- capsule-core/src/crypto/verify_asset.rs | 5 +- capsule-core/src/lifecycle.rs | 5 +- capsule-core/src/validation/structural.rs | 5 +- 10 files changed, 246 insertions(+), 6 deletions(-) create mode 100644 capsule-core/src/crypto/keys/p256.rs diff --git a/capsule-core/src/backup/artifact.rs b/capsule-core/src/backup/artifact.rs index 3e6c6f68..3d433b52 100644 --- a/capsule-core/src/backup/artifact.rs +++ b/capsule-core/src/backup/artifact.rs @@ -588,7 +588,9 @@ mod tests { use crate::crypto::keys::{Amk, AmkVersion, HybridSigningKey}; use crate::crypto::primitives::PROTOCOL_VERSION; use crate::crypto::provenance::action::Action; - use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, ManifestCore as MCore}; + use crate::crypto::provenance::manifest::{ + ASSET_MANIFEST_VERSION, KeyMode, ManifestCore as MCore, + }; const ALBUM: u128 = 0xA1; @@ -625,6 +627,9 @@ mod tests { plaintext_size: enc.plaintext_size, chunk_size: enc.chunk_size, nonce_prefix: enc.nonce_prefix, + key_mode: KeyMode::Derived, + wrapped_file_key: None, + metadata_blob_hash: None, created_by_user: Uuid::from_u128(0x05E2), created_by_device: Uuid::from_u128(0xD1), client_version: "t".into(), diff --git a/capsule-core/src/crypto/keys/mod.rs b/capsule-core/src/crypto/keys/mod.rs index 72f7e02f..18ae8fbf 100644 --- a/capsule-core/src/crypto/keys/mod.rs +++ b/capsule-core/src/crypto/keys/mod.rs @@ -14,6 +14,7 @@ pub mod hybrid_sig; pub mod kem; pub mod keystore; pub mod master; +pub mod p256; pub mod signer; pub mod software; #[cfg(feature = "tpm")] @@ -26,6 +27,7 @@ pub use hybrid_sig::{HybridSignature, HybridSigningKey, HybridVerifyingKey}; pub use kem::DekKeypair; pub use keystore::{Account, AccountFile, DeviceKeys}; pub use master::MasterKey; +pub use p256::{ClassicalAlgorithm, P256HybridSigningKey}; pub use signer::Signer; pub use software::SoftwareSigner; #[cfg(feature = "tpm")] diff --git a/capsule-core/src/crypto/keys/p256.rs b/capsule-core/src/crypto/keys/p256.rs new file mode 100644 index 00000000..15abf014 --- /dev/null +++ b/capsule-core/src/crypto/keys/p256.rs @@ -0,0 +1,73 @@ +//! P-256 hybrid-DSK variant — **contract skeleton** (slice `S-A4` in the repo-root +//! `SLICES.md`; design: ). +//! +//! Shipping secure elements (Secure Enclave, StrongBox, TPM 2.0) expose **ECDSA-P256**, +//! not Ed25519, so the hardware-backed device-key composition pairs a *hardware P-256 +//! classical half* with the software-sealed ML-DSA-65 half. Only the software backend +//! composes end-to-end today; this module is the seam the three hardware backends plug +//! into once the variant lands. +//! +//! What the implementing slice must deliver (recorded here so the contract is binding): +//! +//! - `P256HybridSigningKey` implementing [`Signer`](super::signer::Signer), composing a +//! [`HardwareSigner`](super::hardware::HardwareSigner) P-256 half (DER-encoded ECDSA +//! signatures) with the software ML-DSA-65 half — which requires the hybrid signature +//! and verifying-key types (and the device-directory entry) to become +//! **algorithm-tagged** over [`ClassicalAlgorithm`] rather than assuming Ed25519. +//! - Verification-side dispatch in `verify_asset` on the directory entry's declared +//! classical algorithm, with the existing Ed25519 path byte-for-byte unchanged. +//! - The per-platform smoke (sign + verify + non-exportability) against a real element. + +use std::sync::Arc; + +use super::hardware::{HardwareSigner, HardwareSignerError}; + +/// The classical half of a hybrid device-key composition. Ed25519 is the software +/// default; P-256 is what shipping secure elements provide. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ClassicalAlgorithm { + /// Software (or future hardware) Ed25519 — today's only end-to-end composition. + Ed25519, + /// Hardware ECDSA-P256 (Secure Enclave / StrongBox / TPM 2.0). + EcdsaP256, +} + +/// A hybrid device signing key whose classical half is a **hardware-held P-256 key** +/// and whose ML-DSA-65 half is software-sealed. Skeleton: construction and signing are +/// unimplemented until slice `S-A4`. +pub struct P256HybridSigningKey { + #[allow(dead_code)] + hardware: Arc, + #[allow(dead_code)] + key_alias: String, +} + +impl P256HybridSigningKey { + /// Enroll a P-256 hardware key under `key_alias` and compose it with the software + /// ML-DSA-65 half derived from `ml_seed` — the P-256 analogue of + /// [`HardwareBackedSigner::enroll`](super::hardware::HardwareBackedSigner::enroll). + /// + /// # Panics + /// Unimplemented skeleton (slice `S-A4`). + pub fn enroll( + hardware: Arc, + key_alias: String, + ml_seed: &[u8; 32], + ) -> Result { + let (_, _, _) = (hardware, key_alias, ml_seed); + todo!("S-A4: P-256 hybrid enrollment — see SLICES.md") + } +} + +#[cfg(test)] +mod tests { + /// Acceptance criteria for slice `S-A4`, encoded as the contract test to un-ignore: + /// enroll a mock P-256 element, sign a fixed payload, verify both halves through the + /// algorithm-tagged hybrid verifying key, assert an Ed25519-only verifier rejects it, + /// and assert non-exportability via the existing `assert_non_exportable` contract. + #[test] + #[ignore = "S-A4 contract: P-256 hybrid sign/verify round-trip not yet implemented"] + fn p256_hybrid_round_trip_and_directory_dispatch() { + unimplemented!("implemented by slice S-A4"); + } +} diff --git a/capsule-core/src/crypto/primitives.rs b/capsule-core/src/crypto/primitives.rs index db9e0978..19c744cd 100644 --- a/capsule-core/src/crypto/primitives.rs +++ b/capsule-core/src/crypto/primitives.rs @@ -60,6 +60,11 @@ pub mod info { pub const ASSET_FILE_V1: &[u8] = b"asset-file/v1"; /// Per-metadata-blob key: `HKDF(ikm=AMK, salt=blob_id, info=METADATA_BLOB_V1)`. pub const METADATA_BLOB_V1: &[u8] = b"metadata-blob/v1"; + /// Wrap key for an externally-chosen file key (`key_mode = wrapped`; an adopted + /// web-upload drop): `HKDF(ikm=AMK, salt=file_id || wrap_nonce, info=ASSET_KEYWRAP_V1)`. + /// The sealing/unsealing functions land with the wrapped-key slice; the label is the + /// contract (see ). + pub const ASSET_KEYWRAP_V1: &[u8] = b"asset-keywrap/v1"; /// Default-album *identifier* derived from the account master key (an ID, not a key). pub const DEFAULT_ALBUM_ID_V1: &[u8] = b"default-album-id/v1"; } @@ -163,7 +168,10 @@ mod tests { // Distinct labels keep derived keys in separate domains. assert_ne!(info::ASSET_FILE_V1, info::METADATA_BLOB_V1); assert_ne!(info::ASSET_FILE_V1, info::DEFAULT_ALBUM_ID_V1); + assert_ne!(info::ASSET_KEYWRAP_V1, info::ASSET_FILE_V1); + assert_ne!(info::ASSET_KEYWRAP_V1, info::METADATA_BLOB_V1); assert!(info::ASSET_FILE_V1.ends_with(b"/v1")); assert!(info::METADATA_BLOB_V1.ends_with(b"/v1")); + assert!(info::ASSET_KEYWRAP_V1.ends_with(b"/v1")); } } diff --git a/capsule-core/src/crypto/provenance/manifest.rs b/capsule-core/src/crypto/provenance/manifest.rs index 4663a881..555ee1f7 100644 --- a/capsule-core/src/crypto/provenance/manifest.rs +++ b/capsule-core/src/crypto/provenance/manifest.rs @@ -23,6 +23,74 @@ pub const ASSET_MANIFEST_VERSION: &str = "asset-manifest/v1"; /// Current derivative-manifest schema string. pub const DERIVATIVE_MANIFEST_VERSION: &str = "derivative-manifest/v1"; +/// How a reader obtains the asset's file key (closed enum; SSoT: +/// [Cryptography — Provenance](https://docs/design/cryptography/provenance/)). +/// +/// Wire-presence rule: `Derived` is the default and encodes as an **absent** map key in +/// canonical CBOR (`skip_serializing_if`), so manifests signed before this field existed +/// re-verify byte-identically. Emitting `key_mode: "derived"` explicitly would change the +/// signed bytes and break verification. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum KeyMode { + /// The file key is recomputed from the AMK (`asset-file/v1`); nothing is stored. + #[default] + Derived, + /// The file key was chosen externally (an adopted web-upload drop) and is carried in + /// `wrapped_file_key`, sealed under the AMK (`asset-keywrap/v1`). + Wrapped, +} + +impl KeyMode { + /// Serde helper: the default (`derived`) is wire-absent. + pub fn is_derived(&self) -> bool { + matches!(self, Self::Derived) + } +} + +/// An externally-chosen file key sealed under the AMK: `wrap_nonce || AES-256-GCM(K) || tag` +/// (the `asset-keywrap/v1` derivation — see +/// [Encryption — Asset Key Derivation](https://docs/design/cryptography/encryption/)). +/// Length is fixed by `crypto_suite_id`; the bytes are ciphertext, opaque to the server. +/// +/// Serializes as a CBOR **byte string** (major type 2), like [`Hash32`] — never as an +/// array of integers — so canonical encodings are byte-identical across implementations. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WrappedFileKey(pub Vec); + +impl Serialize for WrappedFileKey { + fn serialize(&self, s: S) -> Result { + s.serialize_bytes(&self.0) + } +} + +impl<'de> Deserialize<'de> for WrappedFileKey { + fn deserialize>(d: D) -> Result { + struct V; + impl<'de> serde::de::Visitor<'de> for V { + type Value = WrappedFileKey; + fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("an AMK-wrapped file key as a byte string") + } + fn visit_bytes(self, v: &[u8]) -> Result { + Ok(WrappedFileKey(v.to_vec())) + } + fn visit_seq>( + self, + mut seq: A, + ) -> Result { + // Tolerate decoders that surface a byte string as a sequence. + let mut out = Vec::new(); + while let Some(b) = seq.next_element()? { + out.push(b); + } + Ok(WrappedFileKey(out)) + } + } + d.deserialize_bytes(V) + } +} + /// The signed core of an asset manifest — every field the two signatures cover. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ManifestCore { @@ -46,6 +114,18 @@ pub struct ManifestCore { pub chunk_size: u32, /// STREAM nonce prefix (random per file). pub nonce_prefix: [u8; 7], + /// How the file key is obtained: `derived` (default; wire-absent) or `wrapped` (an + /// adopted web-upload drop). Closed enum — see [`KeyMode`] for the wire-presence rule. + #[serde(default, skip_serializing_if = "KeyMode::is_derived")] + pub key_mode: KeyMode, + /// The AMK-sealed file key; present iff `key_mode = wrapped`, wire-absent otherwise. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub wrapped_file_key: Option, + /// Content address of the asset's encrypted metadata blob. Present on + /// `create | replace | metadata-update`; wire-absent (key omitted, never null) on + /// `delete | derivative-* | trash-restore`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub metadata_blob_hash: Option, /// User who produced the asset. pub created_by_user: Uuid, /// Device that produced the asset (resolved in the device directory). @@ -123,6 +203,15 @@ pub struct DerivativeCore { pub version: String, /// Primitive bundle. pub crypto_suite_id: u16, + /// Date-based wire protocol version; matches the album pin. Wire-absent only on + /// pre-binding fixtures; REQUIRED on every real write. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub protocol_version: Option, + /// The AMK epoch whose write-tier key produced `write_sig` — the verifier needs it to + /// select the verification key. Wire-absent only on pre-binding fixtures; REQUIRED on + /// every real write (a derivative without it cannot be authorization-verified). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub amk_version: Option, /// The asset this derivative is generated from. pub source_asset_id: Uuid, /// Which kind of derivative. @@ -196,6 +285,9 @@ mod tests { plaintext_size: 1024, chunk_size: 65_520, nonce_prefix: [1, 2, 3, 4, 5, 6, 7], + key_mode: KeyMode::Derived, + wrapped_file_key: None, + metadata_blob_hash: None, created_by_user: Uuid::from_u128(0x05E2), created_by_device: Uuid::from_u128(0xD1), client_version: "capsule-cli/0.1.0".into(), @@ -275,6 +367,8 @@ mod tests { let dm = DerivativeCore { version: DERIVATIVE_MANIFEST_VERSION.into(), crypto_suite_id: CRYPTO_SUITE_ID, + protocol_version: Some(PROTOCOL_VERSION.into()), + amk_version: Some(AmkVersion(1)), source_asset_id: Uuid::from_u128(0xF11E), role: DerivativeRole::Thumbnail, format: "image/avif".into(), @@ -301,4 +395,50 @@ mod tests { fn wt() -> HybridSigningKey { HybridSigningKey::from_seed_bytes(&[3; 32], &[4; 32]) } + + /// The wire-presence contract for the fields added within `asset-manifest/v1` + /// (`key_mode`, `wrapped_file_key`, `metadata_blob_hash`): at their defaults they are + /// ABSENT map keys, so manifests signed before the fields existed re-verify + /// byte-identically. See the wire-presence rules in + /// . + #[test] + fn default_new_fields_are_wire_absent() { + let bytes = core(Action::Create, None).signing_bytes(); + let value: ciborium::Value = cbor::from_slice(&bytes).unwrap(); + let map = value.as_map().expect("manifest core encodes as a CBOR map"); + let keys: Vec<&str> = map.iter().filter_map(|(k, _)| k.as_text()).collect(); + assert!( + !keys.contains(&"key_mode"), + "derived key_mode must be wire-absent" + ); + assert!(!keys.contains(&"wrapped_file_key")); + assert!(!keys.contains(&"metadata_blob_hash")); + // The pre-existing options keep their legacy present-null encoding. + assert!(keys.contains(&"prior_provenance_hash")); + assert!(keys.contains(&"retention_until")); + } + + #[test] + fn wrapped_fields_round_trip_as_byte_strings() { + let mut c = core(Action::Create, None); + c.key_mode = KeyMode::Wrapped; + c.wrapped_file_key = Some(WrappedFileKey(vec![0xAB; 60])); + c.metadata_blob_hash = Some(Hash32([0x4D; 32])); + let m = c.sign(&dev(), &wt()).unwrap(); + let back: AssetManifest = cbor::from_slice(&cbor::to_canonical_vec(&m).unwrap()).unwrap(); + assert_eq!(back, m); + assert_eq!(back.signing_bytes(), m.signing_bytes()); + + let value: ciborium::Value = cbor::from_slice(&m.signing_bytes()).unwrap(); + let map = value.as_map().unwrap(); + let wrapped = map + .iter() + .find(|(k, _)| k.as_text() == Some("wrapped_file_key")) + .map(|(_, v)| v) + .expect("wrapped_file_key present when key_mode = wrapped"); + assert!( + wrapped.is_bytes(), + "wrapped_file_key must encode as a CBOR byte string" + ); + } } diff --git a/capsule-core/src/crypto/provenance/mod.rs b/capsule-core/src/crypto/provenance/mod.rs index 8bbdc61c..18133f53 100644 --- a/capsule-core/src/crypto/provenance/mod.rs +++ b/capsule-core/src/crypto/provenance/mod.rs @@ -17,6 +17,6 @@ pub mod record; pub use action::{Action, DerivativeRole}; pub use manifest::{ ASSET_MANIFEST_VERSION, AssetManifest, DERIVATIVE_MANIFEST_VERSION, DerivativeCore, - DerivativeManifest, ManifestCore, + DerivativeManifest, KeyMode, ManifestCore, WrappedFileKey, }; pub use record::{ChainError, ProvenanceChain, ProvenanceRecord}; diff --git a/capsule-core/src/crypto/provenance/record.rs b/capsule-core/src/crypto/provenance/record.rs index 30021c13..cd21f647 100644 --- a/capsule-core/src/crypto/provenance/record.rs +++ b/capsule-core/src/crypto/provenance/record.rs @@ -139,7 +139,7 @@ mod tests { use crate::crypto::keys::{AmkVersion, HybridSigningKey}; use crate::crypto::primitives::{CRYPTO_SUITE_ID, PROTOCOL_VERSION}; use crate::crypto::provenance::action::Action; - use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, ManifestCore}; + use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, KeyMode, ManifestCore}; const ASSET: u128 = 0xF11E; @@ -162,6 +162,9 @@ mod tests { plaintext_size: 10, chunk_size: 65_520, nonce_prefix: [0; 7], + key_mode: KeyMode::Derived, + wrapped_file_key: None, + metadata_blob_hash: None, created_by_user: Uuid::from_u128(0x05E2), created_by_device: Uuid::from_u128(0xD1), client_version: "t".into(), diff --git a/capsule-core/src/crypto/verify_asset.rs b/capsule-core/src/crypto/verify_asset.rs index c076a3c0..6635a91c 100644 --- a/capsule-core/src/crypto/verify_asset.rs +++ b/capsule-core/src/crypto/verify_asset.rs @@ -188,7 +188,7 @@ mod tests { use crate::crypto::keys::{AmkVersion, HybridSigningKey}; use crate::crypto::primitives::{CRYPTO_SUITE_ID, PROTOCOL_VERSION}; use crate::crypto::provenance::action::Action; - use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, ManifestCore}; + use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, KeyMode, ManifestCore}; const USER: u128 = 0x05E2; const DEVICE: u128 = 0xD1; @@ -253,6 +253,9 @@ mod tests { plaintext_size: 12, chunk_size: 65_520, nonce_prefix: [1, 2, 3, 4, 5, 6, 7], + key_mode: KeyMode::Derived, + wrapped_file_key: None, + metadata_blob_hash: None, created_by_user: Uuid::from_u128(USER), created_by_device: Uuid::from_u128(DEVICE), client_version: "capsule-cli/0.1.0".into(), diff --git a/capsule-core/src/lifecycle.rs b/capsule-core/src/lifecycle.rs index b2c52569..52262e03 100644 --- a/capsule-core/src/lifecycle.rs +++ b/capsule-core/src/lifecycle.rs @@ -43,7 +43,7 @@ use crate::crypto::keys::{ }; use crate::crypto::primitives::{CRYPTO_SUITE_ID, PROTOCOL_VERSION}; use crate::crypto::provenance::action::Action; -use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, ManifestCore}; +use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, KeyMode, ManifestCore}; use crate::crypto::provenance::{AssetManifest, ProvenanceChain, ProvenanceRecord}; use crate::crypto::verify_asset::{VerifyOutcome, verify_asset}; use crate::db::{AssetRow, CachedRepresentationRow, DatabaseDriver}; @@ -494,6 +494,9 @@ impl Workspace { plaintext_size: enc.plaintext_size, chunk_size: enc.chunk_size, nonce_prefix: enc.nonce_prefix, + key_mode: KeyMode::Derived, + wrapped_file_key: None, + metadata_blob_hash: None, created_by_user: self.account.user_id, created_by_device: self.account.device.device_id, client_version: concat!("capsule-core/", env!("CARGO_PKG_VERSION")).into(), diff --git a/capsule-core/src/validation/structural.rs b/capsule-core/src/validation/structural.rs index 759274bc..8aa80918 100644 --- a/capsule-core/src/validation/structural.rs +++ b/capsule-core/src/validation/structural.rs @@ -151,7 +151,7 @@ mod tests { use crate::crypto::keys::{AmkVersion, HybridSigningKey}; use crate::crypto::primitives::PROTOCOL_VERSION; use crate::crypto::provenance::action::Action; - use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, ManifestCore}; + use crate::crypto::provenance::manifest::{ASSET_MANIFEST_VERSION, KeyMode, ManifestCore}; #[test] fn hash_length_and_size_and_content_type() { @@ -222,6 +222,9 @@ mod tests { plaintext_size: 10, chunk_size: 65_520, nonce_prefix: [0; 7], + key_mode: KeyMode::Derived, + wrapped_file_key: None, + metadata_blob_hash: None, created_by_user: Uuid::from_u128(3), created_by_device: Uuid::from_u128(4), client_version: "t".into(), From 7c428329506860dcc13e50f9a70ca0476f85b300 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:40:43 -0400 Subject: [PATCH 08/58] feat(core): add drop and sharing module skeletons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two planned capsule-core modules the module map has carried as (planned) rows, materialized as compiling contracts so the implementing slices (S-A5 sharing, S-A6 drops in SLICES.md) bind to a compiler-checked surface instead of prose: - capsule_core::drop — the web-upload client halves: UploadLink / LinkCaps / DropDescriptor (the unsigned guest wire object, kem_ct as a CBOR byte string) / SealedDrop / PendingDrop types, the UploadLinkIssuer + DropAdopter seams lifecycle::Workspace will implement, and the seal_drop entry point (todo!() until S-A6, WASM target noted). Adoption returns the signed create manifest with key_mode = wrapped, closing the loop with the S1 contract types - capsule_core::sharing — ShareScope / ShareLink / ShareLinkId types and the ShareLinkIssuer seam per the share-links contract skeleton, with the client-side passphrase-unwrap rule in the type docs Each module carries #[ignore]d contract tests naming its slice's acceptance criteria (seal round-trip, adoption rewrap acceptance, opaque-id entropy, client-side passphrase unwrap). Compiles under --features ffi with no uniffi surface changes; all existing tests unchanged. --- capsule-core/src/drop/mod.rs | 195 ++++++++++++++++++++++++++++++++ capsule-core/src/lib.rs | 2 + capsule-core/src/sharing/mod.rs | 95 ++++++++++++++++ 3 files changed, 292 insertions(+) create mode 100644 capsule-core/src/drop/mod.rs create mode 100644 capsule-core/src/sharing/mod.rs diff --git a/capsule-core/src/drop/mod.rs b/capsule-core/src/drop/mod.rs new file mode 100644 index 00000000..3ff9b130 --- /dev/null +++ b/capsule-core/src/drop/mod.rs @@ -0,0 +1,195 @@ +//! Web-upload guest drops — **contract skeleton** (slice `S-A6` in the repo-root +//! `SLICES.md`; SSoT: [Web Upload]). +//! +//! A guest with an upload link seals each asset under a fresh random key `K`, +//! encapsulates `K` to the link's Drop Key, and uploads the sealed bytes to the +//! provisioning user's staging inbox. Nothing becomes a library asset until one of that +//! user's trusted clients **adopts** the drop — decapsulating `K`, rewrapping it under the +//! album AMK (`asset-keywrap/v1`, [`KeyMode::Wrapped`]), and signing an ordinary `create` +//! manifest. The guest is never a signer; drops never flow through `verify_asset`. +//! +//! This module owns the client-side halves: link issuance, drop sealing (compiled to WASM +//! for `capsule-web`), and adoption. The server halves (drop store, inbox, atomic +//! inbox→album promotion) live in `capsule-api-media::drops`. +//! +//! [Web Upload]: https://docs/design/web-upload/ +//! [`KeyMode::Wrapped`]: crate::crypto::provenance::KeyMode + +use serde::{Deserialize, Serialize}; +use thiserror::Error; +use uuid::Uuid; + +use crate::crypto::hash::Hash32; +use crate::crypto::provenance::AssetManifest; + +/// A provisioned upload link: the server-held record plus the fragment-delivered public +/// half. The `opaque_id` follows the share-link rule (random ≥128-bit, never structured); +/// `drop_pubkey` travels only in the URL fragment and never reaches the server. +#[derive(Debug, Clone)] +pub struct UploadLink { + /// The link's random 128-bit opaque id (the URL path component). + pub opaque_id: [u8; 16], + /// The Drop Key public half (KEM encapsulation key; URL fragment only). + pub drop_pubkey: Vec, + /// The caps this link was provisioned with. + pub caps: LinkCaps, +} + +/// Identifies a provisioned upload link for revocation. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct UploadLinkId(pub Uuid); + +/// Identifies a pending drop in the provisioning user's inbox. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct DropId(pub Uuid); + +/// Per-link caps, enforced server-side at the no-key layer on every drop-session +/// creation ([Web Upload — Security Contract](https://docs/design/web-upload/)). +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct LinkCaps { + /// RFC 3339 expiry; `None` = no expiry (revocation still applies). + pub expires_at: Option, + /// Cumulative byte cap across all drops on this link. + pub max_total_bytes: Option, + /// Maximum number of files this link may deposit. + pub max_file_count: Option, + /// Maximum single-file size. + pub max_file_size: Option, + /// Whether the link dies after its first successful drop. + pub single_use: bool, +} + +/// The unsigned descriptor a guest uploads beside the sealed ciphertext. Deliberately +/// **not** an `AssetManifest`: no signatures, no `album_id`, no provenance link. Its +/// integrity is established only when a trusted client decapsulates `K` and the STREAM +/// tags verify. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct DropDescriptor { + /// Closed enum for the link's pinned `protocol_version` (same set as a manifest's). + pub content_type: String, + /// Total plaintext byte length. + pub plaintext_size: u64, + /// The STREAM plaintext chunk size (owned by Encryption). + pub chunk_size: u32, + /// The STREAM nonce prefix used for this seal. + pub nonce_prefix: [u8; 7], + /// Content-address digest of the STREAM ciphertext. + pub ciphertext_hash: Hash32, + /// `K` encapsulated to the link's Drop Key; length fixed by `crypto_suite_id`. + #[serde(with = "serde_bytes")] + pub kem_ct: Vec, + /// Guest-supplied, unverified; advisory only. + pub suggested_filename: Option, +} + +/// A sealed drop ready for upload: the descriptor plus the STREAM ciphertext. +#[derive(Debug, Clone)] +pub struct SealedDrop { + /// The unsigned descriptor. + pub descriptor: DropDescriptor, + /// The STREAM ciphertext bytes. + pub ciphertext: Vec, +} + +/// A drop awaiting review in the provisioning user's inbox. +#[derive(Debug, Clone)] +pub struct PendingDrop { + /// The inbox row id. + pub drop_id: DropId, + /// The guest's descriptor. + pub descriptor: DropDescriptor, + /// The link it arrived through. + pub via_link: UploadLinkId, + /// Server-attested arrival time (RFC 3339, `received_at`). + pub received_at: String, +} + +/// Failure surfaced by the drop lifecycle. +#[derive(Debug, Error)] +pub enum DropError { + /// The link is expired, revoked, or over a cap. + #[error("upload link refused: {0}")] + LinkRefused(&'static str), + /// The KEM decapsulation or STREAM verification failed. + #[error("drop crypto failure: {0}")] + Crypto(&'static str), + /// The drop was not found in the caller's inbox. + #[error("pending drop not found")] + NotFound, +} + +/// Issues and revokes upload links on a trusted (native) client — the seam +/// `lifecycle::Workspace` will implement. Provisioning mints the Drop Key, wraps its +/// private half under the master key + OGK escrow, and registers the link record. +pub trait UploadLinkIssuer { + /// Provision an upload link with `caps`; `passphrase` adds the server-verified + /// Argon2id abuse gate (never transmitted — the record stores a verifier). + fn create_link( + &mut self, + caps: LinkCaps, + passphrase: Option<&str>, + ) -> Result; + + /// Revoke a link; the serve path refuses it within its fail-closed cache window. + fn revoke_link(&mut self, link: UploadLinkId) -> Result<(), DropError>; +} + +/// Reviews and adopts pending drops on a trusted (native) client — decapsulate `K`, +/// rewrap under the destination album's AMK, author the sidecar, sign the `create` +/// manifest with `key_mode = wrapped`, and submit the atomic inbox→album promotion. +pub trait DropAdopter { + /// The provisioning user's pending drops. + fn list_inbox(&self) -> Result, DropError>; + + /// Adopt a drop into `album_id` in place (no byte re-upload). Returns the signed + /// adopting `create` manifest whose `ciphertext_hash` references the inbox blob. + fn adopt(&mut self, drop: DropId, album_id: Uuid) -> Result; + + /// Discard a pending drop; its bytes are GC'd and the quota freed. + fn discard(&mut self, drop: DropId) -> Result<(), DropError>; +} + +/// Seal `plaintext` for a guest drop: draw a fresh random `K`, STREAM-encrypt under it, +/// and encapsulate `K` to `drop_pubkey` (the link's KEM public half, from the URL +/// fragment). Runs in the browser (WASM) and on native clients alike. +/// +/// # Panics +/// Unimplemented skeleton (slice `S-A6`). +pub fn seal_drop( + plaintext: &[u8], + drop_pubkey: &[u8], + content_type: &str, +) -> Result { + let (_, _, _) = (plaintext, drop_pubkey, content_type); + todo!("S-A6: drop sealing — see SLICES.md") +} + +#[cfg(test)] +mod tests { + /// `S-A6` acceptance: seal a plaintext to a Drop Key public half, decapsulate with + /// the private half, STREAM-decrypt, assert byte equality; assert `kem_ct` length + /// matches the suite; assert the descriptor round-trips through canonical CBOR. + #[test] + #[ignore = "S-A6 contract: drop seal round-trip not yet implemented"] + fn drop_seal_round_trip() { + unimplemented!("implemented by slice S-A6"); + } + + /// `S-A6` acceptance: adopt a sealed drop — decapsulate `K`, rewrap under a test AMK + /// (`asset-keywrap/v1`), build the `create` manifest with `key_mode = wrapped`, and + /// assert `verify_asset` accepts while a second member can unwrap and decrypt the + /// unchanged ciphertext. + #[test] + #[ignore = "S-A6 contract: adoption rewrap not yet implemented"] + fn adoption_rewrap_verifies_and_decrypts() { + unimplemented!("implemented by slice S-A6"); + } + + /// `S-A6` acceptance: generated upload-link opaque ids are ≥128-bit CSPRNG values — + /// never UUIDv7 or otherwise structured (identical rule to share links). + #[test] + #[ignore = "S-A6 contract: link issuance not yet implemented"] + fn opaque_id_entropy() { + unimplemented!("implemented by slice S-A6"); + } +} diff --git a/capsule-core/src/lib.rs b/capsule-core/src/lib.rs index 31b1539b..3cd6babf 100644 --- a/capsule-core/src/lib.rs +++ b/capsule-core/src/lib.rs @@ -4,12 +4,14 @@ pub mod constants; pub mod crypto; pub mod db; pub mod domain; +pub mod drop; pub mod exif; pub mod import; pub mod library; pub mod lifecycle; pub mod metadata; pub mod models; +pub mod sharing; pub mod sidecar; pub mod utils; pub mod validation; diff --git a/capsule-core/src/sharing/mod.rs b/capsule-core/src/sharing/mod.rs new file mode 100644 index 00000000..67bd5132 --- /dev/null +++ b/capsule-core/src/sharing/mod.rs @@ -0,0 +1,95 @@ +//! Share links — **contract skeleton** (slice `S-A5` in the repo-root `SLICES.md`; +//! SSoT: [Share Links]). +//! +//! A share link grants **view-only** access to an asset or album to a recipient with no +//! Capsule account: `https://server.tld/s/{opaque-id}#{secret}`. The fragment secret +//! carries the decryption material and never reaches the server; an optional passphrase +//! wraps it a second time via the password-based KDF, unwrapped **client-side** (the +//! server stores and returns only the wrapped material). The serving endpoints live in +//! `capsule-api-media::shares`; this module owns link generation and capability +//! validation on the issuing client. +//! +//! [Share Links]: https://docs/design/share-links/ + +use thiserror::Error; +use uuid::Uuid; + +/// What a share link points at. The `{opaque-id}` itself carries **no** scope — the +/// server resolves scope from the link record, so the URL leaks nothing. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ShareScope { + /// A single asset. + Asset(Uuid), + /// A whole album. + Album(Uuid), +} + +/// Identifies an issued share link for revocation. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ShareLinkId(pub Uuid); + +/// An issued share link: the server-held record plus the fragment secret. +#[derive(Debug, Clone)] +pub struct ShareLink { + /// Revocation handle. + pub link_id: ShareLinkId, + /// The random 128-bit opaque id (URL path component; never structured/UUIDv7). + pub opaque_id: [u8; 16], + /// The fragment secret carrying the decryption material (never sent to the server). + /// When a passphrase is set, this is the **wrapped** form; unwrap is client-side. + pub secret: Vec, + /// RFC 3339 expiry, if any. + pub expires_at: Option, +} + +/// Failure surfaced by share-link issuance/revocation. +#[derive(Debug, Error)] +pub enum SharingError { + /// The scope does not exist or the issuer lacks access to it. + #[error("share scope unavailable")] + ScopeUnavailable, + /// The link was not found (already revoked, or never issued). + #[error("share link not found")] + NotFound, + /// Key material could not be prepared for the link secret. + #[error("share crypto failure: {0}")] + Crypto(&'static str), +} + +/// Issues and revokes view-only share links on a trusted client — the seam +/// `lifecycle::Workspace` will implement. Issuance encapsulates the scope's decryption +/// material around a fresh link secret; the optional passphrase adds the second, +/// client-side-unwrapped encapsulation layer. +pub trait ShareLinkIssuer { + /// Issue a view-only link for `scope`, optionally expiring and/or + /// passphrase-wrapped. + fn create_link( + &mut self, + scope: ShareScope, + expires_at: Option, + passphrase: Option<&str>, + ) -> Result; + + /// Revoke a link; the serve path refuses it within its fail-closed cache window. + fn revoke_link(&mut self, link: ShareLinkId) -> Result<(), SharingError>; +} + +#[cfg(test)] +mod tests { + /// `S-A5` acceptance: generated opaque ids are ≥128-bit CSPRNG values, never + /// structured; a generator producing shorter or guessable ids fails. + #[test] + #[ignore = "S-A5 contract: link issuance not yet implemented"] + fn opaque_id_entropy() { + unimplemented!("implemented by slice S-A5"); + } + + /// `S-A5` acceptance: with a passphrase set, the issued `secret` is the wrapped form + /// and unwraps client-side via the password-based KDF; the passphrase itself never + /// appears in any wire-bound structure. + #[test] + #[ignore = "S-A5 contract: passphrase wrap not yet implemented"] + fn passphrase_unwrap_is_client_side() { + unimplemented!("implemented by slice S-A5"); + } +} From 81073c2eb102e4103123f50fb40a2ac536346576 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:43:03 -0400 Subject: [PATCH 09/58] feat(core): add storage and streaming import seams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The remaining capsule-core seams DEFERRED.md described in prose, materialized as code contracts: - library::space — available_bytes() (the statvfs/GetDiskFreeSpaceEx probe; todo!() until slice S-B3, since it is real I/O) plus the pure streaming_recommended predicate, implemented and tested: the planner stays deterministic and the probe attaches at confirmation. The planner's total_size accounting is explicitly S-B3 work — the current planner emits per-candidate decisions and counts only - library::storage_verify — the verify-before-destroy gate's client types (StorageVerdict / BlobVerdict / BlobRole mirroring the POST /storage/verify response) and the pure release_is_safe conjunction, implemented and tested: durable AND per-blob re-check AND verify_asset acceptance, with the client never trusting the server's aggregate over details it can recompute. Fetching the verdict and wiring the destructive paths is S-C3/S-D4 work, named by the #[ignore]d contract tests --- capsule-core/src/library/mod.rs | 4 + capsule-core/src/library/space.rs | 57 +++++++++ capsule-core/src/library/storage_verify.rs | 130 +++++++++++++++++++++ 3 files changed, 191 insertions(+) create mode 100644 capsule-core/src/library/space.rs create mode 100644 capsule-core/src/library/storage_verify.rs diff --git a/capsule-core/src/library/mod.rs b/capsule-core/src/library/mod.rs index 3a7e6959..67960d75 100644 --- a/capsule-core/src/library/mod.rs +++ b/capsule-core/src/library/mod.rs @@ -8,6 +8,8 @@ pub mod open; pub mod paths; pub mod rebuild; pub mod scrub; +pub mod space; +pub mod storage_verify; pub mod trash; pub use cache::{EvictionReport, cache_sweep}; @@ -20,3 +22,5 @@ pub use paths::{ transcode_h264_path, transcode_live_path, trash_path, uuid_shard, }; pub use rebuild::rebuild_index; +pub use space::{available_bytes, streaming_recommended}; +pub use storage_verify::{BlobRole, BlobVerdict, StorageVerdict, release_is_safe}; diff --git a/capsule-core/src/library/space.rs b/capsule-core/src/library/space.rs new file mode 100644 index 00000000..9317d154 --- /dev/null +++ b/capsule-core/src/library/space.rs @@ -0,0 +1,57 @@ +//! Free-space probe + streaming-import recommendation — the storage-constrained-import +//! seam (slice `S-B3` in the repo-root `SLICES.md`; SSoT: +//! [Import — Pipeline](https://docs/design/import/pipeline/)). +//! +//! The probe is I/O and runs *outside* the pure planner: it is attached at plan +//! confirmation, so the planner stays deterministic. The recommendation predicate below +//! is pure and is the contract the executor's streaming drive mode keys off. Note the +//! current planner emits per-candidate decisions and counts only — the `total_size` +//! accounting and the plan-level `streaming_recommended` attachment are part of `S-B3`. + +use std::path::Path; + +use super::error::LibraryError; + +/// The library volume's available bytes — a thin `statvfs` / `GetDiskFreeSpaceEx` +/// wrapper over the filesystem holding `library_root`. +/// +/// # Panics +/// Unimplemented skeleton (slice `S-B3`). +pub fn available_bytes(library_root: &Path) -> Result { + let _ = library_root; + todo!("S-B3: free-space probe — see SLICES.md") +} + +/// The pure streaming-recommendation predicate: streaming mode is recommended when the +/// plan's total size, plus a configurable headroom margin, meets or exceeds the volume's +/// available bytes. Attached to the plan at confirmation (never computed inside the pure +/// planner). +pub fn streaming_recommended(total_size: u64, available: u64, headroom_margin: u64) -> bool { + total_size.saturating_add(headroom_margin) >= available +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn streaming_recommended_thresholds() { + // Comfortably under free space: no streaming. + assert!(!streaming_recommended(100, 1_000, 50)); + // Within the headroom margin of free space: streaming. + assert!(streaming_recommended(960, 1_000, 50)); + // Over free space outright: streaming. + assert!(streaming_recommended(2_000, 1_000, 0)); + // Saturating: absurd sizes never overflow. + assert!(streaming_recommended(u64::MAX, 1_000, u64::MAX)); + } + + /// `S-B3` acceptance: the probe reports the real volume's free bytes (within I/O + /// slack) on every supported platform, and the executor's streaming drive mode keys + /// off `streaming_recommended` fed by it. + #[test] + #[ignore = "S-B3 contract: free-space probe not yet implemented"] + fn available_bytes_reports_real_free_space() { + unimplemented!("implemented by slice S-B3"); + } +} diff --git a/capsule-core/src/library/storage_verify.rs b/capsule-core/src/library/storage_verify.rs new file mode 100644 index 00000000..870b45db --- /dev/null +++ b/capsule-core/src/library/storage_verify.rs @@ -0,0 +1,130 @@ +//! The verify-before-destroy gate — client half of storage verification (slices `S-C3` +//! server endpoint / `S-D4` client wiring in the repo-root `SLICES.md`; SSoT: +//! [Import — Storage Verification](https://docs/design/import/storage-verification/)). +//! +//! Before any post-write local cleanup of irreplaceable bytes (releasing a device-owned +//! original, deleting a Move-import source, a streaming-mode release), a client requires +//! **both** halves to pass: `verify_asset` accepts the asset (crypto validity — the +//! offline core already implements it) and the server's `POST /storage/verify` verdict is +//! `durable` (stored ∧ indexed ∧ retrievable for every required blob). The predicate here +//! is the pure conjunction those call sites consume; fetching the verdict is `S-D4`. + +use uuid::Uuid; + +use crate::crypto::hash::Hash32; + +/// A blob's role within an asset, as the storage-verification endpoint reports it +/// (closed enum; the value set is owned by the storage-verification doc). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BlobRole { + /// The original ciphertext blob. + Original, + /// The encrypted metadata blob. + Metadata, + /// A derivative (thumbnail / preview / embedding) blob. + Derivative, + /// The provenance chain. + Provenance, +} + +/// The server's key-free per-blob verdict. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct BlobVerdict { + /// The content address the client declared it relies on. + pub hash: Hash32, + /// The blob's role on the asset. + pub role: BlobRole, + /// Present in the blob store at its content address (`stat`), not merely in-flight. + pub stored: bool, + /// Referenced by a committed, `uploaded = true` index row. + pub indexed: bool, + /// Refcount > 0, not mid-GC (`collectable_since`), not quarantined. + pub retrievable: bool, +} + +impl BlobVerdict { + /// One blob's contribution to durability: all three independent facts hold. + pub fn safely_stored(&self) -> bool { + self.stored && self.indexed && self.retrievable + } +} + +/// The per-asset verdict from `POST /storage/verify`. `durable` attests **this home +/// server's** storage only — never replicas or peers. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct StorageVerdict { + /// The asset the verdict is for. + pub asset_id: Uuid, + /// Server-computed: every required blob is stored ∧ indexed ∧ retrievable. + pub durable: bool, + /// Per-blob detail, one entry per hash the client declared. + pub blobs: Vec, + /// The server's trusted clock at verification (RFC 3339, like `received_at`). + pub checked_at: String, +} + +/// The verify-before-destroy predicate: destructive local cleanup of irreplaceable bytes +/// may proceed **only** when the server's verdict is `durable`, every declared blob +/// individually re-checks as safely stored (the client never trusts the server's +/// aggregate over the details it can recompute), and `verify_asset` accepted the asset. +/// +/// A `false` result never triggers a destructive action — the caller retains the local +/// copy, retries with backoff, and surfaces "not yet confirmed on server". +pub fn release_is_safe(verdict: &StorageVerdict, verify_asset_accepted: bool) -> bool { + verify_asset_accepted + && verdict.durable + && !verdict.blobs.is_empty() + && verdict.blobs.iter().all(BlobVerdict::safely_stored) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn blob(stored: bool, indexed: bool, retrievable: bool) -> BlobVerdict { + BlobVerdict { + hash: Hash32([0xAA; 32]), + role: BlobRole::Original, + stored, + indexed, + retrievable, + } + } + + fn verdict(durable: bool, blobs: Vec) -> StorageVerdict { + StorageVerdict { + asset_id: Uuid::from_u128(1), + durable, + blobs, + checked_at: "2026-07-02T00:00:00Z".into(), + } + } + + #[test] + fn release_requires_both_halves_and_consistent_details() { + let good = verdict(true, vec![blob(true, true, true)]); + assert!(release_is_safe(&good, true)); + // Crypto half failed: never release. + assert!(!release_is_safe(&good, false)); + // Server says durable but a detail row disagrees: the client's re-check wins. + let inconsistent = verdict(true, vec![blob(true, true, false)]); + assert!(!release_is_safe(&inconsistent, true)); + // Not durable: never release. + assert!(!release_is_safe( + &verdict(false, vec![blob(true, true, true)]), + true + )); + // An empty verdict confirms nothing. + assert!(!release_is_safe(&verdict(true, vec![]), true)); + } + + /// `S-D4` acceptance: the eviction sweep, Move-import source deletion, and + /// streaming-mode release all gate on this predicate fed by a real + /// `POST /storage/verify` response, and a non-`durable` verdict retains the local + /// copy and surfaces the unconfirmed state. + #[test] + #[ignore = "S-D4 contract: verify-before-destroy wiring not yet implemented"] + fn destructive_paths_gate_on_release_is_safe() { + unimplemented!("implemented by slice S-D4"); + } +} From 575ef2ce719c91296c0b82ba31c57b89bec38688 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:48:42 -0400 Subject: [PATCH 10/58] feat(api): add key-free route skeletons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The planned REST surfaces from the module map, materialized as oapi-annotated salvo stubs (the pattern the share route already ships): compiling handlers with todo!() bodies whose request/response DTOs and status semantics are the contract, mounted so they land in the generated OpenAPI spec the SDK is built from. - capsule-api-media routes/verify.rs — POST /storage/verify (slice S-C3): the batched key-free durability verdict (stored / indexed / retrievable per declared blob hash, deep re-hash opt-in), the server half of the verify-before-destroy gate whose client predicate landed in capsule-core - capsule-api-media routes/drops.rs — the web-upload drop store (slice S-C5): guest drop-session creation under /u/{opaque-id}/drop (indistinguishable 404 for dead links, 403 for cap/quota refusals), the owner inbox at /drops, atomic adoption at /drops/{id}/adopt, and discard; mounts under /v1/storage, /v1/u, /v1/drops behind the existing media feature with new storage/drops OpenAPI tags - capsule-api-auth routes/devices.rs — the enrollment ceremony surface (slice S-C7) at devices/enroll + devices/enroll/redeem, deliberately distinct paths from the existing GET /auth/devices session listing; issuance requires bearer auth (plus the fresh local-authorization rule in the doc), redemption is code-authenticated - capsule-api-upload envelope.rs — the EnvelopeGate middleware seam (slice S-C1) that wires the implemented, exhaustively-tested capsule_core::validation invariants into the write path; unmounted stub until the slice lands Workspace builds and lints clean; no existing route or behavior changes. --- capsule-api/auth/src/routes/devices.rs | 149 ++++++++++++++ capsule-api/auth/src/routes/mod.rs | 13 +- capsule-api/media/src/lib.rs | 27 +++ capsule-api/media/src/routes/drops.rs | 260 +++++++++++++++++++++++++ capsule-api/media/src/routes/mod.rs | 30 +++ capsule-api/media/src/routes/verify.rs | 127 ++++++++++++ capsule-api/src/lib.rs | 18 ++ capsule-api/upload/src/envelope.rs | 42 ++++ capsule-api/upload/src/lib.rs | 1 + 9 files changed, 666 insertions(+), 1 deletion(-) create mode 100644 capsule-api/auth/src/routes/devices.rs create mode 100644 capsule-api/media/src/routes/drops.rs create mode 100644 capsule-api/media/src/routes/verify.rs create mode 100644 capsule-api/upload/src/envelope.rs diff --git a/capsule-api/auth/src/routes/devices.rs b/capsule-api/auth/src/routes/devices.rs new file mode 100644 index 00000000..16e4b4dd --- /dev/null +++ b/capsule-api/auth/src/routes/devices.rs @@ -0,0 +1,149 @@ +//! Device enrollment — the cross-device-add server surface (contract skeleton; slice +//! `S-C7` in the repo-root `SLICES.md`; SSoT: ). +//! +//! An existing signed-in device issues a single-use enrollment code (fresh local device +//! authorization required on top of the session — a stolen session token alone cannot +//! enroll a rogue device); the new device redeems it to establish the relay channel. The +//! code is single-use, expires in 10 minutes, is rate-limited at redemption, and is +//! deleted on redemption or expiry. Paths live under `devices/enroll` — plain +//! `GET /auth/devices` is the existing session listing, a different surface. + +use salvo::oapi::extract::JsonBody; +use salvo::prelude::*; +use serde::{Deserialize, Serialize}; + +/// The issued enrollment code, displayed as a QR (full entropy) with a shorter, +/// rate-limited numeric text fallback. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct EnrollmentCodeResponse { + /// The full-entropy code for the QR payload (base64; ≥64 bits). + pub code: String, + /// The shorter transcribable fallback (safe because redemption is single-use, + /// expiring, and rate-limited; channel integrity rests on the safety code). + pub text_fallback: String, + /// RFC 3339 expiry (10 minutes from issue). + pub expires_at: String, +} + +/// A redemption request from the new device. +#[derive(Debug, Deserialize, ToSchema)] +#[allow(dead_code)] +pub(super) struct RedeemRequest { + /// The scanned or typed enrollment code. + pub code: String, +} + +/// The established relay channel handle the ceremony continues over. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct ChannelResponse { + /// Opaque relay-channel handle for the ephemeral-ECDH ceremony messages. + pub channel_id: String, +} + +#[derive(Serialize)] +struct ErrorResponse { + error: String, +} + +/// Responses for enrollment-code issuance. +#[allow(dead_code)] +pub(super) enum IssueResponses { + /// Code issued. + Ok(EnrollmentCodeResponse), + /// The session lacks the fresh local device authorization the ceremony requires. + LocalAuthRequired, +} + +#[async_trait] +impl Writer for IssueResponses { + async fn write(mut self, req: &mut Request, depot: &mut Depot, res: &mut Response) { + match self { + Self::Ok(data) => { + res.status_code(StatusCode::OK); + Json(data).write(req, depot, res).await; + } + Self::LocalAuthRequired => { + res.status_code(StatusCode::FORBIDDEN); + res.render(Json(ErrorResponse { + error: "fresh local device authorization required".into(), + })); + } + } + } +} + +impl EndpointOutRegister for IssueResponses { + fn register(components: &mut salvo::oapi::Components, operation: &mut salvo::oapi::Operation) { + operation.responses.insert( + String::from("200"), + salvo::oapi::Response::new("Enrollment code issued").add_content( + "application/json", + salvo::oapi::Content::new(EnrollmentCodeResponse::to_schema(components)), + ), + ); + operation.responses.insert( + String::from("403"), + salvo::oapi::Response::new("Fresh local device authorization required"), + ); + } +} + +/// Responses for enrollment-code redemption. +#[allow(dead_code)] +pub(super) enum RedeemResponses { + /// Channel established. + Ok(ChannelResponse), + /// The code is unknown, expired, already redeemed, or rate-limited — + /// indistinguishable by design. + Refused, +} + +#[async_trait] +impl Writer for RedeemResponses { + async fn write(mut self, req: &mut Request, depot: &mut Depot, res: &mut Response) { + match self { + Self::Ok(data) => { + res.status_code(StatusCode::OK); + Json(data).write(req, depot, res).await; + } + Self::Refused => { + res.status_code(StatusCode::NOT_FOUND); + } + } + } +} + +impl EndpointOutRegister for RedeemResponses { + fn register(components: &mut salvo::oapi::Components, operation: &mut salvo::oapi::Operation) { + operation.responses.insert( + String::from("200"), + salvo::oapi::Response::new("Relay channel established").add_content( + "application/json", + salvo::oapi::Content::new(ChannelResponse::to_schema(components)), + ), + ); + operation.responses.insert( + String::from("404"), + salvo::oapi::Response::new( + "Unknown, expired, redeemed, or rate-limited code (indistinguishable)", + ), + ); + } +} + +/// Issue a single-use enrollment code from an existing, locally-authorized device. +#[endpoint(operation_id = "issue_enrollment_code", tags("auth"), security(("bearer" = [])))] +pub async fn issue_enrollment_code(_req: &mut Request, _depot: &mut Depot) -> IssueResponses { + todo!("S-C7: enrollment-code issuance — see SLICES.md") +} + +/// Redeem an enrollment code from the new device (unauthenticated; the code is the +/// credential). +#[endpoint(operation_id = "redeem_enrollment_code", tags("auth"))] +pub async fn redeem_enrollment_code( + _req: &mut Request, + _depot: &mut Depot, + _body: JsonBody, +) -> RedeemResponses { + todo!("S-C7: enrollment-code redemption — see SLICES.md") +} diff --git a/capsule-api/auth/src/routes/mod.rs b/capsule-api/auth/src/routes/mod.rs index 343a0856..f49fa272 100644 --- a/capsule-api/auth/src/routes/mod.rs +++ b/capsule-api/auth/src/routes/mod.rs @@ -1,4 +1,5 @@ mod auth; +mod devices; mod passkey; mod password; mod profile; @@ -28,7 +29,17 @@ pub(super) fn get_router(state: AppState) -> Router { ) .push(Router::with_path("refresh").post(auth::refresh_token)) .push(Router::with_path("validate").post(auth::validate_token)) - .push(Router::with_path("devices").get(auth::get_devices)) + .push( + Router::with_path("devices") + .get(auth::get_devices) + // Device-enrollment ceremony (skeleton — slice S-C7 in SLICES.md); + // distinct from the session listing above. + .push( + Router::with_path("enroll") + .post(devices::issue_enrollment_code) + .push(Router::with_path("redeem").post(devices::redeem_enrollment_code)), + ), + ) .push(Router::with_path("logout").post(auth::logout)) // Password routes .push(Router::with_path("password-reset-request").post(password::reset_password_request)) diff --git a/capsule-api/media/src/lib.rs b/capsule-api/media/src/lib.rs index f378c2d7..eee4e4f7 100644 --- a/capsule-api/media/src/lib.rs +++ b/capsule-api/media/src/lib.rs @@ -29,3 +29,30 @@ pub async fn get_share_router>( Ok(routes::get_share_router(state)) } + +/// Storage-verification router (`POST /storage/verify`). Skeleton — slice `S-C3`. +pub async fn get_storage_router>( + conn: DatabaseConnection, + config: C, +) -> Result { + let state = AppState::new(conn, config.into()); + Ok(routes::get_storage_router(state)) +} + +/// Guest drop-session router (`/u/{opaque-id}/drop`). Skeleton — slice `S-C5`. +pub async fn get_drop_link_router>( + conn: DatabaseConnection, + config: C, +) -> Result { + let state = AppState::new(conn, config.into()); + Ok(routes::get_drop_link_router(state)) +} + +/// Owner drop-inbox router (`/drops`). Skeleton — slice `S-C5`. +pub async fn get_drops_router>( + conn: DatabaseConnection, + config: C, +) -> Result { + let state = AppState::new(conn, config.into()); + Ok(routes::get_drops_router(state)) +} diff --git a/capsule-api/media/src/routes/drops.rs b/capsule-api/media/src/routes/drops.rs new file mode 100644 index 00000000..4fa81e5e --- /dev/null +++ b/capsule-api/media/src/routes/drops.rs @@ -0,0 +1,260 @@ +//! Web-upload guest drops — the drop store, staging inbox, and adoption transition +//! (contract skeleton; slice `S-C5` in the repo-root `SLICES.md`; SSoT: +//! ). +//! +//! Guest-facing (link-capability auth, no account): `POST /u/{opaque-id}/drop` opens a +//! drop session (per-link caps + owner quota checked here; invariants 26–31), and chunks +//! reuse the upload protocol's `PATCH` mechanics. Owner-facing (session auth): +//! `GET /drops` lists the inbox, `POST /drops/{id}/adopt` runs the atomic inbox→album +//! promotion against the adopter's signed `create` manifest (invariant 32), and +//! `DELETE /drops/{id}` discards. A not-found, revoked, or expired link returns an +//! indistinguishable `404` — never `410`. + +use salvo::oapi::extract::{JsonBody, PathParam}; +use salvo::prelude::*; +use serde::{Deserialize, Serialize}; + +/// The unsigned guest descriptor uploaded beside the sealed ciphertext (the canonical +/// shape is `capsule_core::drop::DropDescriptor`; carried here as its JSON projection). +#[derive(Debug, Deserialize, ToSchema)] +#[allow(dead_code)] +pub(super) struct DropDescriptorBody { + /// Closed enum for the link's pinned protocol version. + pub content_type: String, + /// Total plaintext byte length. + pub plaintext_size: u64, + /// STREAM plaintext chunk size. + pub chunk_size: u32, + /// STREAM nonce prefix (7 bytes, hex). + pub nonce_prefix: String, + /// Content address (hex) of the STREAM ciphertext. + pub ciphertext_hash: String, + /// `K` encapsulated to the link's Drop Key (base64); length fixed by the suite. + pub kem_ct: String, + /// Guest-supplied, unverified; advisory only. + pub suggested_filename: Option, +} + +/// A drop session, ready to receive chunks. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct DropSessionResponse { + /// The drop session id (chunks `PATCH` against it). + pub drop_id: String, +} + +/// One pending drop in the provisioning user's inbox. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct PendingDropResponse { + /// The inbox row id. + pub drop_id: String, + /// The guest's descriptor. + pub content_type: String, + /// Declared plaintext size. + pub plaintext_size: u64, + /// Guest-supplied name, unverified. + pub suggested_filename: Option, + /// Server-attested arrival time (RFC 3339). + pub received_at: String, +} + +/// The inbox listing. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct InboxResponse { + /// Pending drops awaiting review. + pub drops: Vec, +} + +/// The adoption request: the adopter's signed `create` manifest (canonical CBOR, +/// base64) whose `ciphertext_hash` references the inbox blob, plus the freshly sealed +/// metadata blob it commits to. +#[derive(Debug, Deserialize, ToSchema)] +#[allow(dead_code)] +pub(super) struct AdoptRequest { + /// The signed `create` manifest (canonical CBOR, base64), `key_mode = wrapped`. + pub manifest_cbor: String, + /// The encrypted metadata blob (base64) matching the manifest's `metadata_blob_hash`. + pub metadata_blob: String, +} + +/// The adoption result. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct AdoptResponse { + /// The promoted asset's id. + pub asset_id: String, +} + +#[derive(Serialize)] +struct ErrorResponse { + error: String, +} + +/// Responses for guest drop-session creation. +#[allow(dead_code)] +pub(super) enum DropSessionResponses { + /// Session opened. + Created(DropSessionResponse), + /// Link not found, revoked, or expired — indistinguishable by design (never `410`). + NotFound, + /// A per-link cap or the owner's quota refused the session. + Refused(String), +} + +#[async_trait] +impl Writer for DropSessionResponses { + async fn write(mut self, req: &mut Request, depot: &mut Depot, res: &mut Response) { + match self { + Self::Created(data) => { + res.status_code(StatusCode::CREATED); + Json(data).write(req, depot, res).await; + } + Self::NotFound => { + res.status_code(StatusCode::NOT_FOUND); + } + Self::Refused(msg) => { + res.status_code(StatusCode::FORBIDDEN); + res.render(Json(ErrorResponse { error: msg })); + } + } + } +} + +impl EndpointOutRegister for DropSessionResponses { + fn register(components: &mut salvo::oapi::Components, operation: &mut salvo::oapi::Operation) { + operation.responses.insert( + String::from("201"), + salvo::oapi::Response::new("Drop session opened").add_content( + "application/json", + salvo::oapi::Content::new(DropSessionResponse::to_schema(components)), + ), + ); + operation.responses.insert( + String::from("404"), + salvo::oapi::Response::new("Link not found, revoked, or expired (indistinguishable)"), + ); + operation.responses.insert( + String::from("403"), + salvo::oapi::Response::new("Cap or quota refused the session"), + ); + } +} + +/// Responses for owner-facing inbox operations. +#[allow(dead_code)] +pub(super) enum InboxResponses { + /// Inbox listed. + Ok(InboxResponse), +} + +#[async_trait] +impl Writer for InboxResponses { + async fn write(mut self, req: &mut Request, depot: &mut Depot, res: &mut Response) { + match self { + Self::Ok(data) => { + res.status_code(StatusCode::OK); + Json(data).write(req, depot, res).await; + } + } + } +} + +impl EndpointOutRegister for InboxResponses { + fn register(components: &mut salvo::oapi::Components, operation: &mut salvo::oapi::Operation) { + operation.responses.insert( + String::from("200"), + salvo::oapi::Response::new("Pending drops").add_content( + "application/json", + salvo::oapi::Content::new(InboxResponse::to_schema(components)), + ), + ); + } +} + +/// Responses for adoption / discard. +#[allow(dead_code)] +pub(super) enum AdoptResponses { + /// The blob was atomically promoted from inbox to album asset. + Ok(AdoptResponse), + /// The drop is not in the caller's inbox. + NotFound, + /// The manifest failed the envelope invariants (1–8, 16–18, 25, 32). + Rejected(String), +} + +#[async_trait] +impl Writer for AdoptResponses { + async fn write(mut self, req: &mut Request, depot: &mut Depot, res: &mut Response) { + match self { + Self::Ok(data) => { + res.status_code(StatusCode::OK); + Json(data).write(req, depot, res).await; + } + Self::NotFound => { + res.status_code(StatusCode::NOT_FOUND); + } + Self::Rejected(msg) => { + res.status_code(StatusCode::BAD_REQUEST); + res.render(Json(ErrorResponse { error: msg })); + } + } + } +} + +impl EndpointOutRegister for AdoptResponses { + fn register(components: &mut salvo::oapi::Components, operation: &mut salvo::oapi::Operation) { + operation.responses.insert( + String::from("200"), + salvo::oapi::Response::new("Drop promoted to album asset").add_content( + "application/json", + salvo::oapi::Content::new(AdoptResponse::to_schema(components)), + ), + ); + operation.responses.insert( + String::from("404"), + salvo::oapi::Response::new("Drop not found in the caller's inbox"), + ); + operation.responses.insert( + String::from("400"), + salvo::oapi::Response::new("Envelope invariants rejected the manifest"), + ); + } +} + +/// Open a drop session against a live upload link (guest; link-capability auth). +#[endpoint(operation_id = "create_drop_session", tags("drops"))] +pub async fn create_drop_session( + _req: &mut Request, + _depot: &mut Depot, + _opaque_id: PathParam, + _body: JsonBody, +) -> DropSessionResponses { + todo!("S-C5: drop-session creation (invariants 26-31) — see SLICES.md") +} + +/// List the provisioning user's pending drops (owner; session auth). +#[endpoint(operation_id = "list_drop_inbox", tags("drops"), security(("bearer" = [])))] +pub async fn list_drop_inbox(_req: &mut Request, _depot: &mut Depot) -> InboxResponses { + todo!("S-C5: drop inbox listing — see SLICES.md") +} + +/// Adopt a pending drop: validate the adopter's `create` manifest and atomically promote +/// the inbox blob to an album asset (owner; session auth). +#[endpoint(operation_id = "adopt_drop", tags("drops"), security(("bearer" = [])))] +pub async fn adopt_drop( + _req: &mut Request, + _depot: &mut Depot, + _drop_id: PathParam, + _body: JsonBody, +) -> AdoptResponses { + todo!("S-C5: atomic inbox-to-album adoption (invariant 32) — see SLICES.md") +} + +/// Discard a pending drop; its bytes are GC'd and the owner's quota freed (owner; +/// session auth). +#[endpoint(operation_id = "discard_drop", tags("drops"), security(("bearer" = [])))] +pub async fn discard_drop( + _req: &mut Request, + _depot: &mut Depot, + _drop_id: PathParam, +) -> AdoptResponses { + todo!("S-C5: drop discard — see SLICES.md") +} diff --git a/capsule-api/media/src/routes/mod.rs b/capsule-api/media/src/routes/mod.rs index 5499c788..2a41fc69 100644 --- a/capsule-api/media/src/routes/mod.rs +++ b/capsule-api/media/src/routes/mod.rs @@ -3,8 +3,10 @@ use salvo::prelude::*; use crate::state::AppState; mod assets; +mod drops; mod exports; mod share; +mod verify; pub fn get_router(state: AppState) -> Router { Router::new() @@ -28,3 +30,31 @@ pub fn get_share_router(state: AppState) -> Router { .hoop(affix_state::inject(state)) .push(Router::with_path("").get(share::get_shared_content)) } + +/// Storage-verification router (mounted at /storage). Skeleton — slice `S-C3`. +pub fn get_storage_router(state: AppState) -> Router { + Router::new() + .hoop(affix_state::inject(state)) + .push(Router::with_path("verify").post(verify::storage_verify)) +} + +/// Guest drop-session router (mounted at /u; link-capability auth). Skeleton — `S-C5`. +/// Drop chunks reuse the upload protocol's `PATCH` mechanics under the session this +/// opens. +pub fn get_drop_link_router(state: AppState) -> Router { + Router::new() + .hoop(affix_state::inject(state)) + .push(Router::with_path("/drop").post(drops::create_drop_session)) +} + +/// Owner-facing drop inbox router (mounted at /drops; session auth). Skeleton — `S-C5`. +pub fn get_drops_router(state: AppState) -> Router { + Router::new() + .hoop(affix_state::inject(state)) + .get(drops::list_drop_inbox) + .push( + Router::with_path("") + .delete(drops::discard_drop) + .push(Router::with_path("adopt").post(drops::adopt_drop)), + ) +} diff --git a/capsule-api/media/src/routes/verify.rs b/capsule-api/media/src/routes/verify.rs new file mode 100644 index 00000000..54cb2f6b --- /dev/null +++ b/capsule-api/media/src/routes/verify.rs @@ -0,0 +1,127 @@ +//! Storage verification — `POST /storage/verify` (contract skeleton; slice `S-C3` in the +//! repo-root `SLICES.md`; SSoT: ). +//! +//! The key-free durability query: for each asset, confirm every declared blob is +//! **stored** (present in `blobs/` at its content address), **indexed** (a committed +//! `uploaded = true` row references it), and **retrievable** (refcount > 0, not mid-GC, +//! not quarantined). The verdict gates every destructive local cleanup on clients (the +//! verify-before-destroy rule; the client-side predicate is +//! `capsule_core::library::release_is_safe`). The request is a read and writes no state. + +use salvo::oapi::extract::JsonBody; +use salvo::prelude::*; +use serde::{Deserialize, Serialize}; + +/// One asset to verify: the exact blob hashes the client is relying on. +#[derive(Debug, Deserialize, ToSchema)] +#[allow(dead_code)] +pub(super) struct AssetVerifyRequest { + /// The asset id. + pub asset_id: String, + /// Content addresses (hex) of every blob the client relies on. + pub blob_hashes: Vec, +} + +/// The `POST /storage/verify` request body. +#[derive(Debug, Deserialize, ToSchema)] +#[allow(dead_code)] +pub(super) struct StorageVerifyRequest { + /// The assets to verify. + pub assets: Vec, + /// Re-read and re-hash blob bytes instead of trusting stat + index (rate-limited and + /// coalesced server-side). + #[serde(default)] + pub deep: bool, +} + +/// One blob's verdict. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct BlobVerdictResponse { + /// The declared content address (hex). + pub hash: String, + /// `original | metadata | derivative | provenance` (closed enum). + pub role: String, + /// Present in the blob store at its content address. + pub stored: bool, + /// Referenced by a committed, `uploaded = true` row. + pub indexed: bool, + /// Refcount > 0, not `collectable_since`, not quarantined. + pub retrievable: bool, +} + +/// One asset's verdict. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct StorageVerdictResponse { + /// The asset id. + pub asset_id: String, + /// All required blobs stored ∧ indexed ∧ retrievable. + pub durable: bool, + /// Per-blob detail, one entry per declared hash (a hash the server does not associate + /// with the asset comes back `stored=false, indexed=false` — never silently omitted). + pub blobs: Vec, + /// The server's trusted clock at verification (RFC 3339). + pub checked_at: String, +} + +/// The `POST /storage/verify` response body. +#[derive(Debug, Serialize, ToSchema)] +pub(super) struct StorageVerifyResponse { + /// One verdict per requested asset. + pub verdicts: Vec, +} + +#[derive(Serialize)] +struct ErrorResponse { + error: String, +} + +/// Possible responses for storage verification. +#[allow(dead_code)] +pub(super) enum StorageVerifyResponses { + /// Verdicts computed (a non-durable asset is still a 200 — the verdict carries it). + Ok(StorageVerifyResponse), + /// Structurally invalid request (unknown asset id shape, malformed hash). + BadRequest(String), +} + +#[async_trait] +impl Writer for StorageVerifyResponses { + async fn write(mut self, req: &mut Request, depot: &mut Depot, res: &mut Response) { + match self { + Self::Ok(data) => { + res.status_code(StatusCode::OK); + Json(data).write(req, depot, res).await; + } + Self::BadRequest(msg) => { + res.status_code(StatusCode::BAD_REQUEST); + res.render(Json(ErrorResponse { error: msg })); + } + } + } +} + +impl EndpointOutRegister for StorageVerifyResponses { + fn register(components: &mut salvo::oapi::Components, operation: &mut salvo::oapi::Operation) { + operation.responses.insert( + String::from("200"), + salvo::oapi::Response::new("Per-asset durability verdicts").add_content( + "application/json", + salvo::oapi::Content::new(StorageVerifyResponse::to_schema(components)), + ), + ); + operation.responses.insert( + String::from("400"), + salvo::oapi::Response::new("Structurally invalid request"), + ); + } +} + +/// Batch-confirm that assets' blobs are stored, indexed, and retrievable on this server. +#[endpoint(operation_id = "storage_verify", tags("storage"), security(("bearer" = [])))] +pub async fn storage_verify( + _req: &mut Request, + _depot: &mut Depot, + _body: JsonBody, +) -> StorageVerifyResponses { + todo!("S-C3: storage-verification endpoint — see SLICES.md") +} diff --git a/capsule-api/src/lib.rs b/capsule-api/src/lib.rs index faed6190..db58c6f2 100644 --- a/capsule-api/src/lib.rs +++ b/capsule-api/src/lib.rs @@ -14,6 +14,8 @@ pub mod tags { pub const UPLOAD: &str = "upload"; pub const MEDIA: &str = "media"; pub const SHARE: &str = "share"; + pub const STORAGE: &str = "storage"; + pub const DROPS: &str = "drops"; pub const LIBRARY: &str = "library"; pub const SYNC: &str = "sync"; } @@ -34,6 +36,8 @@ pub fn create_openapi_spec() -> OpenApi { Tag::new(tags::UPLOAD).description("Capsule Upload API"), Tag::new(tags::MEDIA).description("Capsule Media Serving API"), Tag::new(tags::SHARE).description("Capsule Public Share API"), + Tag::new(tags::STORAGE).description("Capsule Storage Verification API"), + Tag::new(tags::DROPS).description("Capsule Web-Upload Drops API"), Tag::new(tags::LIBRARY).description("Capsule Library API (GraphQL)"), Tag::new(tags::SYNC).description("Capsule Sync API (gRPC)"), ]) @@ -80,6 +84,20 @@ pub async fn create_router(conn: DatabaseConnection, env: &Environment) -> Resul .push( Router::with_path("s") .push(media::get_share_router(conn.clone(), &env.server).await?), + ) + // Key-free durability verdicts (skeleton — slice S-C3 in SLICES.md). + .push( + Router::with_path("storage") + .push(media::get_storage_router(conn.clone(), &env.server).await?), + ) + // Guest drop sessions + owner inbox (skeleton — slice S-C5 in SLICES.md). + .push( + Router::with_path("u") + .push(media::get_drop_link_router(conn.clone(), &env.server).await?), + ) + .push( + Router::with_path("drops") + .push(media::get_drops_router(conn.clone(), &env.server).await?), ); } // TODO: Verify this GRPc route works diff --git a/capsule-api/upload/src/envelope.rs b/capsule-api/upload/src/envelope.rs new file mode 100644 index 00000000..ba171bd2 --- /dev/null +++ b/capsule-api/upload/src/envelope.rs @@ -0,0 +1,42 @@ +//! The refuse-by-default envelope gate — the seam wiring `capsule_core::validation` into +//! every write path (contract skeleton; slice `S-C1` in the repo-root `SLICES.md`; SSoT: +//! ). +//! +//! The pure, key-free invariants (the protocol handshake, the manifest-envelope checks, +//! the idempotency keys) are implemented and exhaustively tested in +//! `capsule_core::validation` — [`protocol_gate`] and [`check_manifest_envelope`] are +//! ready to consume. What this middleware adds when `S-C1` lands is the transport glue: +//! read the `X-Capsule-*` negotiation headers, run the gate before any state is written, +//! and map each rejection to its HTTP status plus `error.*` code (see the design docs' +//! API-surfaces rejection mapping). Until then it is an unmounted stub. +//! +//! [`protocol_gate`]: capsule_core::validation::protocol_gate +//! [`check_manifest_envelope`]: capsule_core::validation::check_manifest_envelope + +use salvo::prelude::*; + +/// Salvo middleware running the fail-closed protocol handshake (invariant 1 and the +/// universal-headers rules) ahead of every write handler it is hooped onto. +#[allow(dead_code)] +pub(crate) struct EnvelopeGate { + /// The lowest protocol version this server accepts (`X-Capsule-Protocol-Min`). + pub min_protocol: String, + /// The highest protocol version this server accepts (`X-Capsule-Protocol-Max`). + pub max_protocol: String, +} + +#[async_trait] +impl Handler for EnvelopeGate { + async fn handle( + &self, + _req: &mut Request, + _depot: &mut Depot, + _res: &mut Response, + _ctrl: &mut FlowCtrl, + ) { + // S-C1 wires capsule_core::validation::protocol_gate here: parse + // X-Capsule-Protocol, reject outside [min, max] with 426 + the error.* code and + // the advertised range, and short-circuit before any handler writes state. + todo!("S-C1: envelope gate wiring — see SLICES.md") + } +} diff --git a/capsule-api/upload/src/lib.rs b/capsule-api/upload/src/lib.rs index 4892f68c..c8d43ea3 100644 --- a/capsule-api/upload/src/lib.rs +++ b/capsule-api/upload/src/lib.rs @@ -10,6 +10,7 @@ use crate::config::validate_config; use crate::state::AppState; mod config; +mod envelope; mod error; mod models; mod routes; From e1fcfb2c35aa33109c0c6e78cfb93329c633e5db Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:51:04 -0400 Subject: [PATCH 11/58] feat(api): add the key-free sync proto contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gRPC half of the API-surface decision: a new capsule.sync.v1 package (slice S-C2 in SLICES.md) carrying the sync feed the download-sync design doc now specifies, beside the frozen legacy plaintext service. - proto/capsule/sync/v1/sync.proto — unary-paged Sync(cursor, page_size) with per-album strictly-increasing sync_seq, the closed ChangeKind enum, and per-role blob references. The load-bearing invariant is written into the proto: the signed AssetManifest travels as OPAQUE canonical-CBOR bytes, never re-modeled as proto fields (re-encoding would detach it from its signatures), and blob bytes never ride this service — ranged fetches stay REST - SyncFeedService stub (Status::unimplemented, the crate's accepted form) mounted at its explicit service path BEFORE the legacy catch-all so it wins matching - GrpcHandler genericized over the tonic service so both services share the one salvo bridge (behavior unchanged) - the legacy photolibrary.metadata.v1 service and proto carry the LEGACY-PLAINTEXT freeze marker pointing at retirement slice S-G2 --- capsule-api/sync/build.rs | 8 +- .../sync/proto/capsule/sync/v1/sync.proto | 85 +++++++++++++++++++ capsule-api/sync/src/lib.rs | 71 ++++++++++++++-- 3 files changed, 156 insertions(+), 8 deletions(-) create mode 100644 capsule-api/sync/proto/capsule/sync/v1/sync.proto diff --git a/capsule-api/sync/build.rs b/capsule-api/sync/build.rs index 8b6ef3d8..a1aee7c8 100644 --- a/capsule-api/sync/build.rs +++ b/capsule-api/sync/build.rs @@ -1,7 +1,13 @@ fn main() { tonic_prost_build::configure() .compile_protos( - &["proto/photolibrary/metadata/v1/metadata.proto"], + &[ + // LEGACY-PLAINTEXT (frozen): SLICES.md S-G2 — the pre-E2EE metadata + // service, retiring once the key-free feed below reaches parity. + "proto/photolibrary/metadata/v1/metadata.proto", + // The key-free sync feed contract (SLICES.md S-C2). + "proto/capsule/sync/v1/sync.proto", + ], &["proto"], ) .expect("failed to compile protobuf definitions"); diff --git a/capsule-api/sync/proto/capsule/sync/v1/sync.proto b/capsule-api/sync/proto/capsule/sync/v1/sync.proto new file mode 100644 index 00000000..7f938469 --- /dev/null +++ b/capsule-api/sync/proto/capsule/sync/v1/sync.proto @@ -0,0 +1,85 @@ +// The key-free sync feed — contract skeleton (slice S-C2 in the repo-root SLICES.md; +// SSoT: https://docs/design/import/download-sync/). +// +// INVARIANT — the signed AssetManifest travels as OPAQUE canonical-CBOR bytes +// (`manifest_cbor`), never re-modeled as proto fields: verification re-serializes the +// canonical CBOR, so a re-encoding would detach the bytes from their signatures. A +// tombstone is the signed delete manifest itself. Blob BYTES never ride this service — +// ranged fetches stay on the REST /blob/{hash} endpoint (api-surfaces design doc). +// +// Negotiation rides call metadata (x-capsule-protocol, x-capsule-crypto-suite, ...) and +// rejections map onto gRPC status codes with the error.* catalog code as the +// cross-transport discriminator — both owned by the api-surfaces design doc. + +syntax = "proto3"; + +package capsule.sync.v1; + +service SyncService { + // A page of changes after `cursor`; monotonic and resumable. Unary-paged: the cursor + // is the resumption token, and re-issuing the same cursor returns the same page + // (the idempotency key of the federation-pull surface too). + rpc Sync(SyncRequest) returns (SyncResponse); +} + +message SyncRequest { + // Opaque, server-MAC'd resumption cursor; empty on first sync. A forged or mutated + // cursor is rejected (invariant 22); the client separately enforces per-album + // sync_seq monotonicity against its high-water mark. + bytes cursor = 1; + // Maximum entries in the returned page (the server may clamp). + uint32 page_size = 2; +} + +message SyncResponse { + repeated SyncEntry entries = 1; + // Cursor for the next page. + bytes next_cursor = 2; +} + +// What changed. Closed enum per protocol_version — an unknown value is a structural +// error on the receiving side, never a "future value to ignore". +enum ChangeKind { + CHANGE_KIND_UNSPECIFIED = 0; + CHANGE_KIND_CREATED = 1; + CHANGE_KIND_METADATA_UPDATED = 2; + CHANGE_KIND_DELETED = 3; +} + +message SyncEntry { + // The album this entry belongs to (16-byte UUID). + bytes album_id = 1; + // Per-album, strictly-increasing sequence — the client's anti-rewind high-water mark. + uint64 sync_seq = 2; + // The album pin this entry conforms to (YYYY-MM-DD). + string protocol_version = 3; + // What changed. + ChangeKind kind = 4; + // The asset id (16-byte UUID). + bytes asset_id = 5; + // The signed AssetManifest as opaque canonical CBOR (see the header invariant). + bytes manifest_cbor = 6; + // The encrypted metadata blob (wire format owned by the encryption design doc); + // empty for deletes — the tombstone is the manifest itself. + bytes metadata_blob = 7; + // Content addresses of the asset's blobs by role — never blob bytes. + BlobManifest blobs = 8; +} + +message BlobManifest { + // The original ciphertext blob. + BlobRef original = 1; + // Derivative blobs (thumbnails, previews, embeddings). + repeated BlobRef derivatives = 2; +} + +message BlobRef { + // Ciphertext content address. + bytes ciphertext_hash = 1; + // original | metadata | derivative | provenance (closed enum). + string role = 2; + // MIME/format string, for derivatives. + string format = 3; + // Ciphertext size in bytes. + uint64 size = 4; +} diff --git a/capsule-api/sync/src/lib.rs b/capsule-api/sync/src/lib.rs index 79f5c3d6..24e0e042 100644 --- a/capsule-api/sync/src/lib.rs +++ b/capsule-api/sync/src/lib.rs @@ -27,6 +27,9 @@ use tower::Service; pub mod config; pub mod proto { + // LEGACY-PLAINTEXT (frozen): SLICES.md S-G2 — the pre-E2EE metadata service. It + // models plaintext photos/albums/tags the key-free server never sees; it is kept + // compiling but frozen, and retires once `capsule.sync.v1` reaches parity. pub mod photolibrary { pub mod metadata { pub mod v1 { @@ -34,8 +37,39 @@ pub mod proto { } } } + + /// The key-free sync feed (contract skeleton — slice `S-C2` in the repo-root + /// `SLICES.md`; SSoT: ). + pub mod capsule { + pub mod sync { + pub mod v1 { + tonic::include_proto!("capsule.sync.v1"); + } + } + } +} + +/// The key-free sync feed service (skeleton — every RPC is `unimplemented` until slice +/// `S-C2`). Entries carry the signed manifest as opaque canonical CBOR plus the small +/// encrypted metadata blob and per-role blob references; blob bytes stay on REST. +#[derive(Default, Debug, Clone)] +pub struct SyncFeedService { + // S-C2 injects the DB connection (sync_seq mint + cursor MAC key) here. +} + +#[tonic::async_trait] +impl proto::capsule::sync::v1::sync_service_server::SyncService for SyncFeedService { + async fn sync( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented( + "S-C2: key-free sync feed — see SLICES.md", + )) + } } +// LEGACY-PLAINTEXT (frozen): SLICES.md S-G2 — see the proto module note above. #[derive(Default, Debug, Clone)] pub struct CapsuleMetadataService { // Inject DB or Config here if needed @@ -154,20 +188,32 @@ impl PhotoLibraryMetadataService for CapsuleMetadataService { } } -/// A Salvo handler that wraps the gRPC service +/// A Salvo handler that wraps a tonic gRPC service (the legacy metadata service and the +/// key-free `SyncService` both ride it). #[derive(Clone)] -pub struct GrpcHandler { - service: PhotoLibraryMetadataServiceServer, +pub struct GrpcHandler { + service: S, } -impl GrpcHandler { - pub fn new(service: PhotoLibraryMetadataServiceServer) -> Self { +impl GrpcHandler { + pub fn new(service: S) -> Self { Self { service } } } #[async_trait] -impl Handler for GrpcHandler { +impl Handler for GrpcHandler +where + S: Service< + hyper::Request, + Response = hyper::Response, + Error = Infallible, + > + Clone + + Send + + Sync + + 'static, + S::Future: Send, +{ async fn handle( &self, req: &mut salvo::Request, @@ -219,12 +265,23 @@ pub async fn get_router>( _conn: DatabaseConnection, _config: C, ) -> Result { + // The key-free sync feed (skeleton — SLICES.md S-C2). Mounted at its explicit + // service path BEFORE the legacy catch-all so it wins matching. + let sync_feed = GrpcHandler::new( + proto::capsule::sync::v1::sync_service_server::SyncServiceServer::new( + SyncFeedService::default(), + ), + ); + + // LEGACY-PLAINTEXT (frozen): SLICES.md S-G2. let service = CapsuleMetadataService::default(); let grpc_service = PhotoLibraryMetadataServiceServer::new(service); let handler = GrpcHandler::new(grpc_service); // gRPC routes need to match the full path including the service name - let router = Router::new().push(Router::with_path("<**rest>").goal(handler)); + let router = Router::new() + .push(Router::with_path("capsule.sync.v1.SyncService/<**rest>").goal(sync_feed)) + .push(Router::with_path("<**rest>").goal(handler)); Ok(router) } From b7f347fb25bd80a0d6aa68be60d94aff64856753 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:52:51 -0400 Subject: [PATCH 12/58] chore: remove dead API artifacts and mark legacy freeze points - drop the unused axum workspace dependency (nothing in the workspace references it; the server is salvo) - remove the stale re-enable-when-production-ready comment on capsule-api-upload, which has shipped in the full feature set for some time - correct capsule-core-ffi's only-FFI-aware-crate claim: capsule-core exports its own uniffi surface on a different uniffi version, and the consolidation is tracked as slice S-F1 - convert the floating TODOs on the plaintext-era media/sync surfaces into greppable LEGACY-PLAINTEXT freeze markers pointing at their SLICES.md retirement/conformance slices --- Cargo.toml | 1 - capsule-api/Cargo.toml | 1 - capsule-api/media/src/error.rs | 4 ++-- capsule-api/media/src/routes/assets.rs | 3 ++- capsule-api/src/lib.rs | 2 +- capsule-core-ffi/src/lib.rs | 10 ++++++---- 6 files changed, 11 insertions(+), 10 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index 0f17d12d..23df4b99 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -49,7 +49,6 @@ publish = false [workspace.dependencies] argon2 = "0.5.3" -axum = { version = "0.8.9", features = ["http2", "macros"] } base64 = "0.22.1" bb8 = "0.9.1" bb8-redis = "0.26.0" diff --git a/capsule-api/Cargo.toml b/capsule-api/Cargo.toml index cad0fe39..a78934fa 100644 --- a/capsule-api/Cargo.toml +++ b/capsule-api/Cargo.toml @@ -9,7 +9,6 @@ default-run = "capsule-api" capsule-api-auth = { path = "./auth", optional = true } capsule-api-library = { path = "./library", optional = true } capsule-api-media = { path = "./media", optional = true } -# TODO: Re-enable when upload is production-ready capsule-api-upload = { path = "./upload", optional = true } capsule-api-sync = { path = "./sync", optional = true } capsule-api-environment = { path = "./environment" } diff --git a/capsule-api/media/src/error.rs b/capsule-api/media/src/error.rs index be1746d9..be676dc8 100644 --- a/capsule-api/media/src/error.rs +++ b/capsule-api/media/src/error.rs @@ -1,8 +1,8 @@ use salvo::prelude::*; use thiserror::Error; -// TODO: Nothing is using this? - +// Currently unconsumed; the key-free media slices (S-C3/S-C5/S-C10 in SLICES.md) adopt +// this as the crate error type and attach the error.* codes per the i18n contract. #[allow(dead_code)] #[derive(Debug, Error)] pub(crate) enum MediaError { diff --git a/capsule-api/media/src/routes/assets.rs b/capsule-api/media/src/routes/assets.rs index 0572c0fb..9793ceed 100644 --- a/capsule-api/media/src/routes/assets.rs +++ b/capsule-api/media/src/routes/assets.rs @@ -15,7 +15,8 @@ use uuid::Uuid; use crate::state::AppState; -// TODO: authorization via access token for asset routes +// LEGACY-PLAINTEXT (frozen): SLICES.md S-G3 — plaintext-era serving assumptions; the +// key-free ranged ciphertext serving (and per-route access-token auth) is slice S-C10. // ============================================================================ // Request/Response Types diff --git a/capsule-api/src/lib.rs b/capsule-api/src/lib.rs index db58c6f2..0671bfab 100644 --- a/capsule-api/src/lib.rs +++ b/capsule-api/src/lib.rs @@ -100,7 +100,7 @@ pub async fn create_router(conn: DatabaseConnection, env: &Environment) -> Resul .push(media::get_drops_router(conn.clone(), &env.server).await?), ); } - // TODO: Verify this GRPc route works + // The gRPC-over-salvo bridge is exercised end-to-end by slice S-C2 (SLICES.md). #[cfg(feature = "sync")] { // gRPC sync routes diff --git a/capsule-core-ffi/src/lib.rs b/capsule-core-ffi/src/lib.rs index 3e2d5f7d..8f320131 100644 --- a/capsule-core-ffi/src/lib.rs +++ b/capsule-core-ffi/src/lib.rs @@ -1,10 +1,12 @@ //! UniFFI bindings exposing the `capsule-core` SQLite catalog and CBOR sidecar //! to Swift (and, in future, other UniFFI targets such as Android/Kotlin). //! -//! This is the **only** FFI-aware crate in the workspace. `capsule-core` stays a -//! pure, portable library; everything platform-specific (filesystem layout, -//! file I/O, PhotoKit, hashing) lives in the Swift client. The types defined -//! here form the explicit Rust ↔ Swift contract: +//! One of **two** uniffi surfaces in the workspace — `capsule-core`'s `ffi` feature +//! exports the crypto `FfiWorkspace` + `HardwareSigner` foreign trait separately, on a +//! different uniffi version; consolidating the two is slice `S-F1` in the repo-root +//! `SLICES.md`. Everything platform-specific (filesystem layout, file I/O, PhotoKit, +//! hashing) lives in the Swift client. The types defined here form the explicit +//! Rust ↔ Swift contract: //! //! - [`Catalog`] — a thread-safe handle over the SQLite catalog. //! - [`AssetRecord`], [`AssetStackRecord`], [`StackMemberRecord`], From 8c1bd096074a7b6889f1236e6300d71119b3f2de Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 21:57:19 -0400 Subject: [PATCH 13/58] docs: add SLICES.md, superseding DEFERRED.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SLICES.md is the executable index of everything the design docs specify that is not yet implemented, decomposed into ~45 independently shippable slices across nine lanes (core-crypto, media/import, server, sdk/clients, federation/sharing, platform/FFI, legacy-retirement, ML, blocked-external, design follow-ups), each with a stable ID, its contract anchors, the in-tree seam it binds to, hard dependency edges, command-checkable done criteria, and a validation tier — plus the dependency DAG and a Baseline section recording what already ships. The slice IDs are load-bearing: every skeleton, #[ignore]d contract test, and LEGACY-PLAINTEXT freeze marker added on this branch names its slice, so rg S-xx finds a slice's whole footprint. DEFERRED.md is deleted — its implemented-baseline content moved into the Baseline section and every deferred item became a slice — and its eleven inbound references (core doc comments, harness READMEs, the TPM note) now point at SLICES.md. --- DEFERRED.md | 169 ----- SLICES.md | 623 ++++++++++++++++++ capsule-core-kotlin/README.md | 2 +- .../capsule/hardware/StrongBoxSigner.kt | 2 +- capsule-core-swift/README.md | 2 +- .../CapsuleHardware/SecureEnclaveSigner.swift | 2 +- capsule-core/README-tpm.md | 2 +- capsule-core/src/crypto/authority/mod.rs | 2 +- capsule-core/src/crypto/keys/keystore.rs | 2 +- capsule-core/src/crypto/keys/signer.rs | 2 +- capsule-core/src/crypto/keys/tpm.rs | 2 +- capsule-core/src/ffi.rs | 2 +- capsule-core/src/lifecycle.rs | 4 +- 13 files changed, 635 insertions(+), 181 deletions(-) delete mode 100644 DEFERRED.md create mode 100644 SLICES.md diff --git a/DEFERRED.md b/DEFERRED.md deleted file mode 100644 index e42f60c0..00000000 --- a/DEFERRED.md +++ /dev/null @@ -1,169 +0,0 @@ -# Deferred Work - -This file records what the **core data-plane implementation** (the `capsule-core` -cryptographic core + offline lifecycle + `capsule demo`) intentionally left for later, why, -and the seam that was left in place so it can drop in without reworking what exists. It -complements the design docs in `capsule-docs/src/content/docs/design/`. - -## What is implemented and validated (offline, real crypto) - -`capsule-core` implements and exhaustively unit-tests the full offline data plane: - -- **Canonical CBOR** (RFC 8949 §4.2) — the byte-identity contract for every signature/hash. -- **Crypto primitives** — SHA-256 (streaming), HKDF-SHA512, Argon2id, AES-256-GCM (STREAM + - standalone metadata-blob), **hybrid Ed25519 + ML-DSA-65** signatures (both halves required), - **X-Wing (X25519 + ML-KEM-768)** hybrid DEK (known-answer-validated against - `draft-connolly-cfrg-xwing-kem`). -- **Key hierarchy** — master key, default-album-id derivation, multi-epoch AMKs (offline - rotation) + per-file/blob keys, software keystore (account ↔ encrypted `AccountFile`), signed - device directory. -- **Encryption** — STREAM asset encryption with independent ranged-chunk decryption; - exact metadata-blob wire format. -- **Provenance** — signed `AssetManifest`/`DerivativeManifest`, append-only hash-chained - provenance, and the single **`verify_asset`** chokepoint (Accept / TerminalReject / Pending) - with an exhaustive negative-case suite. -- **Validation invariants** — the key-less protocol handshake + structural envelope checks + - idempotency keys. -- **CRDT metadata + Sidecar v1** — OR-set tags, LWW caption/rating with superseded log, - monotonic add-id counter; signed `SidecarV1` (schema as CBOR field 0); privacy-on-export. -- **Backup** — deterministic signed tar artifact, AMK ledger, master-key escrow, Shamir - 2-of-3, and dry-run/commit restore with chain reconciliation. -- **Lifecycle `Workspace`** — ties it together and writes through to the queryable - `library.sqlite` index; showcased end-to-end by `capsule demo`. - -## Deferred — with the seam in place - -### Real MLS / OpenMLS group state - -- **Why:** the design's MLS ciphersuite (`MLS_256_XWING_CHACHA20POLY1305_SHA256_Ed25519`, - `0x004D`) exists in `openmls` only via a C (`libcrux`) backend on a non-final IETF draft, - with no IANA codepoint and no RustCrypto PQ backend yet (openmls#1940). -- **Seam:** `capsule_core::crypto::authority::AlbumAuthority` is the trait `verify_asset` - consumes (epoch ceiling, per-epoch write-tier pubkey, AMK presence, admin-chain validity). - `ReferenceAuthority` (an admin-signed epoch ledger) stands in for live MLS and is honored - only via `&dyn AlbumAuthority`, so an `OpenMlsAuthority` drops in unchanged. -- **Now implemented offline:** **multi-epoch rotation** via `ReferenceAuthority` — - `Workspace::rotate_epoch` mints AMK_v{n+1} + a fresh write-tier key and admin-attests the new - epoch (assets imported before a rotation stay verifiable under their original epoch), plus a - serializable, admin-signed `SignedEpochLedger` (`to_ledger`/`from_ledger`, whose admin chain is - re-verified on reload; the local-only AMK-presence flag is restored out-of-band). -- **Still deferred with OpenMLS:** membership add/remove, the `Welcome`/history-delivery flow, - and the album upgrade ceremony — these need live MLS group state, not just the epoch ledger. - -### Hardware-bound key storage - -- **Seam + software fallback + contract — now implemented.** The device signing key (DSK) is - consumed through a `capsule_core::crypto::keys::Signer` trait, so the in-memory - `HybridSigningKey` (software, default) and a hardware-backed key are interchangeable at every - signing site. `HardwareSigner` is a uniffi **foreign trait** (Secure Enclave / StrongBox / TPM - implement it natively under the `ffi` feature); `HardwareBackedSigner` composes its - hardware-produced **Ed25519** half with a software-sealed **ML-DSA-65** half into the hybrid - signature (no secure element holds PQ keys). `Workspace::create_with_hardware_signer` / - `FfiWorkspace.createWithHardwareSigner` build a workspace whose directory + manifests are - hardware-signed; the in-process round-trip + non-exportability contract (`keys.md` Validation) - runs in CI against a mock element. -- **Reference adapters + standalone harnesses — now implemented.** Every backend implements the - `HardwareSigner` contract as a runnable, locally-testable example (the prose `HARDWARE_KEYS.md` - guide is gone — the code is the example): a software fallback - (`capsule_core::crypto::keys::SoftwareSigner`, smoke-tested in CI on Linux), a desktop TPM 2.0 - reference (`crypto::keys::tpm`, behind the `tpm` feature, via `tss-esapi`), and Secure Enclave / - StrongBox adapters in standalone harness packages (`capsule-core-swift`, `capsule-core-kotlin`) - that link the compiled core and run a per-language smoke test (see each package's README; the - Swift `swift test` runs the real Secure Enclave on Apple-Silicon Macs). -- **Still deferred (per-platform glue):** the three hardware backends (Secure Enclave, StrongBox, - TPM) all expose ECDSA-P256, not Ed25519, so they need the **P-256 hybrid-DSK variant** before - they compose into the device key — only the software backend integrates end-to-end today; wiring - the generated bindings + `cdylib`/`staticlib` into the real Xcode / Gradle apps; on-device CI; - the Windows TPM (TBS) path; and hardware binding of the device **encryption** key (DEK). - -### Networked server/client - -- All transport is out of scope here: the HTTP/TUS upload server, GraphQL resolvers, the - `/sync` feed, federation, peering, and the `capsule-sdk` network client. The **pure** - refuse-by-default validation invariants those paths need are implemented in - `capsule_core::validation` and ready to wire into `capsule-api`. -- The **storage-verification endpoint (`POST /storage/verify`) + verify-before-destroy gate** - is deferred with the rest of the networked surface. The endpoint returns a key-free - durability verdict (stored = blob-store `stat`; indexed = committed `uploaded=true` row; - retrievable = refcount > 0 and not `collectable_since`/quarantined/dangling), and the - standing client rule is that no destructive local cleanup of irreplaceable bytes proceeds - without a `durable` verdict. **Seam:** the verdict is pure index+filesystem inspection that - drops into `capsule-api-media`; the verify-before-destroy predicate is a pure `capsule-core` - function the cache-eviction sweep, move-import, and streaming import all consume; the - offline core's `verify_asset` already covers the complementary crypto-validity half of the - gate. -- **Import-upload streaming mode + free-space probe** is deferred with the networked path. For - storage-constrained devices, import runs a sliding-window import→upload→verify→release loop - that bounds peak local disk to the in-flight window, auto-detected from a free-space probe and - halting (no further source admitted into the library) if the server connection drops. **Seam:** - the planner already emits `total_size`; what's added is `capsule_core::library::available_bytes()` - (a thin `statvfs`/`GetDiskFreeSpaceEx` wrapper), the `streaming_recommended` plan flag, and an - executor drive mode that interleaves the existing offline import, the deferred upload client, and - the storage-verify gate above — with no change to the upload wire protocol. -- **Web upload (guest drops)** is deferred with the rest of the networked surface. A browser/WASM - guest, holding only a provisioned **upload link**, seals each asset under a fresh random key - encapsulated to a link-scoped **Drop Key** and uploads it to a per-user staging inbox charged to - the provisioning user's quota; the bytes become a library asset only when one of that user's - trusted clients reviews and **adopts** the drop, rewrapping the guest's key under the album AMK - (`key_mode = wrapped` + `wrapped_file_key`) **without re-uploading**. Designed in the - `web-upload` design doc. **Seam:** every primitive already exists offline — X-Wing - encapsulation, AES-256-GCM-STREAM, content-addressing — so web upload is purely additive: a - `capsule_core::drop` module (sealing + link issuance + adoption rewrap), a `capsule-api-media` - drop store/inbox + the atomic inbox→album promotion, a drop upload protocol that reuses the - existing upload-protocol chunk/finalization mechanics, and the `capsule-web` browser client. The - offline core's `verify_asset`, append-only provenance chain, STREAM construction, and the - sandboxed-decoder rule are reused unchanged; the only crypto-core addition is the **wrapped - file-key mode** — a stored, AMK-wrapped per-file key for externally-authored bytes that cannot be - re-derived from the AMK — landing in `crypto::encryption` key derivation (the `asset-keywrap/v1` - label) and the `crypto::provenance` `AssetManifest` (`key_mode`/`wrapped_file_key`), with - authorization unchanged (the adopter's write-tier signature remains the sole authority). -- The **adaptive cache-eviction policy** (bounded budget, LRU-by-last-access retention of - recently-viewed blobs, tier-ordered eviction original → preview → thumbnail, pinned and - device-owned originals exempt) is **now implemented** (issue #23): the - `cached_representations` table + last-access tracking live in `capsule-core::db` and the sweep - `capsule_core::library::cache_sweep` deletes evicted cache files (never the canonical `media/` - files or the index). The byte budget is a plain parameter, so `capsule-sdk` connection-class - detection drives it unchanged. Still deferred upstream: that connection-class budget detection - and the wider networked server/client (HTTP/TUS, GraphQL, `/sync`, federation, peering). - -### ML / AI - -- Embeddings, `sqlite-vec` vector search, the model registry, semantic/face features, and - moderation are deferred (explicitly out of scope). The sidecar reserves `tags_ai` - (separate OR-set) and the manifest reserves `model_id`/`model_version` for them. - -### Other - -- **Rewrite re-keying + metadata↔manifest binding (contract tightened).** The - offline crypto core already draws a fresh nonce per asset/metadata encryption - and seals every metadata blob fresh. The design contract now additionally - requires: (a) folding that fresh nonce into the key-derivation salt - (`salt = file_id || nonce_prefix` for assets, `blob_id || nonce` for metadata) - so a same-`file_id`/same-epoch `replace` and a constant-`blob_id` - `metadata-update` re-roll the **key** as well as the nonce — making any - `(key, nonce)` reuse structurally impossible; and (b) a new server-visible - `metadata_blob_hash` field on the signed `AssetManifest` plus a client - round-trip-equivalence check (decrypt of the metadata blob must equal the - signed local sidecar) and the matching no-key server check (validation - invariant 25). **Seam:** the key-salt change is local to `crypto::encryption` - key derivation; the manifest field, the round-trip check in `verify_asset`, and - the envelope hash-match check land in `crypto::provenance` / - `crypto::verify_asset` and the `capsule-api` envelope validator when the - networked write path is wired. -- Thumbnail/LQIP generation beyond `capsule-media`'s existing utilities. -- Fusing the crypto data plane into the **existing plaintext import executor** - (`capsule_core::import::executor`) — **partially done.** `capsule_core::lifecycle::Workspace` - now writes through to the shared `library.sqlite` index: every import / metadata edit / - soft-delete upserts the queryable `assets` row (+ user tags) and records a device-owned - `original` cache representation, so crypto-imported assets are timeline-queryable and feed the - Phase-3 cache sweep. Dedup against `assets` is consequently **global** across both import - paths. **Still deferred:** the legacy `import::executor` keeps writing the unsigned - `AssetSidecar`; replacing it with the signed `SidecarV1` + manifest + provenance path needs - the deferred thumbnail/LQIP (media) generation, so the full executor rewrite is a follow-up. - -## How to see it working - -```bash -cargo test --workspace --exclude capsule-sdk # full unit + e2e test surface -cargo run -p capsule-cli -- demo --workdir /tmp/capsule-demo # narrated end-to-end showcase -``` diff --git a/SLICES.md b/SLICES.md new file mode 100644 index 00000000..e1df8c68 --- /dev/null +++ b/SLICES.md @@ -0,0 +1,623 @@ +# Implementation Slices + +This file is the executable index of everything the [design docs](capsule-docs/src/content/docs/design/) +specify that is **not yet implemented**, decomposed into independently shippable +**slices**. It supersedes `DEFERRED.md`: the baseline below records what already ships, +and every formerly-deferred item is now a slice with a bound contract. + +**How to use this file.** + +- Every slice has a stable ID (`S-A1`, `S-C3`, …). Code skeletons, `#[ignore]`d contract + tests, and `LEGACY-PLAINTEXT (frozen)` markers reference these IDs; `rg S-C3` finds a + slice's entire footprint. +- A slice references other slices **only by ID + contract anchor**, never by their + internals. "Depends on" edges are hard (the contract consumed must exist); everything + else can proceed in parallel. +- **Done when** must be checkable by running named commands. A slice that flips its + named `#[ignore]`d tests, keeps `mise run check-rust` (and the other relevant `mise run + check-*` gates) green, and satisfies its owner doc's Validation bullets is done. +- Sizes: **S** ≤ ½ day, **M** ≤ 2 days, **L** ≥ 3 days — for slicing sanity, not + estimation. An L slice that can split should split. +- When a slice lands, update its row to `done` (leave the block for the record). When a + new gap is found, add a slice — never a floating TODO. + +## Baseline — already implemented and validated + +The **offline crypto data plane** in `capsule-core` (see `cargo run -p capsule-cli -- +demo`): canonical CBOR (RFC 8949 §4.2, cross-language conformance gate); the primitives +inventory (SHA-256, HKDF-SHA512, Argon2id, AES-256-GCM STREAM + metadata blobs, hybrid +Ed25519+ML-DSA-65, X-Wing KAT-validated); the key hierarchy with multi-epoch AMK +rotation, software keystore, and signed device directory; the `Signer`/`HardwareSigner` +seams with software + TPM reference adapters (Secure Enclave / StrongBox adapters in +`capsule-core-swift`/`-kotlin`); signed manifests + append-only provenance + the +exhaustive `verify_asset` chokepoint behind the `AlbumAuthority` seam +(`ReferenceAuthority` epoch ledger); the pure key-free validation invariants +(`capsule_core::validation`); CRDT metadata + signed `SidecarV1` + privacy-on-export; +deterministic signed backup (tar, AMK ledger, escrow, Shamir 2-of-3, dry-run/commit +restore); the lifecycle `Workspace` writing through to `library.sqlite`; the cache +eviction sweep (issue #23); the offline import pipeline (scan/plan/execute — still +writing the legacy unsigned sidecar, see S-B2). + +On the server: `capsule-api-auth` (sessions, passkeys, TOTP, OIDC) is real and +testcontainer-tested; the custom chunked upload server exists but is unhardened (S-C1); +everything else networked is skeleton or legacy (below). + +**Contract skeletons in-tree** (this PR): manifest `key_mode`/`wrapped_file_key`/ +`metadata_blob_hash` fields + `asset-keywrap/v1` label; `crypto::keys::p256`; +`capsule_core::{drop, sharing}`; `library::{space, storage_verify}` (with the pure +`streaming_recommended` and `release_is_safe` predicates implemented); +`capsule-api-media::{verify, drops}` + `capsule-api-auth::devices` route stubs; +`capsule-api-upload::envelope` gate stub; the `capsule.sync.v1` proto + `SyncFeedService` +stub. Each names its slice. + +## Slice index + +| ID | Slice | Lane | Depends on | Size | Status | +| ----- | ---------------------------------------------------- | --------------- | ---------------- | ---- | ------- | +| S-A1 | Wrapped file-key mode (seal/unseal + verify) | core-crypto | — | M | ready | +| S-A2 | Re-key salt fold | core-crypto | — | S | ready | +| S-A3 | Metadata↔manifest binding (invariant 25, both sides) | core-crypto | S-A1 | M | ready | +| S-A4 | P-256 hybrid DSK variant | core-crypto | — | L | ready | +| S-A5 | Share-link crypto (`capsule_core::sharing`) | core-crypto | — | M | ready | +| S-A6 | Drop crypto (`capsule_core::drop`, incl. WASM build) | core-crypto | S-A1 | L | ready | +| S-B1 | Thumbnail/LQIP generation | media/import | — | L | ready | +| S-B2 | Signed-path import-executor rewrite | media/import | S-B1 | L | ready | +| S-B3 | Streaming import (probe, `total_size`, drive mode) | media/import | S-D1, S-D4 | L | ready | +| S-C1 | Upload-server hardening (envelope gate + invariants) | server | — | L | ready | +| S-C2 | Key-free sync feed | server | S-C1 | L | ready | +| S-C3 | Storage-verification endpoint | server | — | M | ready | +| S-C4 | Share-link serving endpoints | server | S-A5 | M | ready | +| S-C5 | Drop store, inbox, atomic adoption | server | S-A6, S-C1, S-C6 | L | ready | +| S-C6 | Quota service | server | — | M | ready | +| S-C7 | Device-enrollment endpoints (code + relay channel) | server | S-C9 | M | ready | +| S-C8 | Moderation hooks | server | S-C2 | M | ready | +| S-C9 | Device-directory publish/fetch | server | — | M | ready | +| S-C10 | Key-free media serving conformance | server | — | M | ready | +| S-C11 | Refcount GC + retention purge worker | server | S-C1 | M | ready | +| S-C12 | Backup escrow server surface | server | — | S | ready | +| S-D1 | SDK upload client (+ OpenAPI regen) | sdk/clients | S-C1 | M | ready | +| S-D2 | SDK sync/download client + connection-class budget | sdk/clients | S-C2, S-C9 | L | ready | +| S-D3 | Web guest drop client (WASM) | sdk/clients | S-A6, S-C5 | L | ready | +| S-D4 | Verify-before-destroy wiring | sdk/clients | S-C3 | M | ready | +| S-D5 | CLI auth/sync/list | sdk/clients | S-D1, S-D2 | M | ready | +| S-D6 | Web server gateway (key-free reads) | sdk/clients | S-D2 | L | ready | +| S-E1 | Share-link end-to-end serving | fed/sharing | S-C4 | M | ready | +| S-E2 | Federation capabilities + pulls | fed/sharing | S-C2, S-A3 | L | ready | +| S-E3 | LAN peering | fed/sharing | S-D2, S-C7 | L | ready | +| S-F1 | uniffi consolidation (0.29 catalog vs 0.31 core) | platform/FFI | — | M | ready | +| S-F2 | Secure Enclave / StrongBox hybrid composition | platform/FFI | S-A4, S-F1 | L | ready | +| S-F3 | Xcode/Gradle binding wiring + on-device CI | platform/FFI | S-F2 | L | ready | +| S-F4 | Windows TPM (TBS) backend | platform/FFI | S-A4 | M | ready | +| S-F5 | Hardware DEK binding | platform/FFI | S-F2 | M | ready | +| S-G1 | GraphQL retirement | legacy-retire | S-C2, S-D6 | M | blocked | +| S-G2 | Legacy plaintext proto/service removal | legacy-retire | S-C2, S-D2 | S | blocked | +| S-G3 | Plaintext entity retirement (face/person/smart_tag) | legacy-retire | S-G1, S-H3 | M | blocked | +| S-G4 | Legacy import-executor removal | legacy-retire | S-B2 | S | blocked | +| S-H1 | Embeddings + sqlite-vec index | ML | — | L | ready | +| S-H2 | Model registry + version regen | ML | S-H1 | M | ready | +| S-H3 | Semantic/face features | ML | S-H1 | L | ready | +| S-X1 | OpenMLS backend → `OpenMlsAuthority` | blocked-external | upstream | L | blocked | +| S-X2 | MLS membership + Welcome/history delivery | blocked-external | S-X1 | L | blocked | +| S-X3 | Album upgrade ceremony + MLS resilience | blocked-external | S-X2 | L | blocked | +| S-Z1 | Library-settings document schema (design) | design | — | S | ready | + +Lanes are independent by construction; within a lane, "Depends on" is the only ordering. +`blocked` = a dependency (or an upstream project) gates the start, not review priority. + +```mermaid +graph LR + A1[S-A1 wrapped key] --> A3[S-A3 metadata binding] --> E2[S-E2 federation] + A1 --> A6[S-A6 drop crypto] --> C5[S-C5 drop server] --> D3[S-D3 web drop client] + A5[S-A5 sharing crypto] --> C4[S-C4 share serving] --> E1[S-E1 share e2e] + A4[S-A4 p256] --> F2[S-F2 SE/StrongBox] --> F3[S-F3 app wiring] + F1[S-F1 uniffi] --> F2 + F2 --> F5[S-F5 DEK binding] + A4 --> F4[S-F4 win TPM] + C1[S-C1 upload hardening] --> C2[S-C2 sync feed] --> E2 + C1 --> C5 + C1 --> C11[S-C11 GC/purge] + C1 --> D1[S-D1 sdk upload] + C2 --> C8[S-C8 moderation] + C2 --> D2[S-D2 sdk sync] --> D5[S-D5 cli] + C2 --> G2[S-G2 proto retire] + C6[S-C6 quota] --> C5 + C9[S-C9 directory] --> C7[S-C7 enrollment] --> E3[S-E3 peering] + C9 --> D2 --> D6[S-D6 web gateway] --> G1[S-G1 graphql retire] --> G3[S-G3 entity retire] + C3[S-C3 storage verify] --> D4[S-D4 verify-destroy] --> B3[S-B3 streaming import] + D1 --> B3 + D1 --> D5 + D2 --> E3 + B1[S-B1 thumbnails] --> B2[S-B2 executor rewrite] --> G4[S-G4 executor retire] + H1[S-H1 embeddings] --> H2[S-H2 registry] + H1 --> H3[S-H3 semantic/face] --> G3 + X1[S-X1 openmls] --> X2[S-X2 membership] --> X3[S-X3 upgrade ceremony] +``` + +## Lane A — core crypto + +### S-A1 — Wrapped file-key mode + +- **Contract:** [Encryption — Asset Key Derivation](capsule-docs/src/content/docs/design/cryptography/encryption.md), + [Provenance — Asset Manifest](capsule-docs/src/content/docs/design/cryptography/provenance.md). +- **Deliverable:** `asset-keywrap/v1` seal/unseal in `crypto::encryption` (wrap `K` under + the AMK with a fresh `wrap_nonce` folded into the salt; unwrap to STREAM-decrypt), and + `verify_asset` + `structural_ok` enforcing the presence rules (`wrapped_file_key` + present iff `key_mode = wrapped`; `metadata_blob_hash` presence-by-action). +- **Seam today:** `KeyMode`/`WrappedFileKey` manifest fields and the + `info::ASSET_KEYWRAP_V1` label are in-tree (wire-absent defaults, byte-identity + regression-tested); behavior is deliberately unimplemented. +- **Done when:** the wrapped-mode positive/negative cases in the provenance doc's + Validation section exist and pass (tampered `wrapped_file_key` → terminal-reject; + member unwrap + decrypt round-trip); `mise run check-rust` green. +- **Tier:** Unit (exhaustive negative cases). **Blocks:** S-A3, S-A6. + +### S-A2 — Re-key salt fold + +- **Contract:** [Encryption — Re-keying on Rewrite](capsule-docs/src/content/docs/design/cryptography/encryption.md). +- **Deliverable:** fold the fresh `nonce_prefix` into the file-key salt + (`file_id || nonce_prefix`) and the metadata blob's fresh `nonce` into its key salt + (`blob_id || nonce`), plus the writer's refuse-to-reuse-a-nonce defense in depth. +- **Seam today:** local to `crypto::encryption` key derivation; the design contract is + written, the derivation still salts on the id alone. +- **Done when:** the rewrite re-roll unit tests in the encryption doc's Validation + section pass (same `file_id` + epoch `replace` yields a different key AND nonce); + existing round-trip vectors unchanged for first encryptions. +- **Tier:** Unit. **Blocks:** nothing (independent hardening). + +### S-A3 — Metadata↔manifest binding + +- **Contract:** [Provenance](capsule-docs/src/content/docs/design/cryptography/provenance.md), + [Metadata — Provenance Binding and Sealing Order](capsule-docs/src/content/docs/design/metadata.md), + [Validation invariant 25](capsule-docs/src/content/docs/design/threat-model/validation.md). +- **Deliverable:** `Workspace` writes populate `metadata_blob_hash` per the sealing + order; `verify_asset` runs the round-trip equivalence check (decrypted blob == + signed sidecar, blob hash == manifest field); the pure invariant-25 envelope check in + `capsule_core::validation` for the server side. +- **Depends on:** S-A1 (field enforcement lands together). **Blocks:** S-E2. +- **Done when:** metadata round-trip equivalence tests (metadata + encryption docs) + pass; a one-byte sidecar mutation quarantines. +- **Tier:** Unit. + +### S-A4 — P-256 hybrid DSK variant + +- **Contract:** [Keys — Device Keys](capsule-docs/src/content/docs/design/cryptography/keys.md); + seam docs in `capsule-core/src/crypto/keys/p256.rs`. +- **Deliverable:** algorithm-tagged hybrid signature/verifying-key/directory-entry types + over `ClassicalAlgorithm`, `P256HybridSigningKey: Signer` composing a hardware P-256 + half (DER ECDSA) with the software ML-DSA-65 half, and `verify_asset` dispatch on the + directory entry's declared algorithm — the Ed25519 path byte-for-byte unchanged. +- **Seam today:** `crypto::keys::p256` skeleton + `#[ignore]`d contract test. +- **Done when:** `p256_hybrid_round_trip_and_directory_dispatch` un-ignored and green + against a mock P-256 element; existing Ed25519 vectors untouched. +- **Tier:** Unit + Smoke (mock element). **Blocks:** S-F2, S-F4. + +### S-A5 — Share-link crypto + +- **Contract:** [Share Links](capsule-docs/src/content/docs/design/share-links.md); + seam docs in `capsule-core/src/sharing/mod.rs`. +- **Deliverable:** `ShareLinkIssuer` implemented on `Workspace`: scope-key encapsulation + around a fresh ≥128-bit link secret, optional Argon2id passphrase wrap (client-side + unwrap), revocation records. +- **Done when:** the module's `#[ignore]`d tests flip (opaque-id entropy, client-side + passphrase unwrap) plus the share-links doc's unit Validation bullets. +- **Tier:** Unit. **Blocks:** S-C4. + +### S-A6 — Drop crypto + +- **Contract:** [Web Upload](capsule-docs/src/content/docs/design/web-upload.md); + seam docs in `capsule-core/src/drop/mod.rs`. +- **Deliverable:** `seal_drop` (fresh `K`, STREAM, KEM encapsulation to the Drop Key), + `UploadLinkIssuer` + `DropAdopter` on `Workspace` (Drop Key mint + master-key/OGK + escrow wrap; decapsulate → `asset-keywrap/v1` rewrap → signed `create` with + `key_mode = wrapped`), and the WASM build of the sealing path for `capsule-web`. +- **Depends on:** S-A1. **Blocks:** S-C5, S-D3. +- **Done when:** the module's three `#[ignore]`d tests flip; the web-upload doc's unit + Validation bullets pass; the sealing path compiles to `wasm32-unknown-unknown`. +- **Tier:** Unit (seal round-trip + adoption rewrap). + +## Lane B — media / import + +### S-B1 — Thumbnail/LQIP generation + +- **Contract:** [Thumbnails](capsule-docs/src/content/docs/design/thumbnails.md). +- **Deliverable:** thumbnail/preview generation over `capsule-media` (which today decodes + JPEG only — format decoders grow as needed), chromahash LQIP + `dominant_color` into + the sidecar `lqip` field, `DerivativeManifest`-signed outputs. +- **Done when:** generation produces the committed formats with signed derivative + manifests; LQIP lands in the sidecar and renders as the fallback tier. +- **Tier:** Unit + Smoke. **Blocks:** S-B2. + +### S-B2 — Signed-path import-executor rewrite + +- **Contract:** [Import — Pipeline](capsule-docs/src/content/docs/design/import/pipeline.md) (status note). +- **Deliverable:** the legacy `import::executor` unified onto the signed + `lifecycle::Workspace` path (signed `SidecarV1` + manifest + provenance + derivatives), + retiring the unsigned `AssetSidecar` write path. +- **Depends on:** S-B1 (derivative generation is the missing input). **Blocks:** S-G4. +- **Done when:** an executor import produces `verify_asset`-accepting assets with + derivatives; planner determinism suite unchanged. +- **Tier:** Unit (planner) + Smoke (executor). + +### S-B3 — Streaming import + +- **Contract:** [Import — Pipeline: Import-Upload Streaming Mode](capsule-docs/src/content/docs/design/import/pipeline.md). +- **Deliverable:** `library::available_bytes()` (the probe skeleton in + `library/space.rs`), planner `total_size` accounting (it emits counts only today), + the `streaming_recommended` plan attachment at confirmation, the minimum-headroom + hard error, and the executor's import→upload→verify→release window with + halt-on-disconnect. +- **Depends on:** S-D1 (upload client), S-D4 (release gate). **Done when:** the + pipeline doc's three streaming Validation bullets pass; `space.rs`'s `#[ignore]`d + probe test flips. +- **Tier:** Unit (auto-detect) + Smoke (release gating, halt-on-disconnect). + +## Lane C — server (key-free surfaces) + +### S-C1 — Upload-server hardening + +- **Contract:** [Import — Upload Protocol](capsule-docs/src/content/docs/design/import/upload-protocol.md), + [Validation invariants 1–15](capsule-docs/src/content/docs/design/threat-model/validation.md). +- **Deliverable:** the `EnvelopeGate` (skeleton in `capsule-api-upload/src/envelope.rs`) + wired ahead of every write with `capsule_core::validation::protocol_gate` + + `check_manifest_envelope` (already implemented and unit-tested in core), the + idempotency tuples, the finalization transaction ordering, the startup scrub, and + `error.*` codes on every rejection. +- **Done when:** invariants 1–15 each have a rejecting test against the real server + (testcontainer Postgres); the upload doc's session-lifecycle smoke passes; crash + injection between rename and commit recovers per the atomicity invariants. +- **Tier:** Unit + Smoke + E2E case 2/11. **Blocks:** S-C2, S-C5, S-C11, S-D1. + +### S-C2 — Key-free sync feed + +- **Contract:** `capsule-api/sync/proto/capsule/sync/v1/sync.proto` (in-tree), + [Download & Sync](capsule-docs/src/content/docs/design/import/download-sync.md), + [API Surfaces](capsule-docs/src/content/docs/design/api-surfaces.md). +- **Deliverable:** `SyncFeedService` implemented — per-album `sync_seq` minted in the + finalization transaction, the HMAC'd opaque cursor (invariant 22), entries carrying + the manifest as opaque CBOR + metadata blob + blob refs; gRPC metadata negotiation + per the api-surfaces mapping; the salvo↔tonic bridge verified end-to-end. +- **Depends on:** S-C1. **Blocks:** S-C8, S-D2, S-E2, S-G1, S-G2. +- **Done when:** the download-sync doc's sync-feed Validation bullets (monotonicity, + forward-version rejection, rewind rejection, cursor authenticity) pass server-side. +- **Tier:** Unit + Smoke + E2E case 3. + +### S-C3 — Storage-verification endpoint + +- **Contract:** [Storage Verification](capsule-docs/src/content/docs/design/import/storage-verification.md); + stub `capsule-api-media/src/routes/verify.rs`. +- **Deliverable:** `POST /storage/verify` computing stored/indexed/retrievable from the + blob store + Postgres, the `deep` re-hash (rate-limited, coalesced), and the + GC-grace interaction that keeps a just-verified blob out of byte deletion. +- **Done when:** the storage-verification doc's six Validation bullets pass; the stub's + `todo!()` is gone. **Tier:** Unit + Smoke. **Blocks:** S-D4. + +### S-C4 — Share-link serving + +- **Contract:** [Share Links — Security Contract](capsule-docs/src/content/docs/design/share-links.md). +- **Deliverable:** `/s/{opaque-id}` metadata + blob + wrapped-secret endpoints on the + existing stub route group: indistinguishable 404, per-IP/per-link rate limits, + mandatory privacy strip, fail-closed revocation cache, home-server pointer for peers. +- **Depends on:** S-A5. **Blocks:** S-E1. **Done when:** the doc's six Validation + bullets pass. **Tier:** Unit + Smoke. + +### S-C5 — Drop store, inbox, adoption + +- **Contract:** [Web Upload](capsule-docs/src/content/docs/design/web-upload.md), + [Validation invariants 26–32](capsule-docs/src/content/docs/design/threat-model/validation.md); + stubs `capsule-api-media/src/routes/drops.rs`. +- **Deliverable:** drop sessions under link-capability auth (+ Argon2id passphrase + verifier), chunks via the upload mechanics, the inbox rows, and the single-transaction + inbox→album promotion on adoption (invariant 32). +- **Depends on:** S-A6, S-C1, S-C6 (drops charge the owner's quota at creation). +- **Done when:** invariants 26–32 each have a rejecting test; the adoption-atomicity + crash-injection smoke passes. **Tier:** Unit + Smoke + E2E case 13. **Blocks:** S-D3. + +### S-C6 — Quota service + +- **Contract:** [Quota](capsule-docs/src/content/docs/design/quota.md). +- **Deliverable:** `capsule-api-service::quota` per the doc's contract skeleton — + accounting sums, the five states (incl. the Grace-expired lifecycle-write exemption), + enforcement at session creation/cancellation/metadata-growth, `GET /quota`. +- **Done when:** the quota doc's seven Validation bullets pass. **Tier:** Unit + Smoke. + **Blocks:** S-C5. + +### S-C7 — Device-enrollment endpoints + +- **Contract:** [Device Enrollment](capsule-docs/src/content/docs/design/device-enrollment.md); + stubs `capsule-api-auth/src/routes/devices.rs`. +- **Deliverable:** enrollment-code issue/redeem (single-use, 10-min, rate-limited, + deleted on redemption/expiry), the relay channel, and the directory-update path for + cross-device add. +- **Depends on:** S-C9. **Blocks:** S-E3 (peering assumes enrolled same-user devices). +- **Done when:** the enrollment doc's code-lifecycle Validation bullets (expiry, + single-use, local-auth gate) pass. **Tier:** Unit + Smoke; E2E case 12 needs S-F2. + +### S-C8 — Moderation hooks + +- **Contract:** [Moderation](capsule-docs/src/content/docs/design/moderation.md). +- **Deliverable:** federated-report intake (signed, rate-limited — invariant 24), + suspension (`error.moderation.account_suspended` at session creation), takedown + (`served = false`, 410 to peers, moderation provenance record), server blocklist. +- **Depends on:** S-C2 (report transport rides the peer surface). **Done when:** the + moderation doc's six Validation bullets pass. **Tier:** Unit + Smoke. + +### S-C9 — Device-directory publish/fetch + +- **Contract:** [Keys — Device Directory](capsule-docs/src/content/docs/design/cryptography/keys.md), + [Validation invariant 23](capsule-docs/src/content/docs/design/threat-model/validation.md). +- **Deliverable:** the server surface for publishing and fetching signed + `DeviceDirectory` documents with the monotonic `directory_version` check — without it + no sync consumer can verify manifests. (The directory type + signing is implemented + in core.) +- **Done when:** invariant 23's rejecting test passes; a client can fetch and pin a + directory end-to-end. **Tier:** Unit + Smoke. **Blocks:** S-C7, S-D2. + +### S-C10 — Key-free media serving conformance + +- **Contract:** [Filesystem — Server](capsule-docs/src/content/docs/design/filesystem/server.md), + [Encryption — ranged reads](capsule-docs/src/content/docs/design/cryptography/encryption.md), + [Download & Sync](capsule-docs/src/content/docs/design/import/download-sync.md). +- **Deliverable:** `GET /blob/{hash}` serving opaque ciphertext by content address with + HTTP `Range` at the **65,536-byte ciphertext stride**, replacing the plaintext-era + assumptions in the legacy asset routes (marked `LEGACY-PLAINTEXT`), with access-token + auth per route. +- **Done when:** ranged reads decrypt correctly at chunk boundaries (the encryption + doc's ranged-read test against a real server); no plaintext-era route regressions. +- **Tier:** Unit + Smoke. + +### S-C11 — Refcount GC + retention purge worker + +- **Contract:** [Filesystem — Server: Deletion and GC](capsule-docs/src/content/docs/design/filesystem/server.md), + [Organization — Retention Window](capsule-docs/src/content/docs/design/organization.md). +- **Deliverable:** the two-phase mark-and-sweep over refcounts (grace window honored), + the keyless purge worker enforcing `retention_until` from the envelope, and the + orphan sweep the finalization crash-safety depends on. +- **Depends on:** S-C1. **Done when:** the organization doc's retention smokes pass + (early purge refused; post-window purge proceeds; hostile-purge defense). +- **Tier:** Unit + Smoke + E2E case 7 (with S-D2). + +### S-C12 — Backup escrow server surface + +- **Contract:** [Backup — Master-Key Escrow](capsule-docs/src/content/docs/design/backup-recovery.md). +- **Deliverable:** store/fetch of the wrapped master-key escrow blob (opaque to the + server; the wrap format is implemented in core), with the ≥128-bit recovery-secret + rule surfaced client-side. +- **Done when:** escrow round-trips through the server and unwraps with the passphrase + path already tested in core. **Tier:** Smoke. + +## Lane D — SDK / clients + +### S-D1 — SDK upload client + +- **Contract:** [Import — Upload Protocol](capsule-docs/src/content/docs/design/import/upload-protocol.md); + the `todo!()` stubs in `capsule-sdk/src/upload.rs`. +- **Deliverable:** regenerate `openapi.json` against the hardened server (S-C1) and + implement the chunked, resumable, adaptive upload client (the `X-Capsule-*` headers + are registered in the validation doc's header census). +- **Depends on:** S-C1. **Blocks:** S-B3, S-D5. **Done when:** the upload doc's + client-side Validation bullets pass against a real server; E2E case 2 lives. +- **Tier:** Unit + Smoke. + +### S-D2 — SDK sync/download client + +- **Contract:** [Download & Sync](capsule-docs/src/content/docs/design/import/download-sync.md). +- **Deliverable:** the gRPC sync consumer (cursor high-water marks, per-album + `sync_seq` anti-rewind, forward-version rejection), tiered on-demand fetch with the + degrade ladder (403-as-authorization-change), resumable ranged blob fetch, and the + connection-class detection that feeds the cache-eviction byte budget. +- **Depends on:** S-C2, S-C9. **Blocks:** S-D5, S-D6, S-E3, S-G1, S-G2. +- **Done when:** the download-sync doc's client Validation bullets pass; E2E case 3 + lives. **Tier:** Unit + Smoke. + +### S-D3 — Web guest drop client + +- **Contract:** [Web Upload](capsule-docs/src/content/docs/design/web-upload.md). +- **Deliverable:** the `capsule-web` guest flow at `/u/{opaque-id}#…`: WASM `seal_drop`, + the drop upload, progress + failure UX; strictly contribute-only. +- **Depends on:** S-A6, S-C5. **Done when:** E2E case 13's browser half runs (seal → + stage → adopt on a native client → verify on a second device). **Tier:** Smoke + (browser/WASM). + +### S-D4 — Verify-before-destroy wiring + +- **Contract:** [Storage Verification — Verify Before Destroy](capsule-docs/src/content/docs/design/import/storage-verification.md); + the implemented pure predicate `capsule_core::library::release_is_safe`. +- **Deliverable:** the SDK call to `POST /storage/verify` + the 60-second re-verify + window, wired into the three destructive paths (device-owned-original release, + Move-import source deletion, streaming release) via `release_is_safe`. +- **Depends on:** S-C3. **Blocks:** S-B3. **Done when:** `storage_verify.rs`'s + `#[ignore]`d wiring test flips; the client.md verify-before-release smoke passes. +- **Tier:** Unit + Smoke. + +### S-D5 — CLI auth/sync/list + +- **Contract:** [Clients](capsule-docs/src/content/docs/design/clients.md) (the CLI is a client). +- **Deliverable:** the `todo!()` CLI commands (`auth login/logout`, `sync`, `list`) + over the SDK clients. +- **Depends on:** S-D1, S-D2. **Done when:** `capsule auth login && capsule sync && + capsule list` round-trips against a dev server. **Tier:** Smoke. + +### S-D6 — Web server gateway + +- **Contract:** [API Surfaces](capsule-docs/src/content/docs/design/api-surfaces.md) + (library queries are client-side); the `CapsuleGateway` seam in + `capsule-web/src/data/gateway.ts` (`server-gateway.ts` currently throws). +- **Deliverable:** the web app's real read path — sync-fed local store queried + client-side (the browser's analogue of `library.sqlite`), replacing the mock gateway; + auth is already wired. +- **Depends on:** S-D2 (the feed contract). **Blocks:** S-G1 (query parity is the + retirement precondition). **Done when:** the gateway methods run against a dev server + with the mock gateway deleted. **Tier:** Smoke (`mise run check-web` + bun tests). + +## Lane E — federation / sharing + +### S-E1 — Share-link end-to-end serving + +- **Contract:** [Share Links](capsule-docs/src/content/docs/design/share-links.md). +- **Deliverable:** the full share flow — issue on a native client (S-A5), serve (S-C4), + open in a browser with client-side unwrap — plus scenario #33/#42 checks. +- **Depends on:** S-C4. **Done when:** a passphrase-protected album link opens + read-only in a clean browser profile with the privacy strip verified. **Tier:** Smoke. + +### S-E2 — Federation capabilities + pulls + +- **Contract:** [Federation](capsule-docs/src/content/docs/design/federation.md), + [Validation invariants 19–21](capsule-docs/src/content/docs/design/threat-model/validation.md). +- **Deliverable:** capability issuance/verification/refresh/revocation (`iat`, album + `aud`, `revoked-jti` with pruning + 15-min fail-closed staleness), the pull path over + the sync feed with invariants 1–18 + 25 re-applied, per-peer compartmentalization + (budgets, circuit breaker, probation), soft-fail rejected-hash table, scope + enforcement by blob role. +- **Depends on:** S-C2, S-A3. **Done when:** the federation doc's seven Validation + bullets pass; E2E case 4 lives. **Tier:** Unit + Smoke + E2E case 4. + +### S-E3 — LAN peering + +- **Contract:** [Peering](capsule-docs/src/content/docs/design/peering.md). +- **Deliverable:** `capsule-sdk::peering` — opaque rotating mDNS discovery, mutual-TLS + with the classical half + application-layer hybrid chain check, delta-scoped backup + artifact transfer over ranged GET, ingest through the restore path. +- **Depends on:** S-D2 (cursor model + transport plumbing), S-C7 (enrolled same-user + devices). **Done when:** the peering doc's six Validation bullets pass; E2E case 5 + lives. **Tier:** Unit + Smoke per platform. + +## Lane F — platform / FFI + +### S-F1 — uniffi consolidation + +- **Contract:** the two surfaces' crate docs (`capsule-core-ffi/src/lib.rs`, + `capsule-core/src/ffi.rs`). +- **Deliverable:** one uniffi version and one bindings strategy for the `Catalog` + surface (0.29) and the `FfiWorkspace`/`HardwareSigner` surface (0.31) — either merged + or explicitly layered — keeping `mise run gen-bindings` + `verify-examples` green and + the Swift app's `CatalogFFIBridge` compiling. +- **Done when:** a single uniffi version across the workspace; both binding sets + regenerate non-empty. **Tier:** Smoke (binding generation + harness tests). + **Blocks:** S-F2. + +### S-F2 — Secure Enclave / StrongBox composition + +- **Contract:** [Keys — Device Keys](capsule-docs/src/content/docs/design/cryptography/keys.md); + the harness adapters in `capsule-core-swift` / `capsule-core-kotlin`. +- **Deliverable:** the existing SE/StrongBox `HardwareSigner` adapters composed + end-to-end through `P256HybridSigningKey` into workspace signing, with the + per-platform smoke (sign/verify/non-exportability) running on real hardware where CI + allows. **Depends on:** S-A4, S-F1. **Blocks:** S-F3, S-F5. +- **Done when:** `capsule-core-swift`'s `swift test` exercises the real Secure Enclave + path with the P-256 composition; the Kotlin harness mirrors it. **Tier:** Smoke per + platform; enables E2E case 12. + +### S-F3 — App binding wiring + on-device CI + +- **Contract:** [Clients](capsule-docs/src/content/docs/design/clients.md). +- **Deliverable:** the generated bindings + `cdylib`/`staticlib` wired into the real + Xcode and Gradle apps, with on-device CI lanes. **Depends on:** S-F2. +- **Done when:** both apps build in CI consuming the produced bindings. **Tier:** Smoke. + +### S-F4 — Windows TPM (TBS) backend + +- **Contract:** [Keys — Device Keys](capsule-docs/src/content/docs/design/cryptography/keys.md). +- **Deliverable:** the TBS-path `HardwareSigner` (the tss-esapi reference covers + Linux), P-256 composed. **Depends on:** S-A4. **Done when:** the Windows smoke + mirrors the TPM reference adapter's. **Tier:** Smoke. + +### S-F5 — Hardware DEK binding + +- **Contract:** [Keys — Device Keys](capsule-docs/src/content/docs/design/cryptography/keys.md). +- **Deliverable:** the device **encryption** key's classical half hardware-bound + (P-256 ECDH), mirroring the DSK composition. **Depends on:** S-F2. **Tier:** Smoke. + +## Lane G — legacy retirement (frozen until preconditions) + +Every `LEGACY-PLAINTEXT (frozen)` marker in the tree names its slice here. Frozen code +keeps compiling and takes no new surface. + +### S-G1 — GraphQL retirement + +- **Contract:** [API Surfaces — Legacy: GraphQL](capsule-docs/src/content/docs/design/api-surfaces.md). +- **Deliverable:** delete `capsule-api-library` (schema, disabled dataloaders, the + `library` feature) once the client-side query path reaches parity. +- **Depends on:** S-C2, S-D6 (the parity precondition is explicit: the web app's reads + run on the gateway, not GraphQL — today they run on the mock, so nothing user-facing + breaks earlier, but retirement waits for the real path). **Blocks:** S-G3. + +### S-G2 — Legacy plaintext proto/service removal + +- **Deliverable:** delete `photolibrary.metadata.v1` (proto + `CapsuleMetadataService`) + once `capsule.sync.v1` serves clients. **Depends on:** S-C2, S-D2. + +### S-G3 — Plaintext entity retirement + +- **Contract:** [Filesystem — Server: PostgreSQL](capsule-docs/src/content/docs/design/filesystem/server.md) + (the key-free row set). +- **Deliverable:** retire the server-side plaintext-era entities (`face`, `person`, + `smart_tag`, `memory`, and the plaintext columns on `asset`) with forward-only + migrations down to the key-free set; their features re-land client-side + ([AI/ML](capsule-docs/src/content/docs/design/ai.md) + client-side views). +- **Depends on:** S-G1, S-H3 (feature parity client-side before the server rows go). + +### S-G4 — Legacy import-executor removal + +- **Deliverable:** delete the unsigned `AssetSidecar` write path once S-B2 lands. + **Depends on:** S-B2. + +## Lane H — ML (client-side) + +The `feat/ml` PR stack (#335–342) designed this lane in detail (registry, vector index, +AI tags, embedding provenance, orchestration, CLIP runner behind the default-off +`inference` feature); these slices supersede-or-land that work rather than duplicating +its design here. + +### S-H1 — Embeddings + sqlite-vec index + +- **Contract:** [AI/ML](capsule-docs/src/content/docs/design/ai.md) (embedding + provenance; the vec0 inner-product query model). +- **Deliverable:** the local vector index in `capsule-core::db` with the + `(model_id, model_version)` insert refusal (unknown models rejected; superseded + admitted as stale) and per-asset regen. **Blocks:** S-H2, S-H3. + +### S-H2 — Model registry + version regen + +- **Deliverable:** the canonical inventory rows in code; version-bump stale-flagging + + background per-asset regeneration; E2E case 10. **Depends on:** S-H1. + +### S-H3 — Semantic/face features + +- **Deliverable:** the v1-committed slots (MobileCLIP-B, YOLOv10, SCRFD, + InsightFace-AdaFace) on the deterministic execution path with the platform-partition + fallback; `tags_ai` population. **Depends on:** S-H1. **Blocks:** S-G3. + +## Lane X — blocked on upstream + +### S-X1 — OpenMLS backend → `OpenMlsAuthority` + +- **Contract:** [Cryptography — MLS](capsule-docs/src/content/docs/design/cryptography/mls.md) + (status note), [Keys — Write Authority Interface](capsule-docs/src/content/docs/design/cryptography/keys.md). +- **Blocked on:** an OpenMLS RustCrypto backend for ciphersuite `0x004D` + (openmls#1940) or IETF finalization of the PQ ciphersuites draft. **Unblock check:** + openmls release notes; re-evaluate quarterly. +- **Deliverable:** `OpenMlsAuthority` behind `&dyn AlbumAuthority` — drops in without + touching `verify_asset`; the `ReferenceAuthority` epoch ledger stays as the offline + and test authority. + +### S-X2 — MLS membership + Welcome/history delivery + +- **Deliverable:** the four membership ceremonies, `AlbumKeyDistribution`, history + policies. **Depends on:** S-X1. Enables organization's invitation surface, + moderation's per-user block, enrollment's group joins (their docs carry the status + note). + +### S-X3 — Album upgrade ceremony + MLS resilience + +- **Deliverable:** the tombstone-plus-fork ceremony, re-keying, reconciliation + (`ReconcileOutcome`). **Depends on:** S-X2; E2E case 8. + +## Lane Z — design follow-ups (docs, not code) + +### S-Z1 — Library-settings document schema + +- **Contract:** [Metadata — How Operations Travel](capsule-docs/src/content/docs/design/metadata.md), + [Organization — views](capsule-docs/src/content/docs/design/organization.md). +- **Deliverable:** the concrete schema for the per-owner E2E-encrypted library-settings + document (smart-album definitions, scope-override map) as a design-doc addition — + the docs declare the surface but not its fields. diff --git a/capsule-core-kotlin/README.md b/capsule-core-kotlin/README.md index e6a447cc..390c3011 100644 --- a/capsule-core-kotlin/README.md +++ b/capsule-core-kotlin/README.md @@ -39,5 +39,5 @@ On-device StrongBox (a physical device with a secure element; an emulator only h - **Android SDK.** `./gradlew` needs the Android SDK (`ANDROID_HOME` / `local.properties`). - **StrongBox** is on-device only and (like Secure Enclave and the TPM) exposes ECDSA-P256, not Ed25519, so `StrongBoxSigner` does not yet wire into the Ed25519 `createWithHardwareSigner` path; - that needs the P-256 hybrid-DSK variant tracked in the repo `DEFERRED.md`. Use `SoftwareSigner` + that needs the P-256 hybrid-DSK variant tracked in `SLICES.md` (slice S-A4). Use `SoftwareSigner` for an end-to-end FFI round trip. diff --git a/capsule-core-kotlin/src/main/kotlin/com/justin13888/capsule/hardware/StrongBoxSigner.kt b/capsule-core-kotlin/src/main/kotlin/com/justin13888/capsule/hardware/StrongBoxSigner.kt index 68f6357f..5b96672c 100644 --- a/capsule-core-kotlin/src/main/kotlin/com/justin13888/capsule/hardware/StrongBoxSigner.kt +++ b/capsule-core-kotlin/src/main/kotlin/com/justin13888/capsule/hardware/StrongBoxSigner.kt @@ -21,7 +21,7 @@ import java.security.interfaces.ECPrivateKey * The AndroidKeyStore exposes **ECDSA over NIST P-256**, not Ed25519, so this returns P-256 * material: an X.509 `SubjectPublicKeyInfo` public key and an ASN.1/DER ECDSA signature over * `msg`. It therefore does **not** yet plug into the Ed25519 `createWithHardwareSigner` path — - * wiring a StrongBox device in needs the P-256 hybrid-DSK variant tracked in `DEFERRED.md`. It is + * wiring a StrongBox device in needs the P-256 hybrid-DSK variant tracked in `SLICES.md` (slice S-A4). It is * the real hardware adapter + the on-device non-exportability check; for an end-to-end FFI round * trip use [SoftwareSigner] (genuine Ed25519). * diff --git a/capsule-core-swift/README.md b/capsule-core-swift/README.md index 9e640b9d..05e12fe8 100644 --- a/capsule-core-swift/README.md +++ b/capsule-core-swift/README.md @@ -31,5 +31,5 @@ swift test # software paths run anywhere; the Secure Enclave tes - The **software** path is exercised end to end (the FFI + the hardware-signer foreign trait). - The **Secure Enclave** adapter is P-256 (the Enclave has no Ed25519), so it is not yet wired into the Ed25519 `createWithHardwareSigner` path; that needs the P-256 hybrid-DSK variant tracked in - the repo `DEFERRED.md`. Wiring the bindings + dylib into the real `capsule-swift` Xcode app is + `SLICES.md` (slice S-A4). Wiring the bindings + dylib into the real `capsule-swift` Xcode app is also a follow-up. diff --git a/capsule-core-swift/Sources/CapsuleHardware/SecureEnclaveSigner.swift b/capsule-core-swift/Sources/CapsuleHardware/SecureEnclaveSigner.swift index 14699496..f2468b23 100644 --- a/capsule-core-swift/Sources/CapsuleHardware/SecureEnclaveSigner.swift +++ b/capsule-core-swift/Sources/CapsuleHardware/SecureEnclaveSigner.swift @@ -10,7 +10,7 @@ import Foundation /// The Secure Enclave only does **NIST P-256**, not Ed25519, so this returns P-256 material: a /// 65-byte x9.63 public key and a 64-byte ECDSA `r‖s` signature over `msg`. It therefore does /// **not** yet plug into the Ed25519 `FfiWorkspace.createWithHardwareSigner` path — wiring a -/// Secure Enclave device in needs the P-256 hybrid-DSK variant tracked in `DEFERRED.md`. It is the +/// Secure Enclave device in needs the P-256 hybrid-DSK variant tracked in `SLICES.md` (slice S-A4). It is the /// real hardware adapter + the on-device non-exportability check; for an end-to-end FFI round trip /// in CI/local tests use ``SoftwareSigner`` (genuine Ed25519). /// diff --git a/capsule-core/README-tpm.md b/capsule-core/README-tpm.md index 4c7996f2..d3a96d7e 100644 --- a/capsule-core/README-tpm.md +++ b/capsule-core/README-tpm.md @@ -14,7 +14,7 @@ cargo build -p capsule-core --features tpm On macOS there is no TPM; the reference is verified to compile + link on Linux. (See the algorithm caveat in `src/crypto/keys/tpm.rs`: shipping TPMs do ECDSA-P256, not Ed25519, so this backend -awaits the P-256 hybrid-DSK variant before it composes into the device key — see `DEFERRED.md`.) +awaits the P-256 hybrid-DSK variant before it composes into the device key — see `SLICES.md`.) ## Smoke test with a software TPM (`swtpm`) diff --git a/capsule-core/src/crypto/authority/mod.rs b/capsule-core/src/crypto/authority/mod.rs index d123426c..60e66cb2 100644 --- a/capsule-core/src/crypto/authority/mod.rs +++ b/capsule-core/src/crypto/authority/mod.rs @@ -10,7 +10,7 @@ //! in flight apart from a forged epoch). //! //! Real OpenMLS integration is deferred (its PQ ciphersuite is a non-final draft on a C -//! backend — see `DEFERRED.md`); [`ReferenceAuthority`] is a deterministic, +//! backend — see `SLICES.md`); [`ReferenceAuthority`] is a deterministic, //! admin-signature-backed stand-in that preserves every property `verify_asset` tests for. //! Because `verify_asset` consumes only `&dyn AlbumAuthority`, swapping in an //! `OpenMlsAuthority` later is transparent. diff --git a/capsule-core/src/crypto/keys/keystore.rs b/capsule-core/src/crypto/keys/keystore.rs index 4fc62013..fa8e66bb 100644 --- a/capsule-core/src/crypto/keys/keystore.rs +++ b/capsule-core/src/crypto/keys/keystore.rs @@ -2,7 +2,7 @@ //! keys) and its encrypted-at-rest form [`AccountFile`]. //! //! This stands in for the hardware-bound keystores (Secure Enclave / StrongBox / TPM) the -//! design specifies per platform — those are deferred (see `DEFERRED.md`). Here the master +//! design specifies per platform — those are deferred (see `SLICES.md`). Here the master //! key is wrapped under a passphrase via [`pwkdf`], and the device identity private keys //! are sealed under the master key (the design's "master key wraps device identity private //! keys"). SSoT: [Cryptography — Keys]. diff --git a/capsule-core/src/crypto/keys/signer.rs b/capsule-core/src/crypto/keys/signer.rs index 6acba0aa..d6b672dc 100644 --- a/capsule-core/src/crypto/keys/signer.rs +++ b/capsule-core/src/crypto/keys/signer.rs @@ -8,7 +8,7 @@ //! //! Signing is **fallible** because a hardware element can refuse (the user cancels a //! biometric, the element is unavailable). The software implementation never returns `Err`. -//! The hardware-backed implementation lands with [hardware-bound key storage] (`DEFERRED.md`). +//! The hardware-backed implementation lands with [hardware-bound key storage] (`SLICES.md`). //! //! [Cryptography — Keys § Device Keys]: https://docs/design/cryptography/keys/#device-keys //! [hardware-bound key storage]: https://docs/design/cryptography/keys/ diff --git a/capsule-core/src/crypto/keys/tpm.rs b/capsule-core/src/crypto/keys/tpm.rs index 46466dd4..931aa4cf 100644 --- a/capsule-core/src/crypto/keys/tpm.rs +++ b/capsule-core/src/crypto/keys/tpm.rs @@ -13,7 +13,7 @@ //! therefore demonstrates the TPM key lifecycle and the non-exportability contract, but does //! **not** yet plug into [`HardwareBackedSigner`](super::HardwareBackedSigner), which composes an //! Ed25519 half. Wiring a TPM device in needs the P-256 hybrid-DSK variant tracked in -//! `DEFERRED.md` — exactly the follow-up the Secure Enclave note calls out. +//! `SLICES.md` (slice S-A4) — exactly the follow-up the Secure Enclave note calls out. //! //! # Non-exportability //! diff --git a/capsule-core/src/ffi.rs b/capsule-core/src/ffi.rs index e2523e7b..26905b2d 100644 --- a/capsule-core/src/ffi.rs +++ b/capsule-core/src/ffi.rs @@ -77,7 +77,7 @@ impl FfiWorkspace { impl FfiWorkspace { /// Create a brand-new workspace rooted at `root`, guarded by `passphrase` at the Argon2id /// cost for `tier`. (Re-opening an existing workspace is a separate core capability still - /// pending; see `DEFERRED.md`.) + /// pending; see `SLICES.md`.) #[uniffi::constructor] pub fn create(root: String, passphrase: Vec, tier: DeviceTier) -> FfiResult> { let ws = Workspace::create(&PathBuf::from(root), &passphrase, tier)?; diff --git a/capsule-core/src/lifecycle.rs b/capsule-core/src/lifecycle.rs index 52262e03..9b1d4bd8 100644 --- a/capsule-core/src/lifecycle.rs +++ b/capsule-core/src/lifecycle.rs @@ -19,7 +19,7 @@ //! Clients store **plaintext** locally (original + signed sidecar + provenance chain); //! encryption produces the artifacts that cross a boundary. Offline epoch rotation is supported //! ([`rotate_epoch`](Workspace::rotate_epoch)); the MLS membership ceremony (`Welcome`, -//! add/remove) remains deferred (see `DEFERRED.md`). +//! add/remove) remains deferred (see `SLICES.md`). //! //! [`verify_asset`]: crate::crypto::verify_asset @@ -343,7 +343,7 @@ impl Workspace { /// "AMK bump + write-tier rotation are one commit" atomicity. The admin key (the ledger /// root) is stable across epochs, and existing assets stay verifiable under their original /// epoch. Returns the new epoch. Membership changes / the MLS `Welcome` flow remain deferred - /// (see `DEFERRED.md`). + /// (see `SLICES.md`). pub fn rotate_epoch(&mut self, album_id: Uuid) -> Result { let next = { let album = self From 29a832193a5cb30d0ea153fb80d1dec9c3625b6d Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Thu, 2 Jul 2026 22:21:21 -0400 Subject: [PATCH 14/58] ci: install convco by its fully-qualified cargo backend name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The commit-lint job installed bare convco, which mise 2026.7.0 (released today) no longer resolves — convco has never been in the mise registry under a short name, which is exactly why mise.toml pins it as cargo:convco = 0.6.4. Older mise releases resolved the short name anyway; the new registry does not, so the job died at tool install before linting anything. Use the fully-qualified name the pin declares. --- .github/workflows/ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a6344676..c7438a04 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -103,7 +103,9 @@ jobs: - name: Set up convco (via mise) uses: jdx/mise-action@v2 with: - install_args: convco + # The fully-qualified backend name matching the mise.toml pin — the bare + # short name stopped resolving in the mise 2026.7.0 registry. + install_args: cargo:convco - name: Check commits follow Conventional Commits run: convco check --first-parent --ignore-reverts ${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }} From d9fab536fd41f1355f2b744fc69acef8bb4525d0 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 18:05:44 -0400 Subject: [PATCH 15/58] chore: add rawshift as git submodule --- .gitmodules | 3 +++ rawshift | 1 + 2 files changed, 4 insertions(+) create mode 100644 .gitmodules create mode 160000 rawshift diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 00000000..97e08159 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "rawshift"] + path = rawshift + url = git@github.com:justin13888/rawshift.git diff --git a/rawshift b/rawshift new file mode 160000 index 00000000..14e59f6d --- /dev/null +++ b/rawshift @@ -0,0 +1 @@ +Subproject commit 14e59f6d557f0f4769c5436109fedf6964d3e761 From 8e8e0c303498ed0b6737c16229899ddf1b0a3d77 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:21:49 -0400 Subject: [PATCH 16/58] docs(design): track human review state in frontmatter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every design doc now carries a schema-validated `status: draft | stable` frontmatter field, starting at `draft`. The field exists because all design docs are queued for a manual re-review before v1: it maps 1:1 to that queue and flips to `stable` per doc as review passes, without affecting publication. Starlight's built-in `draft:` was deliberately not used — it unpublishes pages, which is the wrong semantic for review tracking. --- capsule-docs/src/content.config.ts | 13 +++++++++++-- capsule-docs/src/content/docs/design/ai.md | 1 + .../src/content/docs/design/api-surfaces.md | 1 + .../src/content/docs/design/authentication.md | 1 + .../src/content/docs/design/authorization.md | 1 + .../src/content/docs/design/backup-recovery.md | 1 + capsule-docs/src/content/docs/design/clients.md | 1 + .../content/docs/design/cryptography/encryption.md | 1 + .../docs/design/cryptography/failure-modes.md | 1 + .../src/content/docs/design/cryptography/index.md | 1 + .../src/content/docs/design/cryptography/keys.md | 1 + .../src/content/docs/design/cryptography/mls.md | 1 + .../content/docs/design/cryptography/primitives.md | 1 + .../content/docs/design/cryptography/provenance.md | 1 + .../src/content/docs/design/device-enrollment.md | 1 + capsule-docs/src/content/docs/design/federation.md | 1 + .../src/content/docs/design/filesystem/client.md | 1 + .../src/content/docs/design/filesystem/index.md | 1 + .../content/docs/design/filesystem/maintenance.md | 1 + .../src/content/docs/design/filesystem/server.md | 1 + capsule-docs/src/content/docs/design/i18n.md | 1 + .../src/content/docs/design/import/download-sync.md | 1 + .../src/content/docs/design/import/index.md | 1 + .../src/content/docs/design/import/pipeline.md | 1 + .../docs/design/import/storage-verification.md | 1 + .../content/docs/design/import/upload-protocol.md | 1 + capsule-docs/src/content/docs/design/index.md | 1 + capsule-docs/src/content/docs/design/metadata.md | 1 + .../src/content/docs/design/mls-resilience.md | 1 + capsule-docs/src/content/docs/design/moderation.md | 1 + capsule-docs/src/content/docs/design/module-map.md | 1 + .../src/content/docs/design/organization.md | 1 + capsule-docs/src/content/docs/design/peering.md | 1 + capsule-docs/src/content/docs/design/principles.md | 10 ++++++++++ capsule-docs/src/content/docs/design/quota.md | 1 + capsule-docs/src/content/docs/design/share-links.md | 1 + .../src/content/docs/design/threat-model/index.md | 1 + .../content/docs/design/threat-model/scenarios.md | 1 + .../docs/design/threat-model/schema-rules.md | 1 + .../content/docs/design/threat-model/validation.md | 1 + capsule-docs/src/content/docs/design/thumbnails.md | 1 + capsule-docs/src/content/docs/design/versioning.md | 1 + capsule-docs/src/content/docs/design/web-upload.md | 1 + 43 files changed, 62 insertions(+), 2 deletions(-) diff --git a/capsule-docs/src/content.config.ts b/capsule-docs/src/content.config.ts index a7656120..32e14e92 100644 --- a/capsule-docs/src/content.config.ts +++ b/capsule-docs/src/content.config.ts @@ -1,9 +1,18 @@ -import { defineCollection } from 'astro:content'; +import { defineCollection, z } from 'astro:content'; import { docsLoader } from '@astrojs/starlight/loaders'; import { docsSchema } from '@astrojs/starlight/schema'; import { docsVersionsLoader } from 'starlight-versions/loader'; export const collections = { - docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }), + docs: defineCollection({ + loader: docsLoader(), + schema: docsSchema({ + extend: z.object({ + // Design-doc review state: `draft` until a human re-review passes the + // doc, then flipped to `stable`. See Core Principles — Doc Structure. + status: z.enum(['draft', 'stable']).optional(), + }), + }), + }), versions: defineCollection({ loader: docsVersionsLoader() }), }; diff --git a/capsule-docs/src/content/docs/design/ai.md b/capsule-docs/src/content/docs/design/ai.md index af9a342d..15d77881 100644 --- a/capsule-docs/src/content/docs/design/ai.md +++ b/capsule-docs/src/content/docs/design/ai.md @@ -1,6 +1,7 @@ --- title: AI/ML Integrations description: AI feature architecture, the canonical model inventory, embedding provenance, and AI/user metadata separation +status: draft --- Capsule runs a hierarchy of ML models, all **client-side** (the server never holds plaintext). The stable contract is the *structure*: three functional categories, the AI/user namespace separation in [AI Output Containment](#ai-output-containment), the canonical model inventory in [Models and Algorithms](#models-and-algorithms), and the [embedding-provenance](#embedding-provenance) invariant. The specific feature list and per-model choices are current defaults that will evolve with field testing. diff --git a/capsule-docs/src/content/docs/design/api-surfaces.md b/capsule-docs/src/content/docs/design/api-surfaces.md index 117a4c46..0a41ab41 100644 --- a/capsule-docs/src/content/docs/design/api-surfaces.md +++ b/capsule-docs/src/content/docs/design/api-surfaces.md @@ -1,6 +1,7 @@ --- title: API Surfaces description: Which server surface speaks which transport, and how negotiation and rejection map across them +status: draft --- Capsule's server exposes exactly **two** wire transports, chosen per surface: **REST with an OpenAPI schema** for request/response surfaces, and **gRPC** for the sync feed and federation pull. This doc owns the surface ↔ transport map and the cross-transport mapping of the [universal handshake](/design/threat-model/validation/#protocol-and-capability-negotiation) and rejection semantics. The handshake *rules* stay owned by [Threat Model — Validation](/design/threat-model/validation/); error-code identity stays owned by [Internationalization](/design/i18n/#server-error-codes); this doc owns only how both ride each transport. diff --git a/capsule-docs/src/content/docs/design/authentication.md b/capsule-docs/src/content/docs/design/authentication.md index ac9ae11c..6656846e 100644 --- a/capsule-docs/src/content/docs/design/authentication.md +++ b/capsule-docs/src/content/docs/design/authentication.md @@ -1,6 +1,7 @@ --- title: Authentication description: Identity, account portability, session and access tokens +status: draft --- Authentication binds a user identity to their master key, which is the root of every encryption and decryption operation in Capsule. The server can prove "this request is from a session it issued" but cannot prove "this user is who they say they are" — the master key, owned client-side, is the actual identity root. Everything below works to keep that binding intact through the lifetime of a session and across server moves. diff --git a/capsule-docs/src/content/docs/design/authorization.md b/capsule-docs/src/content/docs/design/authorization.md index dc5259ae..71bca1b0 100644 --- a/capsule-docs/src/content/docs/design/authorization.md +++ b/capsule-docs/src/content/docs/design/authorization.md @@ -1,6 +1,7 @@ --- title: Authorization description: The closed lifecycle-action set and how every destructive operation is signed and audited +status: draft --- Authorization in Capsule is **the same proof as a write**: every lifecycle transition — create, replace, delete, metadata-update, derivative add/replace, trash-restore — is an [asset manifest](/design/cryptography/provenance/#asset-manifest) signed under the album's per-epoch write-tier key. There is no weaker path to destroy data than to add it. diff --git a/capsule-docs/src/content/docs/design/backup-recovery.md b/capsule-docs/src/content/docs/design/backup-recovery.md index 2ee6b905..28aeb837 100644 --- a/capsule-docs/src/content/docs/design/backup-recovery.md +++ b/capsule-docs/src/content/docs/design/backup-recovery.md @@ -1,6 +1,7 @@ --- title: Backup and Recovery description: The portable backup artifact, the master-key escrow, and the recovery flows +status: draft --- Capsule treats loss of data — and loss of the keys that decrypt it — as a first-class failure mode. Recovery rests on a single rule: holding the recovery secret must restore every asset, even after every device is lost. This document defines the two artifacts and the mechanisms that uphold it. diff --git a/capsule-docs/src/content/docs/design/clients.md b/capsule-docs/src/content/docs/design/clients.md index 72e7334f..cd26f1b2 100644 --- a/capsule-docs/src/content/docs/design/clients.md +++ b/capsule-docs/src/content/docs/design/clients.md @@ -1,6 +1,7 @@ --- title: Clients description: Native client priorities, what every client must validate, and the sandboxed decoder +status: draft --- Capsule's clients are native per platform, with as little divergence as possible. The cross-platform logic — including the entire [`verify_asset`](/design/cryptography/keys/#write-authorization) chokepoint, the [import pipeline](/design/import/pipeline/), and the [library layout](/design/filesystem/client/) — lives in `capsule-core` and is consumed by every native client through `capsule-sdk`. Each native client's job is the surface above that: rendering, input, and platform integration. diff --git a/capsule-docs/src/content/docs/design/cryptography/encryption.md b/capsule-docs/src/content/docs/design/cryptography/encryption.md index 98dfc606..2c77f63e 100644 --- a/capsule-docs/src/content/docs/design/cryptography/encryption.md +++ b/capsule-docs/src/content/docs/design/cryptography/encryption.md @@ -1,6 +1,7 @@ --- title: Asset and Metadata Encryption description: How Capsule encrypts asset bytes and metadata blobs, including streaming and wire formats +status: draft --- Every asset Capsule stores — original bytes, derivative bytes, metadata blob — is encrypted client-side before it ever crosses a network boundary. The encryption code lives in `capsule-core::crypto::encryption` and is the only place AES-256-GCM is invoked in the codebase. Two constructions live here: diff --git a/capsule-docs/src/content/docs/design/cryptography/failure-modes.md b/capsule-docs/src/content/docs/design/cryptography/failure-modes.md index a7b88b04..927ea6f2 100644 --- a/capsule-docs/src/content/docs/design/cryptography/failure-modes.md +++ b/capsule-docs/src/content/docs/design/cryptography/failure-modes.md @@ -1,6 +1,7 @@ --- title: Failure Modes and Recovery description: What can go wrong with Capsule's cryptographic state, and the independent paths that restore it +status: draft --- Capsule treats loss of data — and loss of the keys that decrypt it — as a first-class concern. This doc catalogues what can go wrong, how each failure is detected or contained, and the redundant, independent paths that restore a user's *entire* asset collection — including after catastrophic software bugs, not just key loss. diff --git a/capsule-docs/src/content/docs/design/cryptography/index.md b/capsule-docs/src/content/docs/design/cryptography/index.md index df01cc2f..1691a1a2 100644 --- a/capsule-docs/src/content/docs/design/cryptography/index.md +++ b/capsule-docs/src/content/docs/design/cryptography/index.md @@ -1,6 +1,7 @@ --- title: Cryptography description: Capsule's cryptographic stack — the entry point to the sub-docs +status: draft --- Cryptography in Capsule is everything that makes E2EE work over an asset-heavy, sync-heavy workload. The choices and constructions are split across focused sub-docs because each is implementable and testable on its own, but they share one home: every primitive and key-handling routine lives in `capsule-core::crypto`, so every client and the server's no-key envelope-validation path use exactly the same code. diff --git a/capsule-docs/src/content/docs/design/cryptography/keys.md b/capsule-docs/src/content/docs/design/cryptography/keys.md index 744b0bc4..f98607ee 100644 --- a/capsule-docs/src/content/docs/design/cryptography/keys.md +++ b/capsule-docs/src/content/docs/design/cryptography/keys.md @@ -1,6 +1,7 @@ --- title: Key Management description: Capsule's key hierarchy, device coordination, and write authorization +status: draft --- Capsule's keys form a single hierarchy with one backed-up root. The hierarchy is implemented in `capsule-core::crypto::keys`. Every signing site consumes the device signing key through the `Signer` seam there, and hardware-backed keys implement the `HardwareSigner` foreign trait (exported under `capsule-core`'s `ffi` feature), so software and hardware keys are interchangeable. Reference adapters ship for software (`crypto::keys::software`, CI-smoked) and TPM 2.0 (`crypto::keys::tpm`, feature-gated); Secure Enclave and StrongBox adapters live in the `capsule-core-swift` / `capsule-core-kotlin` harness packages. End-to-end hardware composition awaits the planned **P-256 hybrid-DSK variant** — secure elements expose ECDSA-P256, not Ed25519 (see [Device Enrollment](/design/device-enrollment/)). diff --git a/capsule-docs/src/content/docs/design/cryptography/mls.md b/capsule-docs/src/content/docs/design/cryptography/mls.md index e22b40b3..30d7f20c 100644 --- a/capsule-docs/src/content/docs/design/cryptography/mls.md +++ b/capsule-docs/src/content/docs/design/cryptography/mls.md @@ -1,6 +1,7 @@ --- title: MLS Group Membership description: How Capsule binds MLS (RFC 9420) to its identity layer and uses it for album membership +status: draft --- Capsule's group layer is the [MLS ciphersuite](/design/cryptography/primitives/#mls-ciphersuite) from the inventory. It will be implemented in `capsule-core::crypto::mls` (planned) as a thin wrapper over OpenMLS — the wrapper is what binds MLS to Capsule's identity layer ([Keys](/design/cryptography/keys/)) and to the in-band AMK distribution. diff --git a/capsule-docs/src/content/docs/design/cryptography/primitives.md b/capsule-docs/src/content/docs/design/cryptography/primitives.md index e7520623..92b13171 100644 --- a/capsule-docs/src/content/docs/design/cryptography/primitives.md +++ b/capsule-docs/src/content/docs/design/cryptography/primitives.md @@ -1,6 +1,7 @@ --- title: Cryptographic Primitives description: Single-source-of-truth inventory of every cryptographic primitive Capsule uses +status: draft --- This doc is **the single source of truth** for every cryptographic primitive Capsule uses. Other docs (and the rest of the cryptography sub-docs) reference these by anchor — they never restate the choice. Swapping a primitive is a single-row edit here, plus a new `crypto_suite_id` and the dedicated section below. diff --git a/capsule-docs/src/content/docs/design/cryptography/provenance.md b/capsule-docs/src/content/docs/design/cryptography/provenance.md index fc112866..4125c1d4 100644 --- a/capsule-docs/src/content/docs/design/cryptography/provenance.md +++ b/capsule-docs/src/content/docs/design/cryptography/provenance.md @@ -1,6 +1,7 @@ --- title: Signed Manifests and Provenance description: Capsule's signed asset manifest, append-only provenance chains, and derivative provenance +status: draft --- Every asset Capsule stores has a verifiable trace of *who* produced it. The trace is anchored in a small **signed manifest** — bound to the ciphertext, cheap to verify, streaming-compatible — and extended by an **append-only, hash-chained provenance log per asset**. Together these are what let an operator distinguish a legitimate delete from a malicious or bug-induced one after the fact, and what defeats the [stale-revival attack](/design/threat-model/scenarios/#damage-scenario--invariant-map). diff --git a/capsule-docs/src/content/docs/design/device-enrollment.md b/capsule-docs/src/content/docs/design/device-enrollment.md index aad20e07..7016dc8b 100644 --- a/capsule-docs/src/content/docs/design/device-enrollment.md +++ b/capsule-docs/src/content/docs/design/device-enrollment.md @@ -1,6 +1,7 @@ --- title: Device Enrollment description: First-device bootstrap and cross-device add ceremonies for Capsule accounts +status: draft --- A Capsule account has one or more devices, each holding a hardware-bound DSK + DEK cross-signed into the user's [device directory](/design/cryptography/keys/#device-directory). This doc owns the two enrollment ceremonies a device can go through to *get into* that directory: diff --git a/capsule-docs/src/content/docs/design/federation.md b/capsule-docs/src/content/docs/design/federation.md index b4d09e43..4ba2d9ed 100644 --- a/capsule-docs/src/content/docs/design/federation.md +++ b/capsule-docs/src/content/docs/design/federation.md @@ -1,6 +1,7 @@ --- title: Federation description: How Capsule servers share albums across users on different home servers +status: draft --- Federation lets an album owned on one Capsule server be shared with users whose accounts live on another. This document covers **server-to-server** federation only; direct device-to-device sync for a single user is [Peering](/design/peering/). diff --git a/capsule-docs/src/content/docs/design/filesystem/client.md b/capsule-docs/src/content/docs/design/filesystem/client.md index 1a1068e3..75fb2482 100644 --- a/capsule-docs/src/content/docs/design/filesystem/client.md +++ b/capsule-docs/src/content/docs/design/filesystem/client.md @@ -1,6 +1,7 @@ --- title: Client Filesystem description: How clients lay out a library on disk — desktop, mobile, local index, and space recovery +status: draft --- Clients hold keys, so a client stores plaintext. Desktop clients keep a self-contained library directory; mobile clients use platform-sandboxed storage. The cross-platform logic lives in `capsule-core::library` (paths, init, open) and `capsule-core::db` (SQLite cache); per-platform glue lives in `capsule-sdk` and native client code. diff --git a/capsule-docs/src/content/docs/design/filesystem/index.md b/capsule-docs/src/content/docs/design/filesystem/index.md index d14dd085..dbd69663 100644 --- a/capsule-docs/src/content/docs/design/filesystem/index.md +++ b/capsule-docs/src/content/docs/design/filesystem/index.md @@ -1,6 +1,7 @@ --- title: Filesystem description: How Capsule structures files on disk — server vs client, and what they share +status: draft --- Capsule's end-to-end encryption splits the filesystem into two fundamentally different roles. The **server** stores only opaque, content-addressed ciphertext — it never holds a decryption key and cannot interpret a single byte it stores. **Clients** hold the keys, so a client filesystem is a working library of plaintext media, sidecar metadata, and rebuildable caches. The two layouts share a small set of principles but otherwise have little in common. diff --git a/capsule-docs/src/content/docs/design/filesystem/maintenance.md b/capsule-docs/src/content/docs/design/filesystem/maintenance.md index fe37e414..5ecdfaa5 100644 --- a/capsule-docs/src/content/docs/design/filesystem/maintenance.md +++ b/capsule-docs/src/content/docs/design/filesystem/maintenance.md @@ -1,6 +1,7 @@ --- title: Library Maintenance and Atomic Writes description: How Capsule keeps client storage consistent, repairs what it can, and writes atomically +status: draft --- The data-integrity principle treats client storage as *potentially lost* (see [Core Principles](/design/principles/)): unlike the server, a client library sits on consumer hardware, syncs only partially, and is edited by a long-lived process that can be killed mid-write. A client therefore never assumes its library is consistent — it periodically *proves* it is, repairs what it can repair safely, and surfaces what it cannot. diff --git a/capsule-docs/src/content/docs/design/filesystem/server.md b/capsule-docs/src/content/docs/design/filesystem/server.md index 121d6646..37149d70 100644 --- a/capsule-docs/src/content/docs/design/filesystem/server.md +++ b/capsule-docs/src/content/docs/design/filesystem/server.md @@ -1,6 +1,7 @@ --- title: Server Filesystem description: The server's blob store layout, Postgres index, and deployment profiles +status: draft --- The server's job is to hold ciphertext blobs and a key-free index that maps assets to blobs. It performs no decoding, no metadata extraction, and no thumbnail generation — it cannot, since it never holds a decryption key. The blob layout below **is** the contract: a server-side rebuild (re-deriving the Postgres index from blob bytes) depends on the file naming and the manifest envelope being exactly as specified here. diff --git a/capsule-docs/src/content/docs/design/i18n.md b/capsule-docs/src/content/docs/design/i18n.md index f15e6602..634fe8bb 100644 --- a/capsule-docs/src/content/docs/design/i18n.md +++ b/capsule-docs/src/content/docs/design/i18n.md @@ -1,6 +1,7 @@ --- title: Internationalization description: The canonical translation source, the codegen that compiles it into every platform, locale resolution, and the server error-code contract +status: draft --- Capsule is internationalized from a **single canonical source**. Every user-facing diff --git a/capsule-docs/src/content/docs/design/import/download-sync.md b/capsule-docs/src/content/docs/design/import/download-sync.md index ba5001b8..086ae5ac 100644 --- a/capsule-docs/src/content/docs/design/import/download-sync.md +++ b/capsule-docs/src/content/docs/design/import/download-sync.md @@ -1,6 +1,7 @@ --- title: Download and Synchronization description: How Capsule clients discover changes, fetch blobs on demand, and auto-sync +status: draft --- Download is the inverse of [upload](/design/import/upload-protocol/), and rests on the same two foundations: blobs are **content-addressed by ciphertext hash**, and the server never holds a key, so it serves only opaque ciphertext. Where the upload path optimises for correctness under interruption, the download path optimises for **bandwidth and storage frugality** — a client fetches the smallest representation that satisfies the user's current intent, and nothing more. diff --git a/capsule-docs/src/content/docs/design/import/index.md b/capsule-docs/src/content/docs/design/import/index.md index 8026f8d0..6bccf346 100644 --- a/capsule-docs/src/content/docs/design/import/index.md +++ b/capsule-docs/src/content/docs/design/import/index.md @@ -1,6 +1,7 @@ --- title: Import and Synchronization description: Overview of how Capsule imports assets and synchronizes them across devices +status: draft --- We define **import** as the process of taking assets from an external source (a camera, a directory on the filesystem) and bringing them into Capsule's management. Once imported, assets travel between devices via the **upload protocol** (client → server) and the **sync feed** (server → client). diff --git a/capsule-docs/src/content/docs/design/import/pipeline.md b/capsule-docs/src/content/docs/design/import/pipeline.md index 75961bd8..37187915 100644 --- a/capsule-docs/src/content/docs/design/import/pipeline.md +++ b/capsule-docs/src/content/docs/design/import/pipeline.md @@ -1,6 +1,7 @@ --- title: Import Pipeline description: How Capsule scans, plans, and executes a local import on a single device +status: draft --- The import pipeline is the workflow a client runs to bring assets from an external source (a camera, a filesystem directory) into Capsule's management. It is implemented in `capsule-core::import` and runs entirely client-side — no server is contacted until the [upload protocol](/design/import/upload-protocol/) is invoked at the tail of the pipeline. diff --git a/capsule-docs/src/content/docs/design/import/storage-verification.md b/capsule-docs/src/content/docs/design/import/storage-verification.md index e38767bf..ac1d7923 100644 --- a/capsule-docs/src/content/docs/design/import/storage-verification.md +++ b/capsule-docs/src/content/docs/design/import/storage-verification.md @@ -1,6 +1,7 @@ --- title: Storage Verification description: The endpoint a client calls to confirm an asset is durably stored, indexed, and retrievable before any destructive local action +status: draft --- [Upload finalization](/design/import/upload-protocol/#finalization-and-integrity) confirms, **once**, that the bytes the server assembled match the declared hash. That `Completed` acknowledgement is a transfer receipt, not a standing durability guarantee a client can re-check later — and [`verify_asset`](/design/cryptography/keys/#write-authorization) proves *cryptographic* validity and authorization, never that the server still holds the bytes. A client that is about to discard local data therefore has no way, today, to ask the server: *do you actually still have this, indexed and retrievable?* diff --git a/capsule-docs/src/content/docs/design/import/upload-protocol.md b/capsule-docs/src/content/docs/design/import/upload-protocol.md index aa5a4ca5..14da256d 100644 --- a/capsule-docs/src/content/docs/design/import/upload-protocol.md +++ b/capsule-docs/src/content/docs/design/import/upload-protocol.md @@ -1,6 +1,7 @@ --- title: Upload Protocol description: The wire protocol between Capsule clients and the server for resumable, content-addressed uploads +status: draft --- The upload protocol is a custom resumable-upload protocol modeled on [TUS](https://tus.io/) but trimmed to Capsule's needs: no per-request capability negotiation, no metadata smuggled in headers, ciphertext-only payloads. Compatibility is gated once, up front, via the universal [protocol handshake](/design/threat-model/validation/#protocol-and-capability-negotiation). diff --git a/capsule-docs/src/content/docs/design/index.md b/capsule-docs/src/content/docs/design/index.md index 74c33392..f2109d47 100644 --- a/capsule-docs/src/content/docs/design/index.md +++ b/capsule-docs/src/content/docs/design/index.md @@ -1,6 +1,7 @@ --- title: Design Overview description: How Capsule's design docs are organized and where to start +status: draft --- Capsule is an end-to-end-encrypted personal photo and media store with optional federation. These design docs are its normative specification: every primitive, schema, and protocol is declared in exactly one **owner doc** and referenced by anchor everywhere else (the [Single Source of Truth rule](/design/principles/#single-source-of-truth)). diff --git a/capsule-docs/src/content/docs/design/metadata.md b/capsule-docs/src/content/docs/design/metadata.md index f429cdce..6286cfde 100644 --- a/capsule-docs/src/content/docs/design/metadata.md +++ b/capsule-docs/src/content/docs/design/metadata.md @@ -1,6 +1,7 @@ --- title: Metadata description: The CBOR sidecar schema v1, the CRDT semantics for collaborative metadata, identifiers, and geolocation +status: draft --- The CBOR sidecar is the canonical, plaintext-local-only metadata record for every asset (see [Filesystem — Client](/design/filesystem/client/)). It is **self-describing**: field 0 carries the schema version so any reader can detect a schema it does not implement *before* parsing the rest. Versioning the schema in-band is what prevents a faulty or old client from corrupting state with a partial parse. diff --git a/capsule-docs/src/content/docs/design/mls-resilience.md b/capsule-docs/src/content/docs/design/mls-resilience.md index 6e339a5e..9f651208 100644 --- a/capsule-docs/src/content/docs/design/mls-resilience.md +++ b/capsule-docs/src/content/docs/design/mls-resilience.md @@ -1,6 +1,7 @@ --- title: MLS Resilience description: How Capsule's MLS layer recovers from lost commits, state divergence, and group corruption +status: draft --- OpenMLS handles MLS (RFC 9420) correctly under normal operation — commits ordered by the group's chain, duplicates rejected, ratchet advanced atomically. But MLS can still hit scenarios the base protocol does not resolve on its own: a commit lost in transit, two devices proposing concurrently with the wrong ordering, a member whose local state has diverged from the server's. This doc owns Capsule's recovery contracts for those edge cases. diff --git a/capsule-docs/src/content/docs/design/moderation.md b/capsule-docs/src/content/docs/design/moderation.md index 4e99fd0b..a225d031 100644 --- a/capsule-docs/src/content/docs/design/moderation.md +++ b/capsule-docs/src/content/docs/design/moderation.md @@ -1,6 +1,7 @@ --- title: Moderation description: Server moderation policy — reports, suspensions, takedowns, blocklists, federated reporting +status: draft --- Capsule is end-to-end encrypted, so a server **cannot** scan content it holds — server-side content or CSAM scanning is impossible by design, and no server-side content scanner will be built. (Client-side, opt-in ML over a user's *own* library — [AI/ML](/design/ai/) — is a different thing entirely: it runs where the keys are, under the user's control; its candidate shared-album-flagging classifiers are client-side and user-initiated, never a server scanner.) Moderation operates entirely on what *is* available: user reports, account-level signals, and federated peer reputation. diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 77696301..04ca5391 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -1,6 +1,7 @@ --- title: Module Map description: Index of every code module to its owning design doc and validation tier +status: draft --- This is the developer's first stop. It maps every Capsule workspace crate and module to the design doc(s) that govern its behavior, and to the validation tier (Unit / Smoke / E2E — see [Validation Tiers](/design/principles/#validation-tiers)) it ships with. The E2E test surface at the bottom is **bounded**: adding a test there means adding the test to the relevant doc's Validation section and justifying why the cross-module surface is irreducible. diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index a8ce9df4..52597f62 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -1,6 +1,7 @@ --- title: Asset Organization description: Albums (container and view), default-album resolution, asset stacks, and trash retention +status: draft --- **Albums** are Capsule's organizational backbone: [container albums](#container-albums) are the cryptographic unit every asset belongs to, while [view albums](#system--smart-albums-views) are derived, key-free presentations. On top of albums, **stacks** group related files (RAW+JPEG pairs, bursts, live photos) so a library stays tidy, and **trash** stages every destructive operation behind a signed retention window so a buggy or hostile actor cannot silently destroy data. Stacks and trash are metadata-only — they never touch the underlying asset bytes. diff --git a/capsule-docs/src/content/docs/design/peering.md b/capsule-docs/src/content/docs/design/peering.md index 1d5161f4..ff122e0b 100644 --- a/capsule-docs/src/content/docs/design/peering.md +++ b/capsule-docs/src/content/docs/design/peering.md @@ -1,6 +1,7 @@ --- title: Peering description: Direct LAN device-to-device sync within a single user's own devices +status: draft --- Peering is **device-to-device** sync within a single user's own devices. It is distinct from [Federation](/design/federation/), which is server-to-server sharing across *different* users. diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 0dc97330..1d717a53 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -1,6 +1,7 @@ --- title: Core Principles description: The core principles that guide the design and development of Capsule +status: draft --- These principles apply universally to all components of Capsule, from clients to server. The owner-doc rules and structural guidance below apply to every doc in `design/`. @@ -94,6 +95,15 @@ Design intent and shipped code drift apart unless the docs say which is which. E The repo-root `SLICES.md` (a plain repository file, not part of this site) is the executable index of every planned surface — its slice IDs are the unit of implementation work. A doc describing a planned surface writes the *contract* in normative present tense, but must not claim the code exists. +### Review Status + +Separately from per-surface implementation status, every design doc carries a `status` frontmatter field tracking **human design review**, validated by the site schema: + +- `draft` — the doc's current content has not passed a human re-review since the last substantive design change. +- `stable` — a human reviewed the doc as written and signed off; substantive edits flip it back to `draft`. + +The field is review-queue metadata, not publication state (drafts still build and publish). + ## Validation Tiers The three test tiers a design doc may reference: diff --git a/capsule-docs/src/content/docs/design/quota.md b/capsule-docs/src/content/docs/design/quota.md index 853655fa..7f36cc7a 100644 --- a/capsule-docs/src/content/docs/design/quota.md +++ b/capsule-docs/src/content/docs/design/quota.md @@ -1,6 +1,7 @@ --- title: Quota description: Storage quota accounting, thresholds, and enforcement points +status: draft --- Storage quota in Capsule is accounted to `upload_user_id` (the authenticated uploader), which is distinct from `owner_id` (the asset's owner). This separation lets a user upload on behalf of a different owner (with verified permission) while keeping storage cost attributed correctly. The accounting model is enforced at the [server filesystem](/design/filesystem/server/#ownership-partitioning-and-quota) and at [upload session creation](/design/import/upload-protocol/#quota-and-permissions); this doc owns the threshold model and what happens when limits are hit. diff --git a/capsule-docs/src/content/docs/design/share-links.md b/capsule-docs/src/content/docs/design/share-links.md index 7e1cb770..9a6619a3 100644 --- a/capsule-docs/src/content/docs/design/share-links.md +++ b/capsule-docs/src/content/docs/design/share-links.md @@ -1,6 +1,7 @@ --- title: Share Links description: Non-registered-account share link generation, permission model, and public-share serving +status: draft --- Share links let a Capsule user grant view (and possibly limited write) access to an album or a specific asset *without* requiring the recipient to have a Capsule account. The recipient is the [non-registered account](/design/authentication/#account-types) class — no master key, no User IK, no MLS membership. The cryptographic shape (the link secret carries the decryption material; an optional passphrase wraps it with the [password-based KDF](/design/cryptography/primitives/#password-based-kdf)) is owned by [Cryptography — Keys: Non-registered accounts](/design/cryptography/keys/#non-registered-accounts); this doc owns everything else. diff --git a/capsule-docs/src/content/docs/design/threat-model/index.md b/capsule-docs/src/content/docs/design/threat-model/index.md index b57d8307..0cf89c5e 100644 --- a/capsule-docs/src/content/docs/design/threat-model/index.md +++ b/capsule-docs/src/content/docs/design/threat-model/index.md @@ -1,6 +1,7 @@ --- title: Threat Model description: How Capsule contains damage from faulty, malicious, or version-mismatched clients +status: draft --- E2EE shifts most of the trust to the client. The server holds no keys; clients write the canonical state. That makes the question "what damage can a client cause?" load-bearing for the design — a single buggy implementation, a hostile keyholder inside an album, a stranded old build, or a too-new prototype all have to fail safely. diff --git a/capsule-docs/src/content/docs/design/threat-model/scenarios.md b/capsule-docs/src/content/docs/design/threat-model/scenarios.md index 0ae41750..294654ce 100644 --- a/capsule-docs/src/content/docs/design/threat-model/scenarios.md +++ b/capsule-docs/src/content/docs/design/threat-model/scenarios.md @@ -1,6 +1,7 @@ --- title: Damage Scenarios and Quarantine description: The damage-scenario → invariant map, quarantine surface inventory, and provenance immutability rules +status: draft --- The lookup table for "what damage X is prevented by which invariant Y in which doc Z." Each row names a concrete vector found during the audit and the single owner-doc anchor that defeats it. The table itself is the operational core of the threat model — adding a row obliges declaring the defense in exactly one owner doc. diff --git a/capsule-docs/src/content/docs/design/threat-model/schema-rules.md b/capsule-docs/src/content/docs/design/threat-model/schema-rules.md index f0c52cbe..f226a769 100644 --- a/capsule-docs/src/content/docs/design/threat-model/schema-rules.md +++ b/capsule-docs/src/content/docs/design/threat-model/schema-rules.md @@ -1,6 +1,7 @@ --- title: Schema Rules and Open Questions description: Schema evolution, forbidden client behaviors, deprecation policy, and unresolved design questions +status: draft --- Capsule schemas evolve over time, but the rules of evolution are fixed — what fields a writer may add, what a receiver may safely ignore, what fields are closed enums, and what timing/grammar rules apply. Each schema's owner doc defines its fields; this doc defines what evolution is allowed across them. Schema-rule enforcement lives in `capsule-core::crypto` (sidecar/manifest decode) and the validation layers of every API crate. diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index b9464752..36674f45 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -1,6 +1,7 @@ --- title: Validation Invariants description: Server and client refuse-by-default checklists; protocol handshake; idempotency; atomicity +status: draft --- The cross-cutting refuse-by-default rules every Capsule receiver runs before persisting any incoming write. These are the operational core of the threat model — a server or client that skips one of them silently widens the blast radius for the entire client class taxonomy. diff --git a/capsule-docs/src/content/docs/design/thumbnails.md b/capsule-docs/src/content/docs/design/thumbnails.md index f822fafb..6e9cf973 100644 --- a/capsule-docs/src/content/docs/design/thumbnails.md +++ b/capsule-docs/src/content/docs/design/thumbnails.md @@ -1,6 +1,7 @@ --- title: Thumbnails and Previews description: Format inventory, LQIP scheme, and derivative provenance for photo and video derivatives +status: draft --- We generate thumbnails and previews for all photos and videos. This doc is the **single source of truth** for the LQIP scheme and the thumbnail/preview formats — per the [SSoT rule](/design/principles/#single-source-of-truth), other docs reference these by link rather than restating the choice. The format table is itself the contract: every receiver (and every federated peer) compares the `DerivativeManifest.format` value against this list, and an unknown value is a structural rejection. diff --git a/capsule-docs/src/content/docs/design/versioning.md b/capsule-docs/src/content/docs/design/versioning.md index e15b7fbf..9d3c3bb9 100644 --- a/capsule-docs/src/content/docs/design/versioning.md +++ b/capsule-docs/src/content/docs/design/versioning.md @@ -1,6 +1,7 @@ --- title: Versioning description: How Capsule pins each album to a protocol version, upgrades safely, and bounds client deprecation +status: draft --- Changes are inevitable. Capsule minimizes breaking changes but generously accepts compatible ones. The aim is backward-compatible reads forever and a deliberately fail-closed write path — a [version-mismatched client](/design/threat-model/) never silently corrupts state; it is rejected at the handshake. diff --git a/capsule-docs/src/content/docs/design/web-upload.md b/capsule-docs/src/content/docs/design/web-upload.md index e22e9b31..8e077aa9 100644 --- a/capsule-docs/src/content/docs/design/web-upload.md +++ b/capsule-docs/src/content/docs/design/web-upload.md @@ -1,6 +1,7 @@ --- title: Web Upload description: Browser-based guest drops — upload-link provisioning, the sealed drop format, and adoption into the library +status: draft --- Web upload lets a Capsule user accept assets from someone who has **no Capsule account** — a guest with a browser, the *web client*. Unlike a native client, the web client holds no album keys, has no place in any album's MLS group, and (being WASM in a browser) cannot run the full signed-import pipeline. So it does not write into the library directly. Instead the user **provisions an upload link**; the guest's browser **encrypts each asset client-side and seals the asset key to the user's public key**; the sealed bytes land in a **staging inbox** charged to the provisioning user's quota; and nothing becomes a library asset until the user **reviews and adopts it on one of their native (trusted) clients**, where provenance is finally appended. The guest is the [non-registered account](/design/authentication/#account-types) class — no master key, no User IK, no MLS membership. From 747029fb9eeeb593c13f46a7e31bf7726ab4983c Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:27:41 -0400 Subject: [PATCH 17/58] docs(design): specify the upload protocol as a complete strict contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The upload protocol is the contract three parallel slices (server hardening, SDK client, web drops) build against, so every behavior a client can observe is now written down: a full error taxonomy with machine-readable codes, a strictness table (unknown fields, empty chunks, wrong media types, and missing checksums are rejected — a buggy client fails loudly on its first bad request instead of corrupting state later), explicit edge cases, and a per-endpoint header census. Contract changes, each with its reason: - Chunk storage is append-only: one file per session, chunks appended in order. Sequential offsets were already mandatory, so the .part-file + reflink assembly step bought nothing and added a failure scenario (and a Linux-only fast path); it is gone from the design. - Session lifetime is a floor and a cap: >=1 hour guaranteed since last accepted chunk, 24 h maximum, and the server may discard in between as space needs reclaiming. A discarded session is a uniform 404 (no tombstones), matching the platform-wide indistinguishable-404 rule. - PATCH bodies are application/octet-stream (415 otherwise) — the payload is opaque ciphertext; TUS v2's application/partial-upload would claim a compatibility we don't implement. - X-Capsule-Checksum is required: the (upload_id, offset, chunk_hash) idempotency tuple that defends against garbage retries is undefined without it. - Chunk bounds [4 KiB, 16 MiB] become protocol surface; suggested sizes and adaptation tiers are explicitly non-normative client tuning, with the adaptive algorithm documented once and its enforcement locus stated (server rejects, capsule-sdk adapts). - Duplicate session creation is split by state (active -> return session; finalized -> 409 duplicate_blob + asset ref), fixing a self-contradiction between the invariants and the dedup section. - HEAD now reports state via X-Capsule-Upload-Status (a HEAD response cannot carry the JSON body the old text promised); session listing is uploader-scoped (the uploader is the party that resumes). - filename/date leave the wire request: plaintext metadata contradicted the 'no metadata smuggled' rule, and date was silently dropped anyway. Invariants 10 and 12 are amended in place (no renumbering; scenario tables keep their references), and the filesystem docs' .part/assembly mentions are updated in lockstep. --- .../src/content/docs/design/api-surfaces.md | 1 + .../content/docs/design/filesystem/index.md | 2 +- .../docs/design/filesystem/maintenance.md | 8 +- .../content/docs/design/filesystem/server.md | 5 +- .../docs/design/import/download-sync.md | 2 +- .../content/docs/design/import/pipeline.md | 2 +- .../docs/design/import/upload-protocol.md | 219 ++++++++++++++---- .../docs/design/threat-model/scenarios.md | 4 +- .../docs/design/threat-model/validation.md | 10 +- .../src/content/docs/design/web-upload.md | 5 +- 10 files changed, 190 insertions(+), 68 deletions(-) diff --git a/capsule-docs/src/content/docs/design/api-surfaces.md b/capsule-docs/src/content/docs/design/api-surfaces.md index 0a41ab41..837da7bf 100644 --- a/capsule-docs/src/content/docs/design/api-surfaces.md +++ b/capsule-docs/src/content/docs/design/api-surfaces.md @@ -69,6 +69,7 @@ REST rejections are HTTP statuses; gRPC rejections are gRPC status codes. The ma | Not found (and indistinguishable-404 surfaces) | `404` | `NOT_FOUND` | | Stale state (chain/cursor/directory regress) | `409` | `FAILED_PRECONDITION` | | Payload too large | `413` | `RESOURCE_EXHAUSTED` | +| Unsupported media type (chunk body) | `415` | `INVALID_ARGUMENT` | | Protocol version outside `[Min, Max]` | `426` | `FAILED_PRECONDITION` | | Rate limited | `429` | `RESOURCE_EXHAUSTED` | diff --git a/capsule-docs/src/content/docs/design/filesystem/index.md b/capsule-docs/src/content/docs/design/filesystem/index.md index dbd69663..09608101 100644 --- a/capsule-docs/src/content/docs/design/filesystem/index.md +++ b/capsule-docs/src/content/docs/design/filesystem/index.md @@ -25,7 +25,7 @@ These follow directly from [Core Principles](/design/principles/): - **Recovery-first.** No database is required to interpret canonical data. On the client, sidecar files are the source of truth and the index is a rebuildable cache. On the server, PostgreSQL is the authoritative index, but it holds only key-free facts. - **Atomic writes.** Every write that must not tear uses temp-file + atomic rename on the same filesystem. Direct overwrites risk corruption on power loss. The full per-granularity rules live in [Maintenance — Atomic Writes](/design/filesystem/maintenance/#atomic-writes-and-crash-recovery). - **Ephemeral derived data.** Only originals and their canonical metadata are irreplaceable. Thumbnails, transcodes, parsed-metadata caches, and the query index can all be regenerated and are treated as such. -- **4 KiB alignment.** Data is processed and written block-aligned to 4 KiB, which matches memory and disks and enables the [reflink assembly path](/design/import/upload-protocol/#server-side-storage-and-assembly). +- **4 KiB alignment.** Data is processed and written block-aligned to 4 KiB, which matches memory and disks and keeps the [append-only upload path](/design/import/upload-protocol/#append-only-storage) page-aligned. - **Content-addressing.** Stored blobs are named by their ciphertext content hash — the same hash everywhere a content address is needed (see [Cryptography — Primitives](/design/cryptography/primitives/)). ## Server vs Client at a Glance diff --git a/capsule-docs/src/content/docs/design/filesystem/maintenance.md b/capsule-docs/src/content/docs/design/filesystem/maintenance.md index 5ecdfaa5..bd241793 100644 --- a/capsule-docs/src/content/docs/design/filesystem/maintenance.md +++ b/capsule-docs/src/content/docs/design/filesystem/maintenance.md @@ -6,13 +6,13 @@ status: draft The data-integrity principle treats client storage as *potentially lost* (see [Core Principles](/design/principles/)): unlike the server, a client library sits on consumer hardware, syncs only partially, and is edited by a long-lived process that can be killed mid-write. A client therefore never assumes its library is consistent — it periodically *proves* it is, repairs what it can repair safely, and surfaces what it cannot. -The maintenance routines live in `capsule-core::library`: [`scrub`](#scrubbing), [self-validation](#self-validation), [repair](#repair), and [`dedup`](#deduplication). The server runs an equivalent scrub of stale `.part`/`.bin` files under `incoming/`. All routines are **conservative** — consistent with "we can NEVER delete data unexpectedly," irreplaceable data is never removed without explicit user confirmation. +The maintenance routines live in `capsule-core::library`: [`scrub`](#scrubbing), [self-validation](#self-validation), [repair](#repair), and [`dedup`](#deduplication). The server runs an equivalent scrub of stale upload files under `incoming/`. All routines are **conservative** — consistent with "we can NEVER delete data unexpectedly," irreplaceable data is never removed without explicit user confirmation. This doc also owns the granularity rules for [atomic writes](#atomic-writes-and-crash-recovery), which other docs reference but should not restate. ## Scrubbing -A startup **scrub** sweeps the debris of interrupted writes. Atomic writes (below) stage to `.tmp` files; a crash between the write and the rename strands them. The scrub walks `media/` and removes `.tmp` files older than **10 minutes** (configurable) — the age floor avoids racing a write that is legitimately in flight elsewhere in the process. It runs at most once every seven days, gated by a `last_scrubbed_at` timestamp in the library config, since stale temp files are harmless clutter rather than an urgent fault. Every removal is logged. The server performs the equivalent sweep of stale `.part`/`.bin` files (see [Atomic Writes and Crash Recovery](#atomic-writes-and-crash-recovery)). +A startup **scrub** sweeps the debris of interrupted writes. Atomic writes (below) stage to `.tmp` files; a crash between the write and the rename strands them. The scrub walks `media/` and removes `.tmp` files older than **10 minutes** (configurable) — the age floor avoids racing a write that is legitimately in flight elsewhere in the process. It runs at most once every seven days, gated by a `last_scrubbed_at` timestamp in the library config, since stale temp files are harmless clutter rather than an urgent fault. Every removal is logged. The server performs the equivalent sweep of stale `incoming/*.bin` upload files (see [Atomic Writes and Crash Recovery](#atomic-writes-and-crash-recovery)). ## Self-Validation @@ -74,10 +74,10 @@ Every write that must not tear uses temp-file + atomic rename, staged on the sam - **Client — single-file writes.** Sidecar and provenance appends stage to `{uuid}.cbor.tmp` and `{uuid}.provenance.cbor.tmp` in the destination directory, then rename into place. A direct overwrite is never used. - **Client — per-asset bundle.** An asset import or update is a *bundle*: original (when present locally), sidecar, and a new provenance record. All `.tmp` files stage first; only after every staged file is on disk do the renames execute, and only in a fixed order (original → sidecar → provenance). A **rewrite** bundle — a `metadata-update` or `replace` whose rename lands over an existing file — first stages the displaced prior version aside (`{uuid}.cbor.prev`, same directory), so rollback can *restore* it rather than merely delete the new target; the `.prev` staging is removed only after the bundle's final rename commits. A failure at any rename then discards every remaining `.tmp` and rolls back the renames already done — deleting freshly-created targets, restoring `.prev` files over rewritten ones — so the on-disk state never reflects a partial bundle and an interrupted rewrite can never lose the pre-edit state. The `.provenance.cbor` is the last to be renamed, so the existence of a new provenance record implies the rest of the bundle is committed. - **Client — stack edit.** A stack edit touches multiple sidecars and writes a single provenance record per affected asset. All `.tmp` files (one per sidecar plus one per provenance file) stage first and rename together; any rename failure discards the entire batch. There is no partial stack. -- **Server — chunk assembly.** Chunks stage as `{upload_id}_{n}.part`; the assembled blob is `{upload_id}.bin`. The blob is renamed into its content-addressed location under `blobs/` only after the ciphertext hash is recomputed and matches the declared value (see [Upload Protocol — Finalization and Integrity](/design/import/upload-protocol/#finalization-and-integrity)). +- **Server — append-only upload file.** Each upload session appends into a single `{upload_id}.bin`. The file is renamed into its content-addressed location under `blobs/` only after the ciphertext hash is recomputed and matches the declared value (see [Upload Protocol — Finalization and Integrity](/design/import/upload-protocol/#finalization-and-integrity)); an interrupted session leaves exactly one file to sweep. - **Server — finalization transaction.** The blob rename into its content-addressed `blobs/` location is a filesystem operation and so necessarily happens *before* the Postgres commit; the manifest-envelope insert, metadata-blob insert, provenance-blob insert, and asset-row `uploaded` flip then commit in a **single PostgreSQL transaction**. That ordering is what makes every crash point safe: a crash *before* the rename leaves only `incoming/` debris (scrubbed below); a crash *after* the rename but *before* the commit leaves a finalized blob in `blobs/` that **no committed row references** — an orphan the [reference-count GC](/design/filesystem/server/#deletion-and-garbage-collection) reclaims, while the idempotent retry re-finalizes against the already-present blob (re-placing a content-addressed hash is a no-op). The "single transaction" guarantee is over the **index rows**; blob *placement* is idempotent and GC-safe precisely because it is content-addressed. The server never exposes an asset whose index bundle is partially persisted — the session stays in `WaitingForProcessing` until a finalization attempt commits the whole bundle or fails it cleanly. -On startup, each side scrubs incomplete work: stale `.part`, `.tmp`, and `.bin` files left by an interrupted upload or import are identified and removed, and the cleanup is logged. A blob or media file is never published, on either side, until its integrity has been verified. +On startup, each side scrubs incomplete work: stale `.tmp` and `incoming/*.bin` files left by an interrupted upload or import are identified and removed (a `.bin` shorter than its session's recorded offset fails the session — the file is authoritative), and the cleanup is logged. A blob or media file is never published, on either side, until its integrity has been verified. ## Encrypted Backups diff --git a/capsule-docs/src/content/docs/design/filesystem/server.md b/capsule-docs/src/content/docs/design/filesystem/server.md index 37149d70..56875a17 100644 --- a/capsule-docs/src/content/docs/design/filesystem/server.md +++ b/capsule-docs/src/content/docs/design/filesystem/server.md @@ -30,7 +30,6 @@ Switching profiles is operationally invisible to clients — the [upload protoco ```text {blob_root}/ ├── incoming/ -│ ├── {upload_id}_{n}.part # in-flight chunk │ └── {upload_id}.bin # assembled blob, pre-verification ├── blobs/ │ └── {hash[0:2]}/{hash[2:4]}/ @@ -41,7 +40,7 @@ Switching profiles is operationally invisible to clients — the [upload protoco ``` - **`{blob_root}`**: absolute path configured at server startup. The entire tree must be on a single filesystem so that finalization renames are atomic. -- **`incoming/`**: live uploads. Chunks land as `{upload_id}_{n}.part`; on finalization they are concatenated into `{upload_id}.bin`. The 4 KiB chunk alignment is what allows each chunk to be reflinked into place on copy-on-write filesystems, turning assembly into a near-instant metadata operation. See the upload protocol in [Import — Upload Protocol](/design/import/upload-protocol/). +- **`incoming/`**: live uploads. Each session owns a single append-only file `{upload_id}.bin`; accepted chunks are appended in order, and the 4 KiB chunk alignment keeps every write block-aligned. There is no per-chunk staging and no assembly step. See [Import — Upload Protocol: Append-Only Storage](/design/import/upload-protocol/#append-only-storage). - **`blobs/`**: the finalized store. A blob's filename is its [ciphertext content hash](/design/cryptography/primitives/); the two-level hex-prefix shard keeps directory sizes bounded for multi-million-blob stores. A finalized blob is immutable. - **`.server/`**: the server operator's own configuration and schema version. This is plaintext server metadata, not user data — it is the one thing under `{blob_root}` that is not an encrypted blob. @@ -90,7 +89,7 @@ The server index records only what can be known without a key: No plaintext capture date, dimensions, EXIF, tags, or filename ever reaches the server. Those live inside the encrypted metadata blob (see [Metadata Encryption](/design/cryptography/encryption/#metadata-encryption)) and are readable only by authorized clients. -Session creation writes a *pending* asset row (`uploaded = false`) that reserves the asset ID the bundle's blobs reference; finalization flips it. See the [session lifecycle](/design/import/upload-protocol/#session-lifecycle). +Session creation writes a *pending* asset row (`uploaded = false`) that reserves the asset ID the bundle's blobs reference; finalization flips it. See the [session state machine](/design/import/upload-protocol/#session-state-machine). ## Ownership, Partitioning, and Quota diff --git a/capsule-docs/src/content/docs/design/import/download-sync.md b/capsule-docs/src/content/docs/design/import/download-sync.md index 086ae5ac..ca11c018 100644 --- a/capsule-docs/src/content/docs/design/import/download-sync.md +++ b/capsule-docs/src/content/docs/design/import/download-sync.md @@ -65,7 +65,7 @@ On mobile clients, auto syncing keeps new assets backed up (not to be confused w ### Synchronization Criteria -Sync is checked conservatively. When a check fires, the client reconciles everything that needs syncing — uploads and downloads — and proceeds as long as the criteria below hold throughout the transfer. If conditions change mid-transfer (e.g. the connection becomes metered), it re-evaluates and pauses gracefully; the server never assumes a transfer runs to completion in one session (see [Upload Protocol — Robustness](/design/import/upload-protocol/#robustness)). +Sync is checked conservatively. When a check fires, the client reconciles everything that needs syncing — uploads and downloads — and proceeds as long as the criteria below hold throughout the transfer. If conditions change mid-transfer (e.g. the connection becomes metered), it re-evaluates and pauses gracefully; the server never assumes a transfer runs to completion in one session (see [Upload Protocol — Idempotency and Resumption](/design/import/upload-protocol/#idempotency-and-resumption)). The actual synchronization criteria are strict and scale with the reconciliation amount (i.e. total upload + download transfer): diff --git a/capsule-docs/src/content/docs/design/import/pipeline.md b/capsule-docs/src/content/docs/design/import/pipeline.md index 37187915..5b0b1857 100644 --- a/capsule-docs/src/content/docs/design/import/pipeline.md +++ b/capsule-docs/src/content/docs/design/import/pipeline.md @@ -71,7 +71,7 @@ Peak local disk is therefore bounded to the in-flight window, not the whole impo **Minimum headroom.** The window must fully materialize at least one asset locally (original + derivatives + metadata) before that asset's upload and release complete, so streaming mode requires headroom for the **largest single asset in the plan**. If even one asset exceeds free space, the plan surfaces a hard "cannot import *X* without freeing *Y*" error at confirmation rather than stalling mid-stream — streaming bounds peak usage to the window; it cannot make a single file smaller than the disk. -**Halt on lost connectivity.** Streaming mode depends on the server: releasing local copies is safe only because the server is confirmed to hold them. If the connection to the server drops, the pipeline **pauses** — it stops admitting new source files into the library (continuing would refill the very disk the mode exists to spare) and waits, rather than importing ahead into space it does not have. In-flight uploads resume via the protocol's [`HEAD` resumption](/design/import/upload-protocol/#session-lifecycle) once connectivity returns; if the source media itself is removed mid-import, the deterministic planner re-derives the remaining work on resume. Bulk streaming import is a *large reconciliation* and obeys the same metered/Wi-Fi [connection rules](/design/import/download-sync/#synchronization-criteria) as auto-sync. +**Halt on lost connectivity.** Streaming mode depends on the server: releasing local copies is safe only because the server is confirmed to hold them. If the connection to the server drops, the pipeline **pauses** — it stops admitting new source files into the library (continuing would refill the very disk the mode exists to spare) and waits, rather than importing ahead into space it does not have. In-flight uploads resume via the protocol's [`HEAD` resumption](/design/import/upload-protocol/#session-state-machine) once connectivity returns; if the source media itself is removed mid-import, the deterministic planner re-derives the remaining work on resume. Bulk streaming import is a *large reconciliation* and obeys the same metered/Wi-Fi [connection rules](/design/import/download-sync/#synchronization-criteria) as auto-sync. **Quota.** Streaming creates one upload session per asset, so server [quota](/design/quota/#enforcement-points) is enforced exactly as for any upload — at session creation, no new enforcement point. If quota is exhausted mid-stream the next session creation is refused; the pipeline pauses and surfaces the remediable state rather than failing the whole import, and no local original is released for an asset that did not upload. diff --git a/capsule-docs/src/content/docs/design/import/upload-protocol.md b/capsule-docs/src/content/docs/design/import/upload-protocol.md index 14da256d..abc60956 100644 --- a/capsule-docs/src/content/docs/design/import/upload-protocol.md +++ b/capsule-docs/src/content/docs/design/import/upload-protocol.md @@ -4,10 +4,12 @@ description: The wire protocol between Capsule clients and the server for resuma status: draft --- -The upload protocol is a custom resumable-upload protocol modeled on [TUS](https://tus.io/) but trimmed to Capsule's needs: no per-request capability negotiation, no metadata smuggled in headers, ciphertext-only payloads. Compatibility is gated once, up front, via the universal [protocol handshake](/design/threat-model/validation/#protocol-and-capability-negotiation). +The upload protocol is a custom resumable-upload protocol modeled on [TUS](https://tus.io/) but trimmed to Capsule's needs: no per-request capability negotiation, no metadata smuggled in headers, ciphertext-only payloads. Compatibility is gated once, up front, via the universal [protocol handshake](/design/threat-model/validation/#protocol-and-capability-negotiation). (We follow TUS v1's offset/PATCH model, not the TUS v2 draft — see the media-type note in [Chunk Rules](#chunk-rules-and-strictness).) This protocol is the most fragile contract between client and server: a client that misunderstands chunk alignment, offset semantics, or finalization can silently corrupt or orphan data. The endpoint table, the chunk rules, the session state machine, and the finalization steps below **are the contract** — every implementation MUST conform exactly. The client implementation lives in `capsule-sdk::upload`; the server in `capsule-api-upload`. The two are tested independently against the protocol surface. +The protocol is **strict by construction**: any request that violates the contract is rejected with a machine-readable error, never silently corrected or partially honored. Leniency here would hide client bugs until they corrupt data; strictness surfaces them as loud, attributable rejections on the first bad request. The [strictness table](#chunk-rules-and-strictness) enumerates every tolerated-vs-rejected behavior, and the [error taxonomy](#error-taxonomy) names the code a client receives for each. + Every upload is **idempotent** but stateful. Uploads can complete partially and are identified by an *upload ID*. ## What Gets Uploaded @@ -18,7 +20,7 @@ An asset is never uploaded as a single plaintext file. Because Capsule is end-to - **Derivative blobs** — thumbnails and previews, generated client-side during import (see [Thumbnails](/design/thumbnails/)), each encrypted independently. - The **metadata blob** — the CBOR metadata document (capture date, dimensions, EXIF-derived fields, the [LQIP](/design/thumbnails/#lqip), provenance), encrypted under the [bulk AEAD](/design/cryptography/primitives/#bulk-aead) (see [Metadata](/design/metadata/)). -Each blob is its own upload with its own upload ID; the protocol does not couple them and imposes **no wire ordering**. The client may transfer the bundle in any order — decoupling lets small derivatives land while a large original is still in flight — but the server **gates visibility** on the pending-asset row: the asset becomes visible to other devices only once its **required members, the original and the metadata blob, are both finalized**. This is enforced without reading plaintext — each blob's role is recorded on its pending row at session creation, and the visibility flip simply checks that the original and metadata roles are present and `uploaded`. Every blob in the bundle — original, derivatives, metadata, provenance — counts toward the uploader's storage [quota](/design/quota/#accounting-model). +Each blob is its own upload with its own upload ID; the protocol does not couple them and imposes **no wire ordering**. Every session declares its blob's **role** at creation — a closed enum `original | derivative | metadata | provenance | backup` in the `POST /upload` body, recorded on the pending row — so the server can reason about bundle completeness without reading plaintext. The client may transfer the bundle in any order — decoupling lets small derivatives land while a large original is still in flight — and the server **gates visibility** on the pending-asset row: the asset becomes visible to other devices once its **manifest and metadata blob are finalized**. Whether the original is also held yet travels as a per-asset completeness fact on the sync feed (`original_held`), which is what makes [staged uploads](/design/import/download-sync/) possible; an asset whose original has not landed is in the derived `awaiting-original` state, and fetching its original returns the transient `error.blob.pending_upload` rather than `410`. Every blob in the bundle — original, derivatives, metadata, provenance — counts toward the uploader's storage [quota](/design/quota/#accounting-model). "Blob" is defined once, in [Filesystem — Server: Uniform, Opaque Blobs](/design/filesystem/server/#uniform-opaque-blobs); this protocol is its transport, not its definition. Every asset and derivative blob carries a signed [manifest envelope](/design/cryptography/provenance/#asset-manifest): at `POST /upload` the server validates the envelope's `created_by_device` against the uploader's [device directory](/design/cryptography/keys/#device-directory) (invariant 7), and the client verifies the full write-tier signature on download via [`verify_asset`](/design/cryptography/keys/#write-authorization). **Backup artifacts are the one exception** — they carry no per-asset provenance of their own (the exporting device is not the original author); their integrity rides the library-level backup MANIFEST instead (see [Backup and Recovery](/design/backup-recovery/)). @@ -29,36 +31,68 @@ The server performs no decoding, no metadata extraction, and no thumbnail genera The upload protocol guarantees the following, and every endpoint upholds them: - **Content-addressed.** Every blob is identified by its [ciphertext content hash](/design/cryptography/primitives/). The plaintext hash is never transmitted to the server. -- **Idempotent.** Re-creating a session for a blob already stored is a no-op that resolves to the existing asset. Re-sending a chunk at an already-acknowledged offset is accepted and simply returns the current offset. -- **Resumable.** A session survives connection loss for the lifetime of its TTL. A client resumes by querying the authoritative offset and continuing from there — no bytes are re-sent unnecessarily. +- **Idempotent.** Re-creating a session for a blob with an *active* session returns that session; re-creating one for a hash already *finalized* is rejected with a conflict that names the existing asset so the client resolves it as a [merge](#deduplication-and-merge). Re-sending a chunk with the same idempotency tuple returns the same response. +- **Resumable.** A session is guaranteed to survive at least the [survival floor](#session-lifetime-and-discard) and may survive up to the 24-hour cap. A client resumes by querying the authoritative offset and continuing from there — no bytes are re-sent unnecessarily. +- **Strict.** Unknown request fields, empty chunks, wrong media types, and malformed headers are rejected outright. A buggy client is stopped at its first bad request, not accommodated. - **Strictly bounded.** The total ciphertext size is declared at session creation and immutable thereafter. The cumulative received bytes may never exceed it, nor exceed the server's per-file limit. - **Verified.** No upload is marked complete until the server has recomputed the ciphertext hash and confirmed it matches the declared value. -- **Recoverable.** Every session is either driven to a terminal state or garbage-collected. There are no permanently orphaned chunks or pending asset rows. +- **Recoverable.** Every session is either driven to a terminal state or garbage-collected. There are no permanently orphaned upload files or pending asset rows. ## Endpoints All endpoints are authenticated with a bearer JWT. -| Method | Path | Purpose | -| -------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `POST` | `/upload` | Create a session. Body declares ciphertext `size`, `hash` (the [content hash](/design/cryptography/primitives/) digest bytes; algorithm fixed by `crypto_suite_id`), `content_type` (closed enum), `crypto_suite_id`, `protocol_version`, `manifest_envelope` (the unencrypted manifest fields the server validates per [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants)), optional `album_id`, optional `owner_id`, optional `intent_id` (required only during an [album upgrade](/design/versioning/#album-upgrade-ceremony)). Returns `201` with `Location: /upload/{id}` and `X-Capsule-Suggested-Chunk-Size`. Rejects with `400` / `403` / `426` per the validation invariants. | -| `HEAD` | `/upload/{id}` | Query progress. Returns `X-Capsule-Offset` (next expected byte), `X-Capsule-Content-Length`, and session status. This is the resumption primitive. | -| `PATCH` | `/upload/{id}` | Append a chunk at `X-Capsule-Offset`, with an optional per-chunk `X-Capsule-Checksum`. Returns `204` and the new offset. | -| `DELETE` | `/upload/{id}` | Cancel the session — removes chunks, the session record, and the pending asset row. | -| `GET` | `/upload/sessions` | List the caller's active sessions, so a client can resume across app restarts or devices. | +| Method | Path | Purpose | +| -------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `POST` | `/upload` | Create a session. The JSON body declares ciphertext `size`, `hash` (lowercase hex; digest length fixed by `crypto_suite_id`), `content_type` (closed enum), `crypto_suite_id`, `protocol_version`, `blob_role` (closed enum), `manifest_envelope` (the unencrypted manifest fields the server validates per [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants)), optional `album_id`, optional `owner_id`, optional `intent_id` (required only during an [album upgrade](/design/versioning/#album-upgrade-ceremony)). Unknown fields are rejected. Returns `201` with `Location: /upload/{id}` and `X-Capsule-Suggested-Chunk-Size`. Rejects per the [error taxonomy](#error-taxonomy). | +| `HEAD` | `/upload/{id}` | Query progress. Returns `X-Capsule-Offset` (next expected byte), `X-Capsule-Content-Length` (declared total), and `X-Capsule-Upload-Status` (the session state). No body — HTTP forbids one on `HEAD`, which is why status rides a header. This is the resumption primitive. | +| `PATCH` | `/upload/{id}` | Append a chunk at `X-Capsule-Offset`, with `Content-Type: application/octet-stream` and the **required** per-chunk `X-Capsule-Checksum`. Returns `204` and the new offset. | +| `DELETE` | `/upload/{id}` | Cancel the session — removes the upload file, the session record, and the pending asset row. Rejected with `409` while finalization is running. | +| `GET` | `/upload/sessions` | List the **uploader's** active sessions (keyed by `upload_user_id` — the resuming party — not `owner_id`), so a client can resume across app restarts. | + +Creating a session writes a *pending* asset row to Postgres (`uploaded = false`) and a session record to the configured **session-state store** (see [Filesystem — Server: Deployment Profiles](/design/filesystem/server/#deployment-profiles): Postgres by default, Valkey in the high-concurrency profile). The pending row reserves the asset ID that derivative and metadata blobs reference. The session record carries everything finalization needs — declared size and hash, `content_type`, `crypto_suite_id`, `protocol_version`, `blob_role`, the manifest envelope, `album_id`/`owner_id`/`intent_id`, and the uploader — so a session is finalizable from its own record with no further client input. + +Per-endpoint header census (the `X-Capsule-*` namespace itself is owned by the [universal header census](/design/threat-model/validation/#universal-headers)): -Creating a session writes a *pending* asset row to Postgres (`uploaded = false`) and a session record to the configured **session-state store** (see [Filesystem — Server: Deployment Profiles](/design/filesystem/server/#deployment-profiles): Postgres by default, Valkey in the high-concurrency profile). The pending row reserves the asset ID that derivative and metadata blobs reference. +| Header | Direction | Where | Required | +| ------------------------------- | --------- | ------------------------ | -------- | +| `X-Capsule-Protocol` (+ `-Min`/`-Max` on responses) | both | every request/response | yes | +| `X-Capsule-Suggested-Chunk-Size` | response | `POST /upload` | yes | +| `X-Capsule-Offset` | both | `PATCH` request; `HEAD`/`PATCH`/`409` responses | yes | +| `X-Capsule-Content-Length` | response | `HEAD` | yes | +| `X-Capsule-Upload-Status` | response | `HEAD` | yes | +| `X-Capsule-Checksum` | request | `PATCH` | **yes** | -## Chunk Rules +## Chunk Rules and Strictness Enforced strictly; a violation fails the request rather than being silently corrected: -- Every chunk except the final one MUST be a multiple of 4 KiB (4096 bytes). This keeps server-side writes block-aligned, which is what makes the [reflink assembly path](#server-side-storage-and-assembly) work. A non-aligned, non-final chunk is rejected with `400`. -- Offsets are strictly sequential. A `PATCH` must arrive at exactly the current received-byte count; an out-of-order or gapped write is rejected with `409`, and the client recovers by issuing `HEAD` to learn the authoritative offset. -- **Idempotency tuple.** The server keys each accepted PATCH by `(upload_id, offset, chunk_hash)` where `chunk_hash` is the SHA-256 of the chunk bytes (carried in the `X-Capsule-Checksum` header). A duplicate PATCH with the same tuple returns the same response — a re-send after a lost ACK is a no-op. A PATCH at an already-acknowledged offset *with a different `chunk_hash`* is rejected with `409` + a corruption error: this is the structural defense against a faulty client that retries with garbage. The complete idempotency contract is owned by [Threat Model — Idempotency Invariants](/design/threat-model/validation/#idempotency-invariants). -- Cumulative size may never exceed the declared `size` nor the server's `max_file_size`. The server checks the cumulative count **at every chunk arrival**, not only at finalization — a buggy client that streams past the declared size is cut off before more bytes are persisted. Either ceiling is rejected (`400` / `413`) and the session is moved to a failed state. +- **Media type.** A `PATCH` body MUST be `Content-Type: application/octet-stream` (parameters ignored) — the payload is literally opaque ciphertext bytes. Anything else, or a missing `Content-Type`, is rejected with `415`. We deliberately do not use the TUS v2 draft's `application/partial-upload`: we implement TUS v1 semantics with our own headers, and advertising v2's media type would claim a compatibility we do not have. +- **Non-empty.** An empty `PATCH` body is rejected with `400`. (An empty chunk can only be a client bug; accepting it as a no-op would mask that bug.) +- **Alignment.** Every chunk except the final one MUST be a multiple of 4 KiB (4096 bytes). This keeps server-side writes page/block-aligned — friendly to direct and batched I/O on the [append path](#append-only-storage) — and doubles as a cheap tripwire for client offset-arithmetic bugs: a client whose chunking drifts mis-aligns long before it corrupts anything. The final chunk is exempt because the blob's total size is arbitrary. A non-aligned, non-final chunk is rejected with `400`. (The [encryption layer's](/design/cryptography/encryption/#stream-construction) 65,536-byte ciphertext stride is a separate concern; 65,536 is a clean multiple of 4,096, so crypto-chunked ciphertext is transport-alignable by construction.) +- **Bounds.** A chunk is at least 4 KiB (except the final chunk) and at most **16 MiB**; a larger body is rejected with `413`. These bounds are protocol surface (see [Protocol Versioning](#protocol-versioning)); the *suggested* sizes and adaptation tiers are not. +- **Sequential offsets.** A `PATCH` must arrive at exactly the current received-byte count; an out-of-order or gapped write is rejected with `409` carrying the authoritative offset, and the client recovers by re-aligning (or issuing `HEAD`). +- **Required checksum + idempotency tuple.** Every `PATCH` carries `X-Capsule-Checksum`: the SHA-256 of the chunk bytes as bare lowercase hex. Missing or malformed → `400`. The server verifies the body against the header **before any write**; a mismatch is `400` with nothing persisted (the transit-corruption defense). The server keys each accepted PATCH by `(upload_id, offset, chunk_hash)`: a duplicate PATCH with the same tuple returns the same response — a re-send after a lost ACK is a no-op — while a PATCH at an already-acknowledged offset *with a different `chunk_hash`* is rejected with `409` + a corruption error: the structural defense against a faulty client that retries with garbage. The checksum is required precisely because this tuple is undefined without it. The complete idempotency contract is owned by [Threat Model — Idempotency Invariants](/design/threat-model/validation/#idempotency-invariants). +- **Cumulative bounds.** Cumulative size may never exceed the declared `size` nor the server's `max_file_size`. The server checks the cumulative count **at every chunk arrival**, not only at finalization — a buggy client that streams past the declared size is cut off before more bytes are persisted. Either ceiling is rejected (`400` / `413`) and the session is moved to `FailedProcessing`. - The upload completes exactly when received bytes equal the declared size; finalization then runs automatically. +Strictness applies to the **transport layer** — the JSON request bodies (unknown fields rejected) and the header set above. It does not extend into the CBOR interiors (manifests, sidecars), where the schema rules deliberately require clients to *preserve* unknown keys for forward compatibility; that asymmetry is [Postel's law as scoped by Core Principles](/design/principles/#postels-law-asymmetric): be strict on the wire you own, tolerant inside the documents that outlive you. + +| Candidate leniency | Behavior | Rejection | +| ------------------------------------------- | ------------ | ---------------------------------------------- | +| Unknown JSON field on `POST /upload` | **rejected** | `400 error.upload.malformed_request` | +| Empty `PATCH` body | **rejected** | `400 error.upload.empty_chunk` | +| Wrong / missing `Content-Type` on `PATCH` | **rejected** | `415 error.upload.unsupported_media_type` | +| Missing / malformed `X-Capsule-Checksum` | **rejected** | `400 error.upload.missing_checksum` | +| Checksum ≠ body hash | **rejected** | `400 error.upload.checksum_mismatch` (no write)| +| Unaligned non-final chunk | **rejected** | `400 error.upload.chunk_not_aligned` | +| Chunk > 16 MiB | **rejected** | `413 error.upload.chunk_too_large` | +| Stale / gapped offset | **rejected** | `409 error.upload.offset_mismatch` + offset | +| Replay, same idempotency tuple | *accepted* | same response as the original (no-op) | +| Same offset, different checksum | **rejected** | `409 error.upload.chunk_conflict` | +| Bytes past declared `size` | **rejected** | `400 error.upload.size_exceeded` → failed | +| `PATCH`/`DELETE` on a finalizing session | **rejected** | `409 error.upload.session_not_active` | + ## Protocol Versioning The upload session is gated by Capsule's universal [protocol handshake](/design/threat-model/validation/#protocol-and-capability-negotiation), so a client never begins a transfer against a server it is not known to be compatible with. This section names the upload-specific specializations. @@ -69,9 +103,9 @@ Versioning is **date-based** (`YYYY-MM-DD` — the day a protocol revision is fr - A `POST /upload` whose version falls outside the accepted range is rejected with `426 Upgrade Required` *before* any session or pending asset row is created. The response names the supported range so the client can show an actionable message ("update Capsule to keep uploading"). Per [Threat Model](/design/threat-model/validation/#protocol-and-capability-negotiation), the same rule applies to every other write surface. - This is a one-shot **compatibility gate**, not negotiation: there is no back-and-forth to settle on a shared version, and the protocol carries no capability flags. A client either speaks a version the server accepts, or it does not upload. - The server supports a *window* of past protocol versions, not only the newest, so a staggered client rollout keeps working. A version leaves the window only after the deprecation period defined in [Threat Model — Min-Supported-Client Deprecation Policy](/design/threat-model/schema-rules/#min-supported-client-deprecation-policy); dropping one is a breaking change announced ahead of time. -- The date is bumped only for an **incompatible** wire change — offset semantics, alignment rules, finalization, the state machine. Purely additive, safely-ignorable changes do not bump it, and server-tunable parameters such as suggested chunk sizes and adaptive-sizing tiers are not protocol surface at all. +- The date is bumped only for an **incompatible** wire change — offset semantics, alignment rules, the hard chunk bounds (raising the 16 MiB maximum is additive; lowering it or raising the 4 KiB minimum is breaking), finalization, the state machine. Purely additive, safely-ignorable changes do not bump it, and server-tunable parameters such as suggested chunk sizes and adaptive-sizing tiers are not protocol surface at all. -## Session Lifecycle +## Session State Machine A session moves through a strict state machine: @@ -80,60 +114,143 @@ Pending ─▶ Uploading ─▶ WaitingForProcessing ─▶ Completed └─▶ FailedProcessing ``` +| From | To | On | +| ---------------------- | ---------------------- | --------------------------------------------------------- | +| `Pending` | `Uploading` | first **accepted** chunk (rejected chunks do not transition) | +| `Uploading` | `WaitingForProcessing` | received bytes == declared `size` | +| `Pending`/`Uploading` | *(discarded)* | TTL expiry or pressure eviction (see below) | +| `WaitingForProcessing` | `Completed` | recomputed hash matches; pending row marked uploaded | +| `WaitingForProcessing` | `FailedProcessing` | hash mismatch, size mismatch, or envelope re-validation failure | + - **Pending** — session created, no bytes received. -- **Uploading** — at least one chunk received, transfer in progress. -- **WaitingForProcessing** — all declared bytes received; finalization (assembly + hash verification) is running. -- **Completed** — hash verified, asset marked uploaded, now visible to other devices. Terminal. -- **FailedProcessing** — terminal failure (hash mismatch, assembly error). Chunks and the pending asset row are removed. Terminal. +- **Uploading** — at least one chunk accepted, transfer in progress. The transition is observable: a `HEAD` reflects it in `X-Capsule-Upload-Status`. +- **WaitingForProcessing** — all declared bytes received; finalization (hash verification) is running. Not evictable, not cancellable. +- **Completed** — hash verified, asset marked uploaded. Terminal. +- **FailedProcessing** — terminal failure. The upload file and the pending asset row are removed at the transition; the session *status record* is retained as a receipt. Terminal. + +Session records live in the [session-state store](/design/filesystem/server/#deployment-profiles) with an uploader-keyed index for listing. This split is intentional: the session store holds only volatile transfer state, so the hot path — offset increments and status transitions — never touches the durable Postgres asset row. (In the default Postgres-only profile, sessions live in an `upload_sessions` table with an `expires_at` column and a periodic sweep; in the high-concurrency profile, they live in Valkey under keys `upload:session:{id}` with atomic `HINCRBY`/`HSET` and native TTL.) Postgres's durable asset record is written exactly twice per upload regardless of profile: once at session creation (the pending row) and once at finalization (mark uploaded). -Session records live in the [session-state store](/design/filesystem/server/#deployment-profiles) with a 24-hour TTL and a per-owner index for listing. This split is intentional: the session store holds only volatile transfer state, so the hot path — offset increments and status transitions — never touches the durable Postgres asset row. (In the default Postgres-only profile, sessions live in an `upload_sessions` table with an `expires_at` column and a periodic sweep; in the high-concurrency profile, they live in Valkey under keys `upload:session:{id}` with atomic `HINCRBY`/`HSET` and native TTL.) Postgres's durable asset record is written exactly twice per upload regardless of profile: once at session creation (the pending row) and once at finalization (mark uploaded). A session that reaches its TTL before completing is garbage-collected — chunks deleted, pending asset row removed — and the client treats an expired session as gone and re-imports, retrying with backoff and halting after a bounded number of attempts. +## Session Lifetime and Discard -**TTL applies only to in-progress transfer.** A session is eligible for TTL eviction only while in `Pending` or `Uploading`. Once it reaches `WaitingForProcessing`, finalization is running and the session is **not** evicted out from under it — finalization either drives it to `Completed` or fails it cleanly to `FailedProcessing`. Both terminal states are **retained for the remainder of the TTL** rather than deleted on transition, so a client whose finalization ACK was lost re-queries (`HEAD`) and observes the terminal outcome — learning the upload already succeeded or failed — instead of seeing a vanished session and blindly re-uploading. (The `FailedProcessing` cleanup of chunks and the pending row happens at the transition; only the session *status record* is what lingers for the TTL.) +A session's lifetime is a **floor and a cap**, not a flat promise: -## Server-Side Storage and Assembly +- **Survival floor (guaranteed).** A session in `Pending` or `Uploading` survives at least **1 hour after its last accepted chunk** (after creation, if none yet). Within the floor it is never evicted, under any pressure — a slow but live upload keeps its session as long as it keeps making progress. +- **Cap.** No session outlives **24 hours from creation**, progressing or not. +- **Discard window (server's right).** Between the floor and the cap, the server MAY discard sessions to reclaim space, least-recently-progressed first (ties broken toward the most on-disk bytes — reclaiming the most space from the most-stalled uploads first). +- **After discard: uniform `404`.** A discarded, expired, or never-existent session is indistinguishable — `HEAD`/`PATCH`/`DELETE` all return `404 error.upload.session_not_found`. No tombstones, no `410`: this matches the platform-wide indistinguishable-404 rule (invariant 26), and the client recovery is identical either way — re-create the session and resume from zero for that blob, retrying with backoff and halting after a bounded number of attempts. +- **Terminal receipts are exempt from pressure eviction** — their upload bytes are already gone (`Completed` renamed the file into the blob store; `FailedProcessing` deleted it), so evicting them reclaims nothing while destroying the lost-ACK receipt. They are removed only at the 24-hour cap. +- **Finalization is never interrupted.** A session in `WaitingForProcessing` is not evicted out from under the finalizer; it is driven to `Completed` or `FailedProcessing`. +- **Startup scrub.** On boot the server reconciles disk against the session store: an upload file with no session is deleted; a session whose file is shorter than its recorded received-byte count is moved to `FailedProcessing` (the file is authoritative — an ACK the disk cannot back must not stand). Discard, expiry, and cancellation all delete the upload file and the pending asset row. -Each chunk is written to disk as `{upload_id}_{n}.part`; the assembled blob is `{upload_id}.bin`. Because this is a hot path, the storage layer is aggressively optimized: +The receipt behavior is why a client whose finalization ACK was lost re-queries (`HEAD`) and observes the terminal outcome — learning the upload already succeeded or failed — instead of seeing a vanished session and blindly re-uploading. -- **Streaming writes.** Chunk bytes are streamed from the request body straight to disk; large transfers must never accumulate in hot memory. On Linux, the write path uses `io_uring`. -- **Reflink assembly.** Finalization concatenates chunks into the final blob with a copy-on-write reflink wherever the filesystem supports one — `FICLONERANGE` on Linux (Btrfs, XFS), `clonefile` on macOS (APFS), the equivalent on ReFS. The 4 KiB chunk alignment is precisely what allows each chunk to be reflinked at its destination offset; only the final (possibly unaligned) chunk needs a plain copy. Reflink turns assembly into a near-instant metadata operation instead of an O(file size) copy. On filesystems without reflink support, the code falls back to a sequential copy. -- **Offloaded blocking work.** Chunk assembly and hashing run on a blocking thread pool, never on the async reactor. -- **Backpressure.** `max_cache_size` bounds the total in-flight upload bytes held on disk; `max_file_size` bounds any single blob. The configuration asserts `max_file_size < max_cache_size` and warns if fewer than ~10 concurrent maximum-size uploads would fit. The distinct task pools — network I/O, file I/O, and hashing — are sized and load-tested independently against realistic hardware limits. +## Append-Only Storage + +Each session owns exactly **one** file, `incoming/{upload_id}.bin`, and every accepted chunk is appended to it in order. There is no per-chunk file and no assembly step — sequential offsets are already a hard protocol rule, so append-only storage is correct by construction, and finalization has one less failure scenario (no reflink/concatenate stage that can half-complete). + +- **Append with cross-check.** Before writing, the server compares the file's current length against the session's expected offset; a divergence is a `500 error.upload.storage_inconsistent` (server-side inconsistency, never the client's fault) and the write is refused. The file length is the on-disk truth; the session counter is its cache. +- **Durability before ACK.** A chunk is flushed to disk before its `204` is sent — an acknowledged byte is a byte on disk. +- **Offloaded blocking work.** Hashing runs on a blocking thread pool, never on the async reactor. (The write path uses plain aligned buffered writes today; `io_uring` is an optimization slot, not contract.) +- **Into the blob store.** Only after [finalization](#finalization-and-integrity) verifies the hash is the file renamed to its content address under `blobs/` — the atomic-rename discipline is owned by [Filesystem — Server](/design/filesystem/server/). +- **Backpressure.** `max_cache_size` bounds the total in-flight upload bytes held on disk; `max_file_size` bounds any single blob. The configuration asserts `max_file_size < max_cache_size` and warns if fewer than ~10 concurrent maximum-size uploads would fit. When the bound is hit, the [discard window](#session-lifetime-and-discard) is the relief valve. ## Finalization and Integrity When received bytes reach the declared size, the server finalizes: -1. Session transitions to **WaitingForProcessing**. -2. Chunks are assembled into the final blob. -3. The server recomputes the [content hash](/design/cryptography/primitives/) over the assembled ciphertext on the blocking pool and compares it to the declared `hash`. -4. **On match** — the pending asset is marked uploaded inside a Postgres transaction and the session transitions to **Completed**. -5. **On mismatch** — the blob and the pending asset row are deleted, the session transitions to **FailedProcessing**, and a checksum-mismatch error is returned. A mismatch is always treated as corruption or tampering and is never silently retried server-side. +1. Session transitions to **WaitingForProcessing** via an **atomic compare-and-set** on the status — two racing finalizers cannot both proceed; the loser observes the transition and returns a conflict. +2. The server recomputes the [content hash](/design/cryptography/primitives/) over the upload file on the blocking pool and compares it to the declared `hash`. +3. The manifest envelope is re-validated (invariant 15) — the same checks as at session creation, inside the finalization transaction, so nothing that changed since creation (a revoked device, a closed album) slips through. +4. **On match** — the file is renamed to its content address, the pending asset is marked uploaded inside a Postgres transaction, and the session transitions to **Completed**. +5. **On mismatch** — the upload file and the pending asset row are deleted, the session transitions to **FailedProcessing**, and `error.upload.content_hash_mismatch` is returned. A mismatch is always treated as corruption or tampering and is never silently retried server-side. The server verifies only the *ciphertext* hash — it has no other option. The client independently verifies the *plaintext* on download via the [STREAM construction](/design/cryptography/encryption/#stream-construction)'s per-chunk authentication tags, which detect truncation, reordering, and chunk deletion. The two checks are complementary: the server guarantees "the bytes I stored are the bytes you declared," and the AEAD guarantees "the plaintext I decrypted is authentic." `Completed` is a one-time transfer receipt, not a standing durability guarantee a client can re-query later. After finalization, a client confirms an asset remains durably stored, indexed, and retrievable — the precondition for releasing its local copy — through the separate [storage-verification endpoint](/design/import/storage-verification/), which re-checks state that server-side GC, migration, or corruption could change after `Completed`. -## Robustness +## Idempotency and Resumption -- An upload is not expected to run to completion in a single connection. The server tolerates arbitrarily long pauses within the session TTL, and clients resume via `HEAD`. [Auto syncing](/design/import/download-sync/#auto-syncing) explicitly assumes interrupted transfers are normal. -- A chunk re-sent at an already-acknowledged offset is idempotent. A chunk at a stale offset receives `409` together with the authoritative offset so the client can re-align. -- Concurrent finalization attempts on a single session are guarded — a second attempt observes a non-`Pending`/`Uploading` status and returns a conflict rather than double-processing. -- Every critical step — session creation, each chunk, assembly, hash verification, finalization — is logged with the upload ID so an interrupted or failed upload can be reconstructed and recovered after the fact. +- **Duplicate session creation** is keyed by `(owner_id, hash, album_id)` and split by state: if an **active** session for the tuple exists, `POST /upload` returns it (`200`, `Location` + current `X-Capsule-Offset`) — no second session is created; if the hash is already **finalized**, the request is rejected with `409 error.upload.duplicate_blob` carrying the existing asset reference, which is the client's trigger to resolve the situation as a [merge](#deduplication-and-merge) instead of a transfer. The dedup check and the pending-row insert run inside a single PostgreSQL transaction (`SELECT ... FOR UPDATE` + `INSERT ... ON CONFLICT`), closing the TOCTOU race at the database layer. +- **Chunk replay** is governed by the idempotency tuple in [Chunk Rules](#chunk-rules-and-strictness): same tuple → same response; same offset with different bytes → `409` corruption. +- **Resumption** is `HEAD`-driven: the response carries the authoritative offset, the declared length, and the session status. An upload is not expected to run to completion in a single connection; the server tolerates arbitrarily long pauses within the [session lifetime](#session-lifetime-and-discard), and [auto syncing](/design/import/download-sync/#auto-syncing) explicitly assumes interrupted transfers are normal. +- **Lost finalization ACK**: re-query `HEAD` and read the terminal receipt (see [Session Lifetime](#session-lifetime-and-discard)). +- Every critical step — session creation, each chunk, finalization — is logged with the upload ID so an interrupted or failed upload can be reconstructed after the fact. - [Streaming import-upload mode](/design/import/pipeline/#import-upload-streaming-mode) for storage-constrained devices uses these same sessions unchanged: it creates, uploads, and finalizes them one bounded window at a time and releases each local original after a durable [storage-verification](/design/import/storage-verification/) check. The wire protocol is identical — only the pipeline's drive pattern differs — and a server connection loss simply leaves the in-flight sessions resumable via `HEAD`. ## Adaptive Chunk Sizing -The server suggests an initial chunk size by file-size tier — `< 10 MB` → 256 KiB, `< 100 MB` → 1 MiB, `≥ 100 MB` → 4 MiB. The client may then adapt *within a tier-bounded range* based on throughput measured over a sliding 30-second window: doubling the chunk size when sustained throughput is high (`> 5 MB/s`), halving it when low (`< 1 MB/s`), and always staying 4 KiB-aligned. The rationale is a direct trade-off — chunks that are too small waste round-trips, while chunks that are too large waste re-transmission on a flaky link and pin more memory per in-flight request. +Chunk-size selection is split cleanly between the two sides: + +- **The server enforces, by rejection only, the protocol bounds**: 4 KiB alignment, the `[4 KiB, 16 MiB]` chunk range, and the cumulative ceilings. It never adapts, negotiates, or corrects. +- **`capsule-sdk` owns adaptation.** The `X-Capsule-Suggested-Chunk-Size` returned at session creation is a *starting point only*, from non-normative file-size tiers (currently `< 10 MB` → 256 KiB, `< 100 MB` → 1 MiB, `≥ 100 MB` → 4 MiB — server-tunable, not protocol surface). -Adaptation is purely a client concern; the server only enforces alignment and bounds. The client must never let adaptation regress effective throughput — if a tier's range is mis-tuned, the conservative choice is the tier minimum. +The client algorithm is normative for `capsule-sdk` (another client may adapt differently, within the protocol bounds): + +- Measure throughput over a sliding **30-second window**. +- **Warm-up:** no adjustment until ≥ 5 chunks *or* ≥ 8 MiB have been sent at the current size — decisions on a cold window oscillate. +- **Double** the chunk size when sustained throughput exceeds 5 MB/s; **halve** it below 1 MB/s. +- Clamp to the tier's range; every tier range sits inside the protocol bounds `[256 KiB, 16 MiB]` for non-final chunks. +- Every candidate size is a 4 KiB multiple **by construction** (the candidate set is doublings/halvings of 4 KiB-aligned tier bounds), so alignment never depends on a runtime check. + +The rationale is a direct trade-off — chunks that are too small waste round-trips, while chunks that are too large waste re-transmission on a flaky link and pin more memory per in-flight request. Under an [adverse connection](/design/import/download-sync/), the conservative choice is the tier minimum; the client must never let adaptation regress effective throughput. We deliberately do **not** expose per-blob upload *ordering* as a protocol concern. Concurrent sessions plus the OS and TCP stack settle ordering naturally; see [Pipeline — Upload Prioritization](/design/import/pipeline/#upload-prioritization) for the client-side heuristics that decide which assets to *start*. +## Error Taxonomy + +Every rejection carries a machine-readable `error.*` code (the [i18n error-code contract](/design/i18n/#server-error-codes)); clients switch on codes, never on bare HTTP statuses. The full upload domain: + +| Condition | HTTP | Code | Invariant | +| ----------------------------------------------------- | ----- | ------------------------------------------ | --------- | +| Protocol version outside window | `426` | `error.protocol.version_unsupported` | 1 | +| Unknown crypto suite | `400` | `error.upload.unknown_crypto_suite` | 2 | +| Hash length ≠ suite digest length / not hex | `400` | `error.upload.invalid_hash` | 3 | +| `size` ∉ `(0, max_file_size]` | `400`/`413` | `error.upload.invalid_size` / `error.upload.file_too_large` | 4 | +| `content_type` outside closed enum | `400` | `error.upload.unsupported_content_type` | 5 | +| Album missing / no write capability / version drift | `403` | `error.upload.album_access_denied` | 6 | +| `created_by_device` not in device directory | `403` | `error.upload.device_not_authorized` | 7 | +| Envelope timestamp outside sanity window | `400` | `error.upload.timestamp_out_of_range` | 8 | +| Top-level fields contradict `manifest_envelope` | `400` | `error.upload.envelope_mismatch` | 15 family | +| Malformed body / unknown JSON fields | `400` | `error.upload.malformed_request` | — | +| On-behalf upload without verified relationship | `403` | `error.upload.owner_not_permitted` | — | +| Quota would be exceeded by declared `size` | `403` | `error.quota.exceeded` | — | +| Hash already finalized (dup create) | `409` | `error.upload.duplicate_blob` | — | +| Wrong/missing `Content-Type` on `PATCH` | `415` | `error.upload.unsupported_media_type` | 10 | +| Empty chunk | `400` | `error.upload.empty_chunk` | 10 | +| Unaligned non-final chunk | `400` | `error.upload.chunk_not_aligned` | 10 | +| Chunk > 16 MiB | `413` | `error.upload.chunk_too_large` | 10 | +| Missing/invalid `X-Capsule-Offset` header | `400` | `error.upload.missing_offset` | 9 | +| Missing/malformed `X-Capsule-Checksum` | `400` | `error.upload.missing_checksum` | 12 | +| Checksum ≠ received bytes | `400` | `error.upload.checksum_mismatch` | 12 | +| Offset ≠ current received count | `409` | `error.upload.offset_mismatch` | 9 | +| Same offset, different chunk hash | `409` | `error.upload.chunk_conflict` | 12 | +| Cumulative bytes exceed declared `size` | `400` | `error.upload.size_exceeded` | 11 | +| Unknown / expired / discarded session | `404` | `error.upload.session_not_found` | — | +| Session already terminal or finalizing | `409` | `error.upload.session_not_active` | — | +| Concurrent finalization lost the CAS | `409` | `error.upload.finalize_in_progress` | — | +| Recomputed hash ≠ declared (finalization) | `400` | `error.upload.content_hash_mismatch` | 14 | +| Envelope re-validation failed (finalization) | `400` | `error.upload.envelope_rejected` | 15 | +| Caller is neither uploader nor owner | `403` | `error.upload.forbidden` | — | +| Upload-file / session-counter divergence | `500` | `error.upload.storage_inconsistent` | — | + +## Edge Cases + +The stateful surface, enumerated — each of these is a test in the Validation section: + +- **`PATCH` after finalization began or completed** → `409 error.upload.session_not_active`; the terminal outcome is read via `HEAD`. +- **`DELETE` during `WaitingForProcessing`** → `409` — finalization is not interruptible; cancel before completion or observe the receipt after. +- **Duplicate `POST /upload`** → active session returned / finalized hash `409 duplicate_blob` (see [Idempotency and Resumption](#idempotency-and-resumption)). +- **Offset beyond current EOF** (client skipped ahead) → `409 offset_mismatch` with the authoritative offset — indistinguishable from any other stale offset, deliberately. +- **Offset correct but `offset + len > size`** → `400 size_exceeded`, session → `FailedProcessing` (the declaration was violated; the session is unsalvageable by definition). +- **Checksum mismatch mid-stream** → `400`, nothing persisted, offset unchanged; the client re-sends the same chunk. +- **`PATCH` against an expired/discarded session** → uniform `404`; the client re-creates and re-uploads that blob. +- **Zero-length blob** — impossible by construction: invariant 4 requires `size > 0` at creation, so no chunk sequence can validly be empty. + ## Deduplication and Merge Because blobs are addressed by their [ciphertext content hash](/design/cryptography/primitives/), the protocol avoids redundant transfers: -- At session creation, the server checks for an asset with the same content hash already owned by the user. An exact duplicate that exists both locally and remotely is rejected up front — nothing is re-uploaded. The dedup check and the pending-row insert run inside a single PostgreSQL transaction (a `SELECT ... FOR UPDATE` followed by `INSERT ... ON CONFLICT`), so two concurrent uploaders cannot both observe "no existing row" and each insert their own — the TOCTOU race is closed at the database layer. +- At session creation, the server checks for an asset with the same content hash already owned by the user; the split behavior (active session returned; finalized hash → `409 duplicate_blob` + existing asset reference) is defined in [Idempotency and Resumption](#idempotency-and-resumption). Nothing is ever re-uploaded. - The [import pipeline](/design/import/pipeline/#plan--confirm) treats already-uploaded *local* assets as non-importable. But because encryption and hashing are deferred until upload, an asset may already exist remotely under a *different* ciphertext (for example, re-encrypted under a newer album key). Import still admits such an asset, and the upload then resolves to a **merge**: the server links the existing stored blob to the new asset and album reference rather than storing a second copy. The original blob's upload short-circuits, and only the new metadata blob is transferred. - **Merge is strictly additive on the server.** A merge **never** deletes an existing blob or rewrites an existing manifest — it only adds a new reference. The blob's reference count goes up, never down, on merge. Reference removal happens only through an explicit `delete` lifecycle action signed by a current writer (see [Authorization](/design/authorization/)), and the underlying blob is hard-purged only after every reference is provably gone. @@ -144,17 +261,19 @@ These checks deduplicate at upload time. Byte-identical assets that still slip i - An upload is attributed to `upload_user_id` (the authenticated uploader) for storage-quota accounting, which is distinct from `owner_id` (the asset's owner). Uploading on behalf of a different owner requires a verified relationship and is permission-checked at session creation. The quota accounting model is owned by [Quota](/design/quota/). - Adding an asset to an album requires write-tier album access (`AMK_write`; see [Cryptography — Keys](/design/cryptography/keys/#album-master-keys-amks)); the server validates album write permission before creating the session. - For an ordinary asset bundle the client resolves a concrete container `album_id` — the user's choice or the [default album](/design/organization/#the-default-album) — **before** encryption, since the bytes are encrypted under that album's AMK. So `album_id` is effectively always present for asset uploads; the `optional` marking on `POST /upload` covers only non-asset/owner-scoped kinds and the `intent_id`-bearing [album upgrade](/design/versioning/#album-upgrade-ceremony). This is why [invariant 6](/design/threat-model/validation/#server-side-validation-invariants) can require `album_id` to exist and be writable. -- Only the uploader may append chunks. The uploader or the owner may query (`HEAD`) or cancel (`DELETE`) a session. +- Only the uploader may append chunks. The uploader or the owner may query (`HEAD`) or cancel (`DELETE`) a session; anyone else receives `403`. Session listing is uploader-scoped — the uploader is the party that resumes. ## Validation The wire protocol is the boundary across two modules, so both sides have rich isolated test surfaces: - **Server protocol conformance (smoke).** Exercise the full state machine against the real server against a testcontainer Postgres (and Valkey for the high-concurrency profile): create session → PATCH chunks → finalize → verify Completed. Mock the client at the HTTP layer using a generated request fixture set. -- **Server chunk-rule rejection (unit).** Each rule (non-aligned non-final chunk, gapped offset, duplicate offset with different hash, cumulative-over-size, oversize file) has a unit test asserting the exact rejection code. -- **Server idempotency (unit).** Replay each idempotent endpoint with identical input; assert byte-identical response. -- **Server finalization integrity (smoke).** Concatenate chunks; recompute hash; assert match. Inject a corrupted chunk; assert FailedProcessing and full cleanup of the pending row + chunks. -- **Client protocol conformance (smoke).** The client `capsule-sdk::upload` runs against a mocked HTTP layer that replays the rejection codes the server's unit tests exercise; assert the client handles each correctly (re-align on 409, abort-and-reimport on 426, etc.). +- **Server strictness rejection (unit).** Every row of the [strictness table](#chunk-rules-and-strictness) and every row of the [error taxonomy](#error-taxonomy) has a unit test asserting the exact HTTP status **and** `error.*` code. +- **Server idempotency (unit).** Replay each idempotent endpoint with identical input; assert byte-identical response. Duplicate create returns the active session; finalized-hash create returns `duplicate_blob`. +- **Server lifetime (unit).** A session with an accepted chunk in the last hour is never evicted under injected pressure; a stalled session past the floor is; terminal receipts survive pressure and die at the cap. +- **Server append-only crash safety (smoke).** Kill the server between the append and the counter increment; on restart the scrub reconciles (file length is truth) and `HEAD` reports the on-disk offset. Kill between rename and commit; assert the atomicity invariants recover. +- **Server finalization integrity (smoke).** Upload; recompute hash; assert match. Inject a corrupted chunk stream; assert FailedProcessing and full cleanup of the pending row + upload file. +- **Client protocol conformance (smoke).** The client `capsule-sdk::upload` runs against a mocked HTTP layer that replays every taxonomy code; assert the client's recovery matrix (re-align on `offset_mismatch`, re-create on `session_not_found`, merge on `duplicate_blob`, abort-with-upgrade on `426`, re-send on `checksum_mismatch`). - **Client resume semantics (smoke).** Start an upload, interrupt at random offset, resume; assert no bytes re-sent that the server already has. The cross-module case — real client → real server full upload — is bounded E2E surface listed in [Module Map](/design/module-map/#e2e-test-surface). Because both sides have rich smoke coverage, the E2E case can be a single happy-path round-trip rather than the full rejection matrix. diff --git a/capsule-docs/src/content/docs/design/threat-model/scenarios.md b/capsule-docs/src/content/docs/design/threat-model/scenarios.md index 294654ce..82e98099 100644 --- a/capsule-docs/src/content/docs/design/threat-model/scenarios.md +++ b/capsule-docs/src/content/docs/design/threat-model/scenarios.md @@ -12,7 +12,7 @@ The lookup table for "what damage X is prevented by which invariant Y in which d | --- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | Old client writes a sidecar after stripping unknown fields | Sidecar signature covers `_unknown`; old client refuses to write when `sidecar_schema` > its max known | [Metadata — Schema Versioning Rules](/design/metadata/#schema-versioning-rules) | | 2 | Faulty client uploads bytes that don't match the declared content type | Server's `content_type` allow-list per protocol version (no-key check) + receiving client decoder sandbox | [Validation §](/design/threat-model/validation/#server-side-validation-invariants), [Clients — Sandboxed Decoder](/design/clients/#sandboxed-decoder) | -| 3 | Buggy client uploads chunk with wrong offset and re-tries | Idempotency tuple `(upload_id, offset, chunk_hash)`; duplicate at offset with different hash → reject | [Import — Upload Protocol](/design/import/upload-protocol/#chunk-rules) | +| 3 | Buggy client uploads chunk with wrong offset and re-tries | Idempotency tuple `(upload_id, offset, chunk_hash)`; duplicate at offset with different hash → reject | [Import — Upload Protocol](/design/import/upload-protocol/#chunk-rules-and-strictness) | | 4 | Hostile peer sends an old-but-validly-signed manifest to revive a deleted asset | `prior_provenance_hash` chain advance check on both client and server | [Cryptography — Provenance](/design/cryptography/provenance/#provenance-of-library-modifications), [Validation §](/design/threat-model/validation/#server-side-validation-invariants) | | 5 | Malicious client re-signs an existing manifest under a weaker `crypto_suite_id` | Signatures cover `crypto_suite_id` and `protocol_version` | [Cryptography — Write Authorization](/design/cryptography/keys/#write-authorization) | | 6 | Two devices concurrently caption the same photo | Caption LWW + `superseded_captions` array surfaces the loser | [Metadata — Surfacing Concurrent Edits](/design/metadata/#surfacing-concurrent-edits) | @@ -20,7 +20,7 @@ The lookup table for "what damage X is prevented by which invariant Y in which d | 8 | Buggy client overwrites a good thumbnail with a corrupt one | Every derivative carries a signed `DerivativeManifest` on its own chain; overwrite is a `derivative-replace` lifecycle action | [Cryptography — Derivative Provenance](/design/cryptography/provenance/#derivative-provenance) | | 9 | A client declares `timestamp = 2099-01-01` to distort the audit | `timestamp` is self-asserted and audit-only — never load-bearing for ordering or authorization (those ride the chain + epoch); a gross-drift sanity bound surfaces it, and the server's own `received_at` is the authoritative clock | [Cryptography — Write Authorization](/design/cryptography/keys/#write-authorization) | | 10 | Server-side TOCTOU on blob dedup creates a duplicate | Dedup-check and pending-row insert are atomic on a single Postgres transaction | [Filesystem — Content-Addressing and Deduplication](/design/filesystem/server/#content-addressing-and-deduplication) | -| 11 | A faulty client uploads bytes that exceed its declared size | Server bounds cumulative received at every chunk, not only at finalization | [Import — Chunk Rules](/design/import/upload-protocol/#chunk-rules) | +| 11 | A faulty client uploads bytes that exceed its declared size | Server bounds cumulative received at every chunk, not only at finalization | [Import — Chunk Rules](/design/import/upload-protocol/#chunk-rules-and-strictness) | | 12 | A new client writes a manifest with a `crypto_suite_id` the server does not recognize | Refuse-by-default at handshake: 400 before any session is created | [Validation — Protocol Negotiation](/design/threat-model/validation/#protocol-and-capability-negotiation) | | 13 | A federated peer floods the rejected-hash table to exhaust memory | Per-peer quota; bounded LRU memory | [Federation — Soft-Fail Semantics](/design/federation/#soft-fail-semantics) | | 14 | A model swap silently invalidates the AI tag namespace | Every `tags_ai` entry carries `model_id`+`model_version`; the vector-index insert API and query layer (`capsule-core::db`) reject unknown-model inserts and exclude stale entries — cross-model comparison is forbidden | [Metadata — Tag Provenance and Namespacing](/design/metadata/#tag-provenance-and-namespacing) | diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index 36674f45..0c9f14b9 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -30,9 +30,11 @@ Invariants carry **stable numbers** (referenced across docs as "invariant 17", " ### On each `PATCH /upload/{id}` chunk - **9.** Offset is exactly the current received-byte count. Otherwise `409`, with `X-Capsule-Offset` returned. -- **10.** Non-final chunk size is a multiple of 4 KiB. Otherwise `400`. +- **10.** The chunk body is well-shaped: `Content-Type: application/octet-stream` (otherwise `415`), non-empty (otherwise `400`), a non-final chunk is a multiple of 4 KiB (otherwise `400`), and no chunk exceeds the 16 MiB protocol maximum (otherwise `413`). - **11.** Cumulative received ≤ declared `size`. Otherwise `400` / `413`, session moves to `FailedProcessing`. -- **12.** The `(upload_id, offset, chunk_hash)` idempotency tuple is new OR matches an exact prior PATCH. Otherwise (same offset, different hash) `409` + corruption error. +- **12.** The `(upload_id, offset, chunk_hash)` idempotency tuple is new OR matches an exact prior PATCH, where `chunk_hash` is the SHA-256 of the chunk bytes carried in the **required** `X-Capsule-Checksum` header (missing/malformed → `400`; header-vs-body mismatch → `400` with nothing persisted). Otherwise (same offset, different hash) `409` + corruption error. + +Session TTL, the ≥ 1-hour survival floor, and pressure-discard semantics are server lifecycle behavior, not refuse-by-default write checks — they are owned by [Upload Protocol — Session Lifetime and Discard](/design/import/upload-protocol/#session-lifetime-and-discard). ### At finalization @@ -111,7 +113,7 @@ The rules are stated once, in REST terms (headers + HTTP statuses). On the gRPC | `X-Capsule-Protocol-Max` | server on every response | the highest protocol version this server accepts | | `X-Capsule-Min-Client-Build` | server on responses | semver deprecation cutoff; advisory unless the path is hard-deprecated | -This table is also the **census of the `X-Capsule-*` header namespace**. Surface-specific headers register here by pointer: the upload protocol's `X-Capsule-Offset`, `X-Capsule-Content-Length`, `X-Capsule-Checksum`, and `X-Capsule-Suggested-Chunk-Size` (semantics owned by [Import — Upload Protocol](/design/import/upload-protocol/#endpoints)). A new `X-Capsule-*` header MUST be registered here when introduced — two homes for the namespace is how headers drift. +This table is also the **census of the `X-Capsule-*` header namespace**. Surface-specific headers register here by pointer: the upload protocol's `X-Capsule-Offset`, `X-Capsule-Content-Length`, `X-Capsule-Upload-Status` (server → client on `HEAD /upload/{id}`), `X-Capsule-Checksum` (**required** on `PATCH /upload/{id}`), and `X-Capsule-Suggested-Chunk-Size` (semantics owned by [Import — Upload Protocol](/design/import/upload-protocol/#endpoints)). A new `X-Capsule-*` header MUST be registered here when introduced — two homes for the namespace is how headers drift. ### Fail-Closed Rules @@ -130,7 +132,7 @@ Every write surface has a single idempotency key. Duplicates are no-ops; conflic | Surface | Idempotency key | Duplicate behavior | | ----------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------- | | Upload chunk (`PATCH /upload/{id}`) | `(upload_id, offset, chunk_hash)` | Returns current offset; no double-write | -| Session creation (`POST /upload`) | `(owner_id, hash, album_id)` — server's existing dedup check | Returns the existing session; no second session | +| Session creation (`POST /upload`) | `(owner_id, hash, album_id)` — server's existing dedup check | Active session: returned as-is, no second session. Hash already finalized: `409 error.upload.duplicate_blob` + the existing asset reference (the client's merge trigger) | | Lifecycle manifest write | `(asset_id, prior_provenance_hash, manifest_hash)` | No-op append; chain advances exactly once | | Metadata-update operation | Operation id (UUIDv7) + `(asset_id, prior_provenance_hash)` | Re-applying the same op is structurally identical | | Federation capability proof | `(peer_id, jti)` | Refresh with same `jti` returns the same response | diff --git a/capsule-docs/src/content/docs/design/web-upload.md b/capsule-docs/src/content/docs/design/web-upload.md index 8e077aa9..8788de00 100644 --- a/capsule-docs/src/content/docs/design/web-upload.md +++ b/capsule-docs/src/content/docs/design/web-upload.md @@ -62,7 +62,7 @@ For each selected asset the web client: 1. Draws a random 32-byte asset key **`K`** from the browser CSPRNG. 2. Encrypts the asset under `K` with the *unchanged* [STREAM construction](/design/cryptography/encryption/#stream-construction) (chunking and nonce shape owned there), and computes `ciphertext_hash` incrementally. 3. **Encapsulates `K`** to `{drop_pubkey}` with the link's KEM, producing `kem_ct`. -4. Emits an unsigned **`DropDescriptor`** and uploads it alongside the ciphertext via the drop upload protocol — the [upload protocol](/design/import/upload-protocol/)'s chunk and finalization mechanics under link-capability auth, with the drop endpoints in the [Contract Skeleton](#contract-skeleton): +4. Emits an unsigned **`DropDescriptor`** and uploads it alongside the ciphertext via the drop upload protocol — the [upload protocol](/design/import/upload-protocol/)'s chunk and finalization mechanics under link-capability auth, with the drop endpoints in the [Contract Skeleton](#contract-skeleton). Its [session-lifetime and discard policy](/design/import/upload-protocol/#session-lifetime-and-discard) applies to drop sessions unchanged — a discarded drop session is a uniform `404`, which is also exactly what link probing must see: ```rust DropDescriptor { @@ -131,7 +131,8 @@ fn seal_drop(plaintext: impl Read, drop_pubkey: KemPublicKey, crypto_suite_id: u // in capsule-api-media::drops // POST /u/{opaque-id}/drop → open a drop session (link-capability auth; quota + caps checked here) -// PATCH /u/{opaque-id}/drop/{id} → append a chunk (reuses upload-protocol chunk rules) +// PATCH /u/{opaque-id}/drop/{id} → append a chunk (upload-protocol chunk rules verbatim: +// required X-Capsule-Checksum, application/octet-stream, alignment) // GET /drops → provisioning user's inbox (session-token auth) // POST /drops/{id}/adopt → create-manifest write referencing the inbox blob; atomic promotion // DELETE /drops/{id} → discard a pending drop From f7672faaba2da27dbb0559690dc16455d2682728 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:28:36 -0400 Subject: [PATCH 18/58] feat(i18n): catalog the upload error domain The upload contract mandates a machine-readable error.* code on every rejection, but the catalog only contained error.auth.*. This adds the full error.upload.* domain from the upload protocol's error taxonomy, plus three codes other docs already referenced without a catalog entry (error.protocol.version_unsupported, error.quota.exceeded, error.blob.pending_upload). Generated per-platform files are regenerated with 'mise run i18n'; the upload server slice (S-C1) can now reference compile-checked capsule_i18n::error_codes::UPLOAD_* constants, the same pattern capsule-api-auth already uses. --- .../src/androidMain/res/values/strings.xml | 32 ++ capsule-i18n/src/bundles/en.json | 32 ++ capsule-i18n/src/generated.rs | 96 ++++++ capsule-swift/Generated/Localizable.xcstrings | 320 ++++++++++++++++++ capsule-web/src/i18n/messages/en.json | 32 ++ locales/en.json | 128 +++++++ 6 files changed, 640 insertions(+) diff --git a/capsule-android/src/androidMain/res/values/strings.xml b/capsule-android/src/androidMain/res/values/strings.xml index 903d8483..03aebd00 100644 --- a/capsule-android/src/androidMain/res/values/strings.xml +++ b/capsule-android/src/androidMain/res/values/strings.xml @@ -5,6 +5,38 @@ Invalid email or password. Too many attempts. Please wait and try again. An account with these details already exists. + The original photo hasn\'t been uploaded from its device yet. + This version of Capsule is no longer supported. Please update the app. + Your storage is full. + You don\'t have permission to add photos to this album. + Upload data was corrupted in transit. Retrying… + The upload sent conflicting data and was stopped. + An upload chunk was incorrectly sized. + An upload chunk exceeded the maximum chunk size. + The upload failed integrity verification and was discarded. + This device isn\'t authorized to upload. Re-enroll it and try again. + This file is already stored. + An upload chunk cannot be empty. + The upload request contradicts its signed manifest. + The upload\'s signed manifest failed validation. + This file exceeds the server\'s size limit. + This upload is already being finalized. + You don\'t have access to this upload. + The upload\'s declared content hash is invalid. + The upload\'s declared size is invalid. + The upload request was malformed. + The upload request was missing its integrity checksum. + The upload request was missing its position marker. + The upload got out of sync and is realigning. + You can\'t upload on behalf of this user. + This upload has already finished. + The upload session expired. Starting over… + The upload sent more data than it declared. + The server hit a storage problem. Please try again. + Your device\'s clock appears to be wrong. Check its date and time. + This app version uses an encryption suite the server doesn\'t recognize. + This file type is not supported. + Upload data was sent in an unsupported format. Artist Credits Date diff --git a/capsule-i18n/src/bundles/en.json b/capsule-i18n/src/bundles/en.json index ba209e04..1fcd7200 100644 --- a/capsule-i18n/src/bundles/en.json +++ b/capsule-i18n/src/bundles/en.json @@ -4,6 +4,38 @@ "error.auth.invalid_credentials": "Invalid email or password.", "error.auth.rate_limited": "Too many attempts. Please wait and try again.", "error.auth.user_already_exists": "An account with these details already exists.", + "error.blob.pending_upload": "The original photo hasn't been uploaded from its device yet.", + "error.protocol.version_unsupported": "This version of Capsule is no longer supported. Please update the app.", + "error.quota.exceeded": "Your storage is full.", + "error.upload.album_access_denied": "You don't have permission to add photos to this album.", + "error.upload.checksum_mismatch": "Upload data was corrupted in transit. Retrying…", + "error.upload.chunk_conflict": "The upload sent conflicting data and was stopped.", + "error.upload.chunk_not_aligned": "An upload chunk was incorrectly sized.", + "error.upload.chunk_too_large": "An upload chunk exceeded the maximum chunk size.", + "error.upload.content_hash_mismatch": "The upload failed integrity verification and was discarded.", + "error.upload.device_not_authorized": "This device isn't authorized to upload. Re-enroll it and try again.", + "error.upload.duplicate_blob": "This file is already stored.", + "error.upload.empty_chunk": "An upload chunk cannot be empty.", + "error.upload.envelope_mismatch": "The upload request contradicts its signed manifest.", + "error.upload.envelope_rejected": "The upload's signed manifest failed validation.", + "error.upload.file_too_large": "This file exceeds the server's size limit.", + "error.upload.finalize_in_progress": "This upload is already being finalized.", + "error.upload.forbidden": "You don't have access to this upload.", + "error.upload.invalid_hash": "The upload's declared content hash is invalid.", + "error.upload.invalid_size": "The upload's declared size is invalid.", + "error.upload.malformed_request": "The upload request was malformed.", + "error.upload.missing_checksum": "The upload request was missing its integrity checksum.", + "error.upload.missing_offset": "The upload request was missing its position marker.", + "error.upload.offset_mismatch": "The upload got out of sync and is realigning.", + "error.upload.owner_not_permitted": "You can't upload on behalf of this user.", + "error.upload.session_not_active": "This upload has already finished.", + "error.upload.session_not_found": "The upload session expired. Starting over…", + "error.upload.size_exceeded": "The upload sent more data than it declared.", + "error.upload.storage_inconsistent": "The server hit a storage problem. Please try again.", + "error.upload.timestamp_out_of_range": "Your device's clock appears to be wrong. Check its date and time.", + "error.upload.unknown_crypto_suite": "This app version uses an encryption suite the server doesn't recognize.", + "error.upload.unsupported_content_type": "This file type is not supported.", + "error.upload.unsupported_media_type": "Upload data was sent in an unsupported format.", "label_artist": "Artist", "label_credits": "Credits", "label_date": "Date", diff --git a/capsule-i18n/src/generated.rs b/capsule-i18n/src/generated.rs index 7abb05e5..8bd634dc 100644 --- a/capsule-i18n/src/generated.rs +++ b/capsule-i18n/src/generated.rs @@ -24,4 +24,100 @@ pub mod error_codes { /// `error.auth.user_already_exists` pub const AUTH_USER_ALREADY_EXISTS: &str = "error.auth.user_already_exists"; + + /// `error.blob.pending_upload` + pub const BLOB_PENDING_UPLOAD: &str = "error.blob.pending_upload"; + + /// `error.protocol.version_unsupported` + pub const PROTOCOL_VERSION_UNSUPPORTED: &str = "error.protocol.version_unsupported"; + + /// `error.quota.exceeded` + pub const QUOTA_EXCEEDED: &str = "error.quota.exceeded"; + + /// `error.upload.album_access_denied` + pub const UPLOAD_ALBUM_ACCESS_DENIED: &str = "error.upload.album_access_denied"; + + /// `error.upload.checksum_mismatch` + pub const UPLOAD_CHECKSUM_MISMATCH: &str = "error.upload.checksum_mismatch"; + + /// `error.upload.chunk_conflict` + pub const UPLOAD_CHUNK_CONFLICT: &str = "error.upload.chunk_conflict"; + + /// `error.upload.chunk_not_aligned` + pub const UPLOAD_CHUNK_NOT_ALIGNED: &str = "error.upload.chunk_not_aligned"; + + /// `error.upload.chunk_too_large` + pub const UPLOAD_CHUNK_TOO_LARGE: &str = "error.upload.chunk_too_large"; + + /// `error.upload.content_hash_mismatch` + pub const UPLOAD_CONTENT_HASH_MISMATCH: &str = "error.upload.content_hash_mismatch"; + + /// `error.upload.device_not_authorized` + pub const UPLOAD_DEVICE_NOT_AUTHORIZED: &str = "error.upload.device_not_authorized"; + + /// `error.upload.duplicate_blob` + pub const UPLOAD_DUPLICATE_BLOB: &str = "error.upload.duplicate_blob"; + + /// `error.upload.empty_chunk` + pub const UPLOAD_EMPTY_CHUNK: &str = "error.upload.empty_chunk"; + + /// `error.upload.envelope_mismatch` + pub const UPLOAD_ENVELOPE_MISMATCH: &str = "error.upload.envelope_mismatch"; + + /// `error.upload.envelope_rejected` + pub const UPLOAD_ENVELOPE_REJECTED: &str = "error.upload.envelope_rejected"; + + /// `error.upload.file_too_large` + pub const UPLOAD_FILE_TOO_LARGE: &str = "error.upload.file_too_large"; + + /// `error.upload.finalize_in_progress` + pub const UPLOAD_FINALIZE_IN_PROGRESS: &str = "error.upload.finalize_in_progress"; + + /// `error.upload.forbidden` + pub const UPLOAD_FORBIDDEN: &str = "error.upload.forbidden"; + + /// `error.upload.invalid_hash` + pub const UPLOAD_INVALID_HASH: &str = "error.upload.invalid_hash"; + + /// `error.upload.invalid_size` + pub const UPLOAD_INVALID_SIZE: &str = "error.upload.invalid_size"; + + /// `error.upload.malformed_request` + pub const UPLOAD_MALFORMED_REQUEST: &str = "error.upload.malformed_request"; + + /// `error.upload.missing_checksum` + pub const UPLOAD_MISSING_CHECKSUM: &str = "error.upload.missing_checksum"; + + /// `error.upload.missing_offset` + pub const UPLOAD_MISSING_OFFSET: &str = "error.upload.missing_offset"; + + /// `error.upload.offset_mismatch` + pub const UPLOAD_OFFSET_MISMATCH: &str = "error.upload.offset_mismatch"; + + /// `error.upload.owner_not_permitted` + pub const UPLOAD_OWNER_NOT_PERMITTED: &str = "error.upload.owner_not_permitted"; + + /// `error.upload.session_not_active` + pub const UPLOAD_SESSION_NOT_ACTIVE: &str = "error.upload.session_not_active"; + + /// `error.upload.session_not_found` + pub const UPLOAD_SESSION_NOT_FOUND: &str = "error.upload.session_not_found"; + + /// `error.upload.size_exceeded` + pub const UPLOAD_SIZE_EXCEEDED: &str = "error.upload.size_exceeded"; + + /// `error.upload.storage_inconsistent` + pub const UPLOAD_STORAGE_INCONSISTENT: &str = "error.upload.storage_inconsistent"; + + /// `error.upload.timestamp_out_of_range` + pub const UPLOAD_TIMESTAMP_OUT_OF_RANGE: &str = "error.upload.timestamp_out_of_range"; + + /// `error.upload.unknown_crypto_suite` + pub const UPLOAD_UNKNOWN_CRYPTO_SUITE: &str = "error.upload.unknown_crypto_suite"; + + /// `error.upload.unsupported_content_type` + pub const UPLOAD_UNSUPPORTED_CONTENT_TYPE: &str = "error.upload.unsupported_content_type"; + + /// `error.upload.unsupported_media_type` + pub const UPLOAD_UNSUPPORTED_MEDIA_TYPE: &str = "error.upload.unsupported_media_type"; } diff --git a/capsule-swift/Generated/Localizable.xcstrings b/capsule-swift/Generated/Localizable.xcstrings index bdfed31b..bfa149e0 100644 --- a/capsule-swift/Generated/Localizable.xcstrings +++ b/capsule-swift/Generated/Localizable.xcstrings @@ -51,6 +51,326 @@ } } }, + "error.blob.pending_upload": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The original photo hasn't been uploaded from its device yet." + } + } + } + }, + "error.protocol.version_unsupported": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This version of Capsule is no longer supported. Please update the app." + } + } + } + }, + "error.quota.exceeded": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "Your storage is full." + } + } + } + }, + "error.upload.album_access_denied": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "You don't have permission to add photos to this album." + } + } + } + }, + "error.upload.checksum_mismatch": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "Upload data was corrupted in transit. Retrying…" + } + } + } + }, + "error.upload.chunk_conflict": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload sent conflicting data and was stopped." + } + } + } + }, + "error.upload.chunk_not_aligned": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "An upload chunk was incorrectly sized." + } + } + } + }, + "error.upload.chunk_too_large": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "An upload chunk exceeded the maximum chunk size." + } + } + } + }, + "error.upload.content_hash_mismatch": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload failed integrity verification and was discarded." + } + } + } + }, + "error.upload.device_not_authorized": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This device isn't authorized to upload. Re-enroll it and try again." + } + } + } + }, + "error.upload.duplicate_blob": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This file is already stored." + } + } + } + }, + "error.upload.empty_chunk": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "An upload chunk cannot be empty." + } + } + } + }, + "error.upload.envelope_mismatch": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload request contradicts its signed manifest." + } + } + } + }, + "error.upload.envelope_rejected": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload's signed manifest failed validation." + } + } + } + }, + "error.upload.file_too_large": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This file exceeds the server's size limit." + } + } + } + }, + "error.upload.finalize_in_progress": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This upload is already being finalized." + } + } + } + }, + "error.upload.forbidden": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "You don't have access to this upload." + } + } + } + }, + "error.upload.invalid_hash": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload's declared content hash is invalid." + } + } + } + }, + "error.upload.invalid_size": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload's declared size is invalid." + } + } + } + }, + "error.upload.malformed_request": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload request was malformed." + } + } + } + }, + "error.upload.missing_checksum": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload request was missing its integrity checksum." + } + } + } + }, + "error.upload.missing_offset": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload request was missing its position marker." + } + } + } + }, + "error.upload.offset_mismatch": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload got out of sync and is realigning." + } + } + } + }, + "error.upload.owner_not_permitted": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "You can't upload on behalf of this user." + } + } + } + }, + "error.upload.session_not_active": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This upload has already finished." + } + } + } + }, + "error.upload.session_not_found": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload session expired. Starting over…" + } + } + } + }, + "error.upload.size_exceeded": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload sent more data than it declared." + } + } + } + }, + "error.upload.storage_inconsistent": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The server hit a storage problem. Please try again." + } + } + } + }, + "error.upload.timestamp_out_of_range": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "Your device's clock appears to be wrong. Check its date and time." + } + } + } + }, + "error.upload.unknown_crypto_suite": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This app version uses an encryption suite the server doesn't recognize." + } + } + } + }, + "error.upload.unsupported_content_type": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "This file type is not supported." + } + } + } + }, + "error.upload.unsupported_media_type": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "Upload data was sent in an unsupported format." + } + } + } + }, "label_artist": { "localizations": { "en": { diff --git a/capsule-web/src/i18n/messages/en.json b/capsule-web/src/i18n/messages/en.json index ba209e04..1fcd7200 100644 --- a/capsule-web/src/i18n/messages/en.json +++ b/capsule-web/src/i18n/messages/en.json @@ -4,6 +4,38 @@ "error.auth.invalid_credentials": "Invalid email or password.", "error.auth.rate_limited": "Too many attempts. Please wait and try again.", "error.auth.user_already_exists": "An account with these details already exists.", + "error.blob.pending_upload": "The original photo hasn't been uploaded from its device yet.", + "error.protocol.version_unsupported": "This version of Capsule is no longer supported. Please update the app.", + "error.quota.exceeded": "Your storage is full.", + "error.upload.album_access_denied": "You don't have permission to add photos to this album.", + "error.upload.checksum_mismatch": "Upload data was corrupted in transit. Retrying…", + "error.upload.chunk_conflict": "The upload sent conflicting data and was stopped.", + "error.upload.chunk_not_aligned": "An upload chunk was incorrectly sized.", + "error.upload.chunk_too_large": "An upload chunk exceeded the maximum chunk size.", + "error.upload.content_hash_mismatch": "The upload failed integrity verification and was discarded.", + "error.upload.device_not_authorized": "This device isn't authorized to upload. Re-enroll it and try again.", + "error.upload.duplicate_blob": "This file is already stored.", + "error.upload.empty_chunk": "An upload chunk cannot be empty.", + "error.upload.envelope_mismatch": "The upload request contradicts its signed manifest.", + "error.upload.envelope_rejected": "The upload's signed manifest failed validation.", + "error.upload.file_too_large": "This file exceeds the server's size limit.", + "error.upload.finalize_in_progress": "This upload is already being finalized.", + "error.upload.forbidden": "You don't have access to this upload.", + "error.upload.invalid_hash": "The upload's declared content hash is invalid.", + "error.upload.invalid_size": "The upload's declared size is invalid.", + "error.upload.malformed_request": "The upload request was malformed.", + "error.upload.missing_checksum": "The upload request was missing its integrity checksum.", + "error.upload.missing_offset": "The upload request was missing its position marker.", + "error.upload.offset_mismatch": "The upload got out of sync and is realigning.", + "error.upload.owner_not_permitted": "You can't upload on behalf of this user.", + "error.upload.session_not_active": "This upload has already finished.", + "error.upload.session_not_found": "The upload session expired. Starting over…", + "error.upload.size_exceeded": "The upload sent more data than it declared.", + "error.upload.storage_inconsistent": "The server hit a storage problem. Please try again.", + "error.upload.timestamp_out_of_range": "Your device's clock appears to be wrong. Check its date and time.", + "error.upload.unknown_crypto_suite": "This app version uses an encryption suite the server doesn't recognize.", + "error.upload.unsupported_content_type": "This file type is not supported.", + "error.upload.unsupported_media_type": "Upload data was sent in an unsupported format.", "label_artist": "Artist", "label_credits": "Credits", "label_date": "Date", diff --git a/locales/en.json b/locales/en.json index 246d35ca..e9d8c8cd 100644 --- a/locales/en.json +++ b/locales/en.json @@ -54,5 +54,133 @@ "error.auth.rate_limited": { "message": "Too many attempts. Please wait and try again.", "context": "High-level error for HTTP 429. Server sends code `error.auth.rate_limited`." + }, + "error.protocol.version_unsupported": { + "message": "This version of Capsule is no longer supported. Please update the app.", + "context": "HTTP 426 from the universal protocol handshake (invariant 1): the client's X-Capsule-Protocol date falls outside the server's accepted window." + }, + "error.quota.exceeded": { + "message": "Your storage is full.", + "context": "HTTP 403 at upload-session creation when the declared size would cross the uploader's hard quota limit." + }, + "error.blob.pending_upload": { + "message": "The original photo hasn't been uploaded from its device yet.", + "context": "Transient state, not a durability loss: an asset visible in the sync feed whose original blob has not been uploaded yet (staged uploads / awaiting-original). Distinct from 410 Gone." + }, + "error.upload.malformed_request": { + "message": "The upload request was malformed.", + "context": "HTTP 400 on POST /upload: unknown or missing JSON fields. The transport JSON is strict (deny unknown fields)." + }, + "error.upload.unknown_crypto_suite": { + "message": "This app version uses an encryption suite the server doesn't recognize.", + "context": "HTTP 400, invariant 2: crypto_suite_id not in the server's inventory." + }, + "error.upload.invalid_hash": { + "message": "The upload's declared content hash is invalid.", + "context": "HTTP 400, invariant 3: hash is not lowercase hex or its length doesn't match the suite's digest length." + }, + "error.upload.invalid_size": { + "message": "The upload's declared size is invalid.", + "context": "HTTP 400, invariant 4: declared size is zero or negative." + }, + "error.upload.file_too_large": { + "message": "This file exceeds the server's size limit.", + "context": "HTTP 413, invariant 4: declared size (or cumulative bytes) exceeds max_file_size." + }, + "error.upload.unsupported_content_type": { + "message": "This file type is not supported.", + "context": "HTTP 400, invariant 5: content_type outside the closed enum for the pinned protocol version." + }, + "error.upload.album_access_denied": { + "message": "You don't have permission to add photos to this album.", + "context": "HTTP 403, invariant 6: album missing, no write capability, or album protocol version drift." + }, + "error.upload.device_not_authorized": { + "message": "This device isn't authorized to upload. Re-enroll it and try again.", + "context": "HTTP 403, invariant 7: manifest envelope's created_by_device is not in the uploader's device directory." + }, + "error.upload.timestamp_out_of_range": { + "message": "Your device's clock appears to be wrong. Check its date and time.", + "context": "HTTP 400, invariant 8: envelope timestamp outside the gross-drift sanity window (default ±30 days)." + }, + "error.upload.envelope_mismatch": { + "message": "The upload request contradicts its signed manifest.", + "context": "HTTP 400: top-level request fields (crypto_suite_id, protocol_version, album_id) disagree with the manifest_envelope. Always a client bug." + }, + "error.upload.owner_not_permitted": { + "message": "You can't upload on behalf of this user.", + "context": "HTTP 403: owner_id differs from the uploader without a verified relationship." + }, + "error.upload.duplicate_blob": { + "message": "This file is already stored.", + "context": "HTTP 409 on POST /upload when the content hash is already finalized for this owner+album. Carries the existing asset reference; the client resolves it as a merge, not an error surface." + }, + "error.upload.unsupported_media_type": { + "message": "Upload data was sent in an unsupported format.", + "context": "HTTP 415, invariant 10: PATCH body Content-Type is not application/octet-stream." + }, + "error.upload.empty_chunk": { + "message": "An upload chunk cannot be empty.", + "context": "HTTP 400, invariant 10: zero-length PATCH body. Always a client bug; never accepted as a no-op." + }, + "error.upload.chunk_not_aligned": { + "message": "An upload chunk was incorrectly sized.", + "context": "HTTP 400, invariant 10: non-final chunk not a multiple of 4 KiB." + }, + "error.upload.chunk_too_large": { + "message": "An upload chunk exceeded the maximum chunk size.", + "context": "HTTP 413, invariant 10: chunk body larger than the 16 MiB protocol maximum." + }, + "error.upload.missing_offset": { + "message": "The upload request was missing its position marker.", + "context": "HTTP 400, invariant 9 family: X-Capsule-Offset header missing or unparseable on PATCH." + }, + "error.upload.missing_checksum": { + "message": "The upload request was missing its integrity checksum.", + "context": "HTTP 400, invariant 12: X-Capsule-Checksum header missing or not valid lowercase-hex SHA-256. The checksum is required — the idempotency tuple is undefined without it." + }, + "error.upload.checksum_mismatch": { + "message": "Upload data was corrupted in transit. Retrying…", + "context": "HTTP 400, invariant 12: SHA-256 of the received body doesn't match X-Capsule-Checksum. Nothing was persisted; the client re-sends the same chunk." + }, + "error.upload.offset_mismatch": { + "message": "The upload got out of sync and is realigning.", + "context": "HTTP 409, invariant 9: PATCH offset differs from the current received-byte count. Response carries the authoritative X-Capsule-Offset; the client realigns." + }, + "error.upload.chunk_conflict": { + "message": "The upload sent conflicting data and was stopped.", + "context": "HTTP 409, invariant 12: a PATCH at an already-acknowledged offset with a different chunk hash — the structural defense against a faulty client retrying with garbage." + }, + "error.upload.size_exceeded": { + "message": "The upload sent more data than it declared.", + "context": "HTTP 400, invariant 11: cumulative received bytes would exceed the declared size. Session moves to FailedProcessing." + }, + "error.upload.session_not_found": { + "message": "The upload session expired. Starting over…", + "context": "HTTP 404: unknown, expired, or pressure-discarded session — deliberately indistinguishable. The client re-creates the session and re-uploads that blob." + }, + "error.upload.session_not_active": { + "message": "This upload has already finished.", + "context": "HTTP 409: PATCH/DELETE against a session that is finalizing or terminal. The outcome is read via HEAD." + }, + "error.upload.finalize_in_progress": { + "message": "This upload is already being finalized.", + "context": "HTTP 409: a concurrent finalization attempt lost the atomic status compare-and-set." + }, + "error.upload.content_hash_mismatch": { + "message": "The upload failed integrity verification and was discarded.", + "context": "HTTP 400 at finalization, invariant 14: recomputed ciphertext hash differs from the declared hash. Treated as corruption/tampering; blob and pending row are deleted." + }, + "error.upload.envelope_rejected": { + "message": "The upload's signed manifest failed validation.", + "context": "HTTP 400 at finalization, invariant 15: envelope re-validation (rerun of invariants 1–8) failed inside the finalization transaction." + }, + "error.upload.forbidden": { + "message": "You don't have access to this upload.", + "context": "HTTP 403: caller is neither the session's uploader nor the asset owner." + }, + "error.upload.storage_inconsistent": { + "message": "The server hit a storage problem. Please try again.", + "context": "HTTP 500: the upload file's on-disk length diverged from the session's expected offset. Server-side inconsistency, never the client's fault." } } From a457da0206845689d033de7ec619274b4758e48b Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:37:04 -0400 Subject: [PATCH 19/58] refactor(api): align the upload server to the strict contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings the unhardened upload skeleton up to the contract the design doc now freezes, so the hardening slice (S-C1) and the SDK client slice (S-D1) can build against real types instead of guessing: - Chunk storage is append-only: one incoming/{upload_id}.bin per session, length cross-checked against the expected offset before every write, fsync before ACK. The .part-file + Linux-only reflink assembly path (and its libc dependency) is deleted — sequential offsets were already enforced, so assembly bought nothing and added a failure scenario. - The session record now carries everything finalization needs (crypto_suite_id, protocol_version, blob_role, manifest_envelope, intent_id, last_progress_at for the survival-floor anchor), and the wire request matches the documented body exactly: deny_unknown_fields, and the plaintext filename/date fields are gone (metadata rides the encrypted metadata blob; date was silently dropped anyway). - Silent leniencies removed: empty PATCH bodies are 400 (was: silent no-op that skipped all validation), Content-Type must be application/octet-stream (415), X-Capsule-Checksum is required and verified against the body before any write, oversized chunks are 413, and a duplicate content hash returns 409 duplicate_blob with the existing asset id (was: swallowed into a 500). - Every rejection now renders the {error, code} JSON shape with its compile-checked capsule_i18n::error_codes constant, stale-offset 409s carry the authoritative X-Capsule-Offset, HEAD reports state via the new X-Capsule-Upload-Status header (a HEAD response cannot carry a body), and the Pending->Uploading transition actually happens on the first accepted chunk. Deep envelope enforcement (invariants 1-8, 12's replay store, the CAS finalize, the discard sweeper and startup scrub) remains S-C1, as indexed in SLICES.md. --- capsule-api/upload/Cargo.toml | 2 +- capsule-api/upload/src/envelope.rs | 5 +- capsule-api/upload/src/error.rs | 177 ++++++++++++------- capsule-api/upload/src/models/requests.rs | 67 ++++++- capsule-api/upload/src/models/responses.rs | 70 ++++++-- capsule-api/upload/src/models/session.rs | 64 ++++++- capsule-api/upload/src/routes/tus.rs | 131 +++++++------- capsule-api/upload/src/service/storage.rs | 196 +++++---------------- capsule-api/upload/src/service/upload.rs | 165 +++++++++-------- capsule-api/upload/src/session/mod.rs | 57 +++++- capsule-sdk/src/upload.rs | 41 +++-- 11 files changed, 577 insertions(+), 398 deletions(-) diff --git a/capsule-api/upload/Cargo.toml b/capsule-api/upload/Cargo.toml index adbeccba..ad9bdede 100644 --- a/capsule-api/upload/Cargo.toml +++ b/capsule-api/upload/Cargo.toml @@ -15,6 +15,7 @@ capsule-api-environment = { path = "../environment", features = ["upload"] } capsule-api-model = { path = "../model" } capsule-api-service = { path = "../service" } capsule-core = { path = "../../capsule-core" } +capsule-i18n = { path = "../../capsule-i18n" } capsule-media = { path = "../../capsule-media" } bb8-redis = { workspace = true } eyre = { workspace = true } @@ -39,4 +40,3 @@ nanoid = { workspace = true } indexmap = { workspace = true } bytes = { workspace = true } jsonwebtoken = { workspace = true } -libc = "0.2.186" diff --git a/capsule-api/upload/src/envelope.rs b/capsule-api/upload/src/envelope.rs index ba171bd2..1587d2b4 100644 --- a/capsule-api/upload/src/envelope.rs +++ b/capsule-api/upload/src/envelope.rs @@ -7,8 +7,9 @@ //! `capsule_core::validation` — [`protocol_gate`] and [`check_manifest_envelope`] are //! ready to consume. What this middleware adds when `S-C1` lands is the transport glue: //! read the `X-Capsule-*` negotiation headers, run the gate before any state is written, -//! and map each rejection to its HTTP status plus `error.*` code (see the design docs' -//! API-surfaces rejection mapping). Until then it is an unmounted stub. +//! and map each rejection to its HTTP status plus `error.*` code — the upload doc's +//! Error Taxonomy table is the SSoT for that mapping (`UploadError::code()` already +//! carries the catalog constants). Until then it is an unmounted stub. //! //! [`protocol_gate`]: capsule_core::validation::protocol_gate //! [`check_manifest_envelope`]: capsule_core::validation::check_manifest_envelope diff --git a/capsule-api/upload/src/error.rs b/capsule-api/upload/src/error.rs index bb8c51ca..17bc7128 100644 --- a/capsule-api/upload/src/error.rs +++ b/capsule-api/upload/src/error.rs @@ -1,6 +1,13 @@ +use auth::models::errors::ApiError; +use capsule_i18n::error_codes; use salvo::prelude::*; use thiserror::Error; +/// The upload error domain. Every client-visible rejection maps to a stable +/// `error.upload.*` catalog code (the upload-protocol design doc's Error +/// Taxonomy is the SSoT); clients switch on the code, never on the bare HTTP +/// status. Deep enforcement of the envelope invariants is S-C1 — the variants +/// exist now so the taxonomy is frozen. #[allow(dead_code)] #[derive(Debug, Error)] pub(crate) enum UploadError { @@ -18,88 +25,128 @@ pub(crate) enum UploadError { RunError(#[from] bb8_redis::bb8::RunError), #[error("Session not found")] SessionNotFound, - #[error("Upload already complete")] - UploadComplete, - #[error("Upload session is being processed by another instance")] - UploadInstanceConflict, + #[error("Session is already finished or being processed")] + SessionNotActive, + #[error("Session is already being finalized")] + FinalizeInProgress, #[error("Invalid offset: expected {expected}, got {actual}")] InvalidOffset { expected: u64, actual: u64 }, #[error("Invalid upload: {0}")] InvalidUpload(String), + #[error("Upload exceeds its declared size")] + SizeExceeded, + #[error("This content is already stored as asset {asset_id}")] + DuplicateBlob { asset_id: String }, + #[error("Chunk body must be application/octet-stream")] + UnsupportedMediaType, + #[error("Empty chunk")] + EmptyChunk, + #[error("Chunk size must be 4 KiB-aligned (except the final chunk)")] + ChunkNotAligned, + #[error("Chunk exceeds the 16 MiB protocol maximum")] + ChunkTooLarge, + #[error("Missing or malformed X-Capsule-Checksum header")] + MissingChecksum, + #[error("Chunk checksum mismatch: header {header}, body {body}")] + ChunkChecksumMismatch { header: String, body: String }, + #[error("Content hash mismatch: expected {expected}, got {actual}")] + ContentHashMismatch { expected: String, actual: String }, #[error("Processing error: {0}")] ProcessingError(String), #[error("Parse error: {0}")] ParseError(#[from] std::string::ParseError), - #[error("Checksum mismatch: expected {expected}, got {actual}")] - ChecksumMismatch { expected: String, actual: String }, - #[error("Invalid chunk size: {0}")] - InvalidChunkSize(String), + #[error("Upload file length {on_disk} diverged from expected offset {expected}")] + StorageInconsistent { expected: u64, on_disk: u64 }, #[error("Unknown error: {0}")] Unknown(String), } +impl UploadError { + /// The stable `error.*` catalog code for this rejection, when one applies. + /// Constants come from `capsule_i18n::error_codes` so a typo is a compile + /// error and the code stays in sync with the canonical catalog. + pub(crate) fn code(&self) -> Option<&'static str> { + match self { + UploadError::FileTooLarge => Some(error_codes::UPLOAD_FILE_TOO_LARGE), + UploadError::SessionNotFound => Some(error_codes::UPLOAD_SESSION_NOT_FOUND), + UploadError::SessionNotActive => Some(error_codes::UPLOAD_SESSION_NOT_ACTIVE), + UploadError::FinalizeInProgress => Some(error_codes::UPLOAD_FINALIZE_IN_PROGRESS), + UploadError::InvalidOffset { .. } => Some(error_codes::UPLOAD_OFFSET_MISMATCH), + UploadError::SizeExceeded => Some(error_codes::UPLOAD_SIZE_EXCEEDED), + UploadError::DuplicateBlob { .. } => Some(error_codes::UPLOAD_DUPLICATE_BLOB), + UploadError::UnsupportedMediaType => Some(error_codes::UPLOAD_UNSUPPORTED_MEDIA_TYPE), + UploadError::EmptyChunk => Some(error_codes::UPLOAD_EMPTY_CHUNK), + UploadError::ChunkNotAligned => Some(error_codes::UPLOAD_CHUNK_NOT_ALIGNED), + UploadError::ChunkTooLarge => Some(error_codes::UPLOAD_CHUNK_TOO_LARGE), + UploadError::MissingChecksum => Some(error_codes::UPLOAD_MISSING_CHECKSUM), + UploadError::ChunkChecksumMismatch { .. } => { + Some(error_codes::UPLOAD_CHECKSUM_MISMATCH) + } + UploadError::ContentHashMismatch { .. } => { + Some(error_codes::UPLOAD_CONTENT_HASH_MISMATCH) + } + UploadError::StorageInconsistent { .. } => { + Some(error_codes::UPLOAD_STORAGE_INCONSISTENT) + } + UploadError::InvalidUpload(_) => Some(error_codes::UPLOAD_MALFORMED_REQUEST), + _ => None, + } + } + + fn status(&self) -> StatusCode { + match self { + UploadError::FileTooLarge | UploadError::ChunkTooLarge => StatusCode::PAYLOAD_TOO_LARGE, + UploadError::SessionNotFound => StatusCode::NOT_FOUND, + UploadError::SessionNotActive + | UploadError::FinalizeInProgress + | UploadError::DuplicateBlob { .. } + | UploadError::InvalidOffset { .. } => StatusCode::CONFLICT, + UploadError::UnsupportedMediaType => StatusCode::UNSUPPORTED_MEDIA_TYPE, + UploadError::InvalidUpload(_) + | UploadError::SizeExceeded + | UploadError::EmptyChunk + | UploadError::ChunkNotAligned + | UploadError::MissingChecksum + | UploadError::ChunkChecksumMismatch { .. } + | UploadError::ContentHashMismatch { .. } + | UploadError::ParseError(_) => StatusCode::BAD_REQUEST, + UploadError::IoError(_) + | UploadError::DbError(_) + | UploadError::SerdeError(_) + | UploadError::ValkeyError(_) + | UploadError::RunError(_) + | UploadError::ProcessingError(_) + | UploadError::StorageInconsistent { .. } + | UploadError::Unknown(_) => StatusCode::INTERNAL_SERVER_ERROR, + } + } +} + #[async_trait] impl Writer for UploadError { async fn write(self, _req: &mut Request, _depot: &mut Depot, res: &mut Response) { - let (status, message) = match self { - UploadError::FileTooLarge => ( - StatusCode::PAYLOAD_TOO_LARGE, - String::from("File exceeds size limit"), - ), - UploadError::IoError(_) => ( - StatusCode::INTERNAL_SERVER_ERROR, - String::from("File system error"), - ), - UploadError::DbError(_) => ( - StatusCode::INTERNAL_SERVER_ERROR, - String::from("Database error"), - ), - UploadError::SerdeError(_) => ( - StatusCode::INTERNAL_SERVER_ERROR, - String::from("Serialization error"), - ), - UploadError::ValkeyError(_) => ( - StatusCode::INTERNAL_SERVER_ERROR, - String::from("Cache error"), - ), - UploadError::RunError(_) => ( - StatusCode::INTERNAL_SERVER_ERROR, - String::from("Cache connection error"), - ), - UploadError::SessionNotFound => ( - StatusCode::NOT_FOUND, - String::from("Upload session not found"), - ), - UploadError::UploadComplete => ( - StatusCode::CONFLICT, - String::from("Upload already complete"), - ), - UploadError::InvalidOffset { expected, actual } => ( - StatusCode::CONFLICT, - format!("Invalid offset. Expected {expected}, got {actual}"), - ), - UploadError::InvalidUpload(msg) => (StatusCode::BAD_REQUEST, msg), - UploadError::ProcessingError(_) => ( - StatusCode::INTERNAL_SERVER_ERROR, - String::from("Processing error"), - ), - UploadError::ParseError(_) => (StatusCode::BAD_REQUEST, String::from("Parse error")), - UploadError::ChecksumMismatch { expected, actual } => ( - StatusCode::BAD_REQUEST, - format!("Checksum mismatch. Expected {expected}, got {actual}"), - ), - UploadError::InvalidChunkSize(msg) => ( - StatusCode::BAD_REQUEST, - format!("Invalid chunk size: {msg}"), - ), - UploadError::UploadInstanceConflict => ( - StatusCode::CONFLICT, - String::from("Upload session is being processed by another instance"), - ), - UploadError::Unknown(msg) => (StatusCode::INTERNAL_SERVER_ERROR, msg), + let status = self.status(); + // Internal errors keep their detail out of the response body. + let message = if status == StatusCode::INTERNAL_SERVER_ERROR { + match &self { + UploadError::StorageInconsistent { .. } => self.to_string(), + _ => String::from("Internal server error"), + } + } else { + self.to_string() }; + // Stale-offset conflicts carry the authoritative offset so the client + // can re-align without a extra HEAD round-trip. + if let UploadError::InvalidOffset { expected, .. } = &self { + res.add_header("X-Capsule-Offset", expected.to_string(), true) + .ok(); + } + res.status_code(status); - res.render(Text::Plain(message)); + match self.code() { + Some(code) => res.render(Json(ApiError::with_code(message, code))), + None => res.render(Json(ApiError::new(message))), + } } } diff --git a/capsule-api/upload/src/models/requests.rs b/capsule-api/upload/src/models/requests.rs index 719f8dc9..77987dbf 100644 --- a/capsule-api/upload/src/models/requests.rs +++ b/capsule-api/upload/src/models/requests.rs @@ -1,22 +1,71 @@ -use chrono::{DateTime, Utc}; use salvo::oapi::ToSchema; use serde::{Deserialize, Serialize}; -/// Request body for creating an upload session +use crate::models::session::BlobRole; + +/// Request body for creating an upload session. +/// +/// The transport JSON is strict (`deny_unknown_fields`): an unknown field is a +/// client bug and is rejected with `400 error.upload.malformed_request` rather +/// than silently ignored. Plaintext metadata (filename, capture date, …) is +/// deliberately absent — it rides the encrypted metadata blob, never the wire +/// request (upload-protocol design doc, §Chunk Rules and Strictness). #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +#[serde(deny_unknown_fields)] pub(crate) struct CreateUploadRequest { - /// Original filename from client - pub filename: String, - /// File size in bytes + /// Ciphertext size in bytes pub size: u64, - /// SHA-256 hash of the complete file (64-char lowercase hex) + /// Ciphertext content hash, lowercase hex; digest length fixed by `crypto_suite_id` pub hash: String, - /// MIME type (e.g., "image/jpeg") + /// MIME type (closed enum per protocol version; e.g. "image/jpeg") pub content_type: String, + /// Crypto suite the blob is sealed under + pub crypto_suite_id: u16, + /// Protocol date (`YYYY-MM-DD`) the client speaks + pub protocol_version: String, + /// The blob's role in its asset bundle + pub blob_role: BlobRole, + /// The unencrypted manifest fields the server validates (invariants 1–8, 15, 25). + /// The top-level `crypto_suite_id`/`protocol_version`/`album_id` MUST agree with + /// the envelope's — a contradiction is `400 error.upload.envelope_mismatch` (S-C1). + pub manifest_envelope: ManifestEnvelope, /// Optional album to add asset to pub album_id: Option, /// Optional owner ID (defaults to authenticated user) pub owner_id: Option, - /// Date asset was created/taken (from client EXIF or filesystem) - pub date: Option>, + /// Album-upgrade intent id (required only during an album upgrade ceremony) + pub intent_id: Option, +} + +/// The server-visible mirror of the signed manifest's envelope fields, as declared +/// at `POST /upload` (owned by the provenance design doc; validated per the +/// threat-model invariants). Strict like the rest of the transport JSON — the +/// Postel unknown-key tolerance applies to the signed CBOR interiors, not to this +/// JSON projection. +#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +#[serde(deny_unknown_fields)] +pub(crate) struct ManifestEnvelope { + pub crypto_suite_id: u16, + pub protocol_version: String, + pub album_id: Option, + /// The asset id this blob belongs to (UUIDv7, same id across sidecar/manifest) + pub file_id: String, + pub amk_version: u32, + /// Ciphertext content hash, lowercase hex — must equal the top-level `hash` + pub ciphertext_hash: String, + pub plaintext_size: u64, + /// STREAM plaintext chunk size (owned by the encryption doc) + pub chunk_size: u32, + /// `derived | wrapped` (closed enum owned by the provenance doc) + pub key_mode: String, + pub metadata_blob_hash: Option, + pub created_by_user: String, + pub created_by_device: String, + pub client_version: String, + /// RFC3339; gross-drift sanity checked (invariant 8) + pub timestamp: String, + /// Lifecycle action (closed enum; `create` for a fresh bundle) + pub action: String, + pub prior_provenance_hash: Option, + pub retention_until: Option, } diff --git a/capsule-api/upload/src/models/responses.rs b/capsule-api/upload/src/models/responses.rs index 9825b473..9b60a1b6 100644 --- a/capsule-api/upload/src/models/responses.rs +++ b/capsule-api/upload/src/models/responses.rs @@ -4,6 +4,7 @@ use salvo::oapi::{EndpointOutRegister, ToSchema}; use salvo::prelude::*; use serde::{Deserialize, Serialize}; +use crate::error::UploadError; use crate::models::session::{UploadSession, UploadSessionStatus}; /// Response for a successful upload creation @@ -41,6 +42,8 @@ pub(crate) enum CreateUploadResponses { Unauthorized(String), Forbidden, BadRequest(String), + /// A typed upload rejection: renders with its taxonomy status + error.* code. + Error(UploadError), InternalServerError(InternalServerError), } @@ -70,6 +73,9 @@ impl Writer for CreateUploadResponses { res.status_code(StatusCode::BAD_REQUEST); res.render(Text::Plain(msg)); } + Self::Error(e) => { + e.write(req, depot, res).await; + } Self::InternalServerError(e) => { e.write(req, depot, res).await; } @@ -98,6 +104,16 @@ impl EndpointOutRegister for CreateUploadResponses { String::from("403"), salvo::oapi::Response::new("Forbidden - insufficient permissions"), ); + operation.responses.insert( + String::from("409"), + salvo::oapi::Response::new( + "Conflict - content hash already finalized (error.upload.duplicate_blob; merge trigger)", + ), + ); + operation.responses.insert( + String::from("413"), + salvo::oapi::Response::new("Payload too large - declared size exceeds the limit"), + ); operation.responses.insert( String::from("500"), salvo::oapi::Response::new("Internal server error"), @@ -119,6 +135,8 @@ impl Writer for HeadUploadResponses { async fn write(self, req: &mut Request, depot: &mut Depot, res: &mut Response) { match self { Self::Success(response) => { + // A HEAD response carries no body; progress and state ride headers + // (X-Capsule-Upload-Status is census-registered). res.status_code(StatusCode::OK); res.add_header("X-Capsule-Offset", response.offset.to_string(), true) .ok(); @@ -126,8 +144,13 @@ impl Writer for HeadUploadResponses { res.add_header("X-Capsule-Content-Length", total.to_string(), true) .ok(); } + res.add_header( + "X-Capsule-Upload-Status", + response.status.as_header_value(), + true, + ) + .ok(); res.add_header("Cache-Control", "no-store", true).ok(); - res.render(Json(response)); } Self::Unauthorized(msg) => { res.status_code(StatusCode::UNAUTHORIZED); @@ -148,11 +171,11 @@ impl Writer for HeadUploadResponses { impl EndpointOutRegister for HeadUploadResponses { fn register(components: &mut salvo::oapi::Components, operation: &mut salvo::oapi::Operation) { + let _ = components; operation.responses.insert( String::from("200"), - salvo::oapi::Response::new("Upload status").add_content( - "application/json", - salvo::oapi::Content::new(HeadUploadResponse::to_schema(components)), + salvo::oapi::Response::new( + "Upload progress and state via X-Capsule-Offset / X-Capsule-Content-Length / X-Capsule-Upload-Status headers (no body)", ), ); operation.responses.insert( @@ -176,12 +199,14 @@ impl EndpointOutRegister for HeadUploadResponses { /// Responses for patch upload (append chunk) endpoint pub(crate) enum PatchUploadResponses { - Success { new_offset: u64 }, + Success { + new_offset: u64, + }, BadRequest(String), Unauthorized(String), Forbidden, - NotFound, - Conflict(String), + /// A typed upload rejection: renders with its taxonomy status + error.* code. + Error(UploadError), InternalServerError(InternalServerError), } @@ -205,12 +230,8 @@ impl Writer for PatchUploadResponses { Self::Forbidden => { res.status_code(StatusCode::FORBIDDEN); } - Self::NotFound => { - res.status_code(StatusCode::NOT_FOUND); - } - Self::Conflict(msg) => { - res.status_code(StatusCode::CONFLICT); - res.render(Text::Plain(msg)); + Self::Error(e) => { + e.write(req, depot, res).await; } Self::InternalServerError(e) => { e.write(req, depot, res).await; @@ -243,7 +264,19 @@ impl EndpointOutRegister for PatchUploadResponses { ); operation.responses.insert( String::from("409"), - salvo::oapi::Response::new("Conflict - offset mismatch"), + salvo::oapi::Response::new( + "Conflict - offset mismatch / chunk conflict / session not active", + ), + ); + operation.responses.insert( + String::from("413"), + salvo::oapi::Response::new("Payload too large - chunk exceeds the 16 MiB maximum"), + ); + operation.responses.insert( + String::from("415"), + salvo::oapi::Response::new( + "Unsupported media type - chunk body must be application/octet-stream", + ), ); operation.responses.insert( String::from("500"), @@ -258,6 +291,8 @@ pub(crate) enum DeleteUploadResponses { Unauthorized(String), Forbidden, NotFound, + /// A typed upload rejection: renders with its taxonomy status + error.* code. + Error(UploadError), InternalServerError(InternalServerError), } @@ -278,6 +313,9 @@ impl Writer for DeleteUploadResponses { Self::NotFound => { res.status_code(StatusCode::NOT_FOUND); } + Self::Error(e) => { + e.write(req, depot, res).await; + } Self::InternalServerError(e) => { e.write(req, depot, res).await; } @@ -303,6 +341,10 @@ impl EndpointOutRegister for DeleteUploadResponses { String::from("404"), salvo::oapi::Response::new("Upload session not found"), ); + operation.responses.insert( + String::from("409"), + salvo::oapi::Response::new("Conflict - finalization in progress is not interruptible"), + ); operation.responses.insert( String::from("500"), salvo::oapi::Response::new("Internal server error"), diff --git a/capsule-api/upload/src/models/session.rs b/capsule-api/upload/src/models/session.rs index 2c55e2c3..2c88bef5 100644 --- a/capsule-api/upload/src/models/session.rs +++ b/capsule-api/upload/src/models/session.rs @@ -2,6 +2,37 @@ use chrono::{DateTime, Utc}; use salvo::oapi::ToSchema; use serde::{Deserialize, Serialize}; +/// The blob's role within its asset bundle, declared at session creation and +/// recorded on the pending row (closed enum; the visibility gate and staged +/// uploads reason over it — see the upload-protocol design doc). +#[derive(Debug, Copy, Clone, Eq, PartialEq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub(crate) enum BlobRole { + Original, + Derivative, + Metadata, + Provenance, + Backup, +} + +impl BlobRole { + pub(crate) fn as_str(self) -> &'static str { + match self { + BlobRole::Original => "original", + BlobRole::Derivative => "derivative", + BlobRole::Metadata => "metadata", + BlobRole::Provenance => "provenance", + BlobRole::Backup => "backup", + } + } +} + +/// Volatile transfer state for one upload session. +/// +/// The record carries everything finalization needs — sizes, hash, crypto/protocol +/// pins, blob role, the manifest envelope, and the parties — so a session is +/// finalizable from its own record with no further client input (upload-protocol +/// design doc, §Endpoints). #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] pub(crate) struct UploadSession { /// Upload Session ID @@ -18,6 +49,18 @@ pub(crate) struct UploadSession { pub content_type: Option, /// Expected SHA-256 hash for verification on finalize (64-char lowercase hex) pub expected_hash: String, + /// Crypto suite the blob was sealed under (validated against the inventory by + /// the envelope gate, S-C1) + pub crypto_suite_id: u16, + /// Pinned protocol date (`YYYY-MM-DD`) from session creation + pub protocol_version: String, + /// This blob's role in its bundle + pub blob_role: BlobRole, + /// Album-upgrade intent, when part of an upgrade ceremony + pub intent_id: Option, + /// The server-visible manifest envelope, serialized JSON. Stored opaquely on + /// the session; deep validation is the envelope gate's job (S-C1). + pub manifest_envelope: String, // Upload state pub received_bytes: u64, @@ -26,15 +69,19 @@ pub(crate) struct UploadSession { /// Creation timestamp pub created_at: DateTime, - /// Expiration timestamp + /// Timestamp of the last accepted chunk (creation time if none). Anchors the + /// ≥1-hour survival floor; the pressure sweeper (S-C1) evicts + /// least-recently-progressed first. + pub last_progress_at: DateTime, + /// Expiration timestamp (the 24-hour cap) pub expires_at: DateTime, } #[derive(Debug, Copy, Clone, Eq, PartialEq, Serialize, Deserialize, ToSchema)] pub(crate) enum UploadSessionStatus { - /// Active with no active upload + /// Active with no accepted chunk yet Pending, - /// Active with an active upload + /// Active with at least one accepted chunk Uploading, /// Waiting for processing to complete WaitingForProcessing, @@ -63,4 +110,15 @@ impl UploadSessionStatus { UploadSessionStatus::Completed | UploadSessionStatus::FailedProcessing ) } + + /// Wire value for the `X-Capsule-Upload-Status` response header on `HEAD`. + pub(crate) fn as_header_value(&self) -> &'static str { + match self { + UploadSessionStatus::Pending => "pending", + UploadSessionStatus::Uploading => "uploading", + UploadSessionStatus::WaitingForProcessing => "waiting_for_processing", + UploadSessionStatus::Completed => "completed", + UploadSessionStatus::FailedProcessing => "failed_processing", + } + } } diff --git a/capsule-api/upload/src/routes/tus.rs b/capsule-api/upload/src/routes/tus.rs index dabbdb05..c939c5c5 100644 --- a/capsule-api/upload/src/routes/tus.rs +++ b/capsule-api/upload/src/routes/tus.rs @@ -1,7 +1,9 @@ use auth::utils::headers::validate_user_from_headers; +use capsule_core::utils::hash::hash_bytes; use salvo::oapi::extract::{JsonBody, PathParam}; use salvo::prelude::*; +use crate::error::UploadError; use crate::models::requests::CreateUploadRequest; use crate::models::responses::{ CreateUploadResponse, CreateUploadResponses, DeleteUploadResponses, HeadUploadResponse, @@ -10,15 +12,17 @@ use crate::models::responses::{ use crate::models::session::UploadSessionStatus; use crate::state::AppState; -// TODO: Thoroughly review and test this module. - // Constants for chunk sizes (4KB aligned) const KB: u64 = 1024; const CHUNK_SIZE_256KB: u64 = 256 * KB; const CHUNK_SIZE_1MB: u64 = 1024 * KB; const CHUNK_SIZE_4MB: u64 = 4 * 1024 * KB; +/// Protocol-surface maximum chunk size (upload-protocol doc, §Chunk Rules). +const MAX_CHUNK_SIZE: u64 = 16 * 1024 * KB; -/// Calculate suggested chunk size based on total file size +/// Calculate suggested chunk size based on total file size. +/// A starting point only — adaptation is the client's concern; these tiers are +/// server-tunable and not protocol surface. fn get_suggested_chunk_size(total_size: Option) -> u64 { match total_size { Some(size) if size < 10 * 1024 * KB => CHUNK_SIZE_256KB, // < 10MB @@ -51,7 +55,7 @@ pub async fn create_upload( }; // Use user_id as owner_id if not specified - let owner_id = request.owner_id.unwrap_or_else(|| user_id.clone()); + let owner_id = request.owner_id.clone().unwrap_or_else(|| user_id.clone()); // Permission check if owner is different if owner_id != user_id { @@ -69,15 +73,7 @@ pub async fn create_upload( match state .upload_service - .create_session( - &owner_id, - &user_id, - Some(request.content_type), - request.size, - request.hash, - request.album_id, - request.filename, - ) + .create_session(&request, &owner_id, &user_id) .await { Ok(session) => { @@ -88,7 +84,9 @@ pub async fn create_upload( suggested_chunk_size, }) } - Err(e) => CreateUploadResponses::InternalServerError(eyre::eyre!(e).into()), // TODO: It swallows all UploadError including duplicate. + // Typed rejections (duplicate_blob 409, malformed 400, …) render with + // their taxonomy status + error.* code instead of collapsing to 500. + Err(e) => CreateUploadResponses::Error(e), } } @@ -159,15 +157,24 @@ pub async fn patch_upload( Err(e) => return PatchUploadResponses::Unauthorized(e.to_string()), }; - // Verify ownership - only the uploader can append chunks - match state.upload_service.get_session(&id).await { + // Fetch the session once: ownership check + final-chunk alignment exemption. + let session = match state.upload_service.get_session(&id).await { Ok(Some(session)) => { if session.upload_user_id != user_id { return PatchUploadResponses::Forbidden; } + session } - Ok(None) => return PatchUploadResponses::NotFound, + Ok(None) => return PatchUploadResponses::Error(UploadError::SessionNotFound), Err(e) => return PatchUploadResponses::InternalServerError(eyre::eyre!(e).into()), + }; + + // Strict media type: the body is opaque ciphertext bytes (415 otherwise). + let is_octet_stream = req + .content_type() + .is_some_and(|mime| mime.essence_str() == "application/octet-stream"); + if !is_octet_stream { + return PatchUploadResponses::Error(UploadError::UnsupportedMediaType); } // Parse X-Capsule-Offset header @@ -181,8 +188,19 @@ pub async fn patch_upload( } }; - // Parse optional X-Capsule-Checksum header for verification - let _checksum: Option = req.header::("X-Capsule-Checksum"); + // X-Capsule-Checksum is REQUIRED: bare lowercase-hex SHA-256 of the chunk + // bytes. The (upload_id, offset, chunk_hash) idempotency tuple (invariant 12) + // is undefined without it. + let checksum = match req.header::("X-Capsule-Checksum") { + Some(c) + if c.len() == 64 + && c.bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) => + { + c + } + _ => return PatchUploadResponses::Error(UploadError::MissingChecksum), + }; let body = match req.payload().await { Ok(b) => b, @@ -192,34 +210,31 @@ pub async fn patch_upload( }; let bytes = body.clone(); - // Validate 4KB alignment + // Strictness: empty chunks are a client bug, never a silent no-op. if bytes.is_empty() { - // Empty chunk - nothing to do, just return current offset - return match state.upload_service.get_session(&id).await { - Ok(Some(session)) => PatchUploadResponses::Success { - new_offset: session.received_bytes, - }, - Ok(None) => PatchUploadResponses::NotFound, - Err(e) => PatchUploadResponses::InternalServerError(eyre::eyre!(e).into()), - }; + return PatchUploadResponses::Error(UploadError::EmptyChunk); } - if bytes.len() % 4096 != 0 { - // Allow non-aligned chunks only if it's the final chunk - let session = match state.upload_service.get_session(&id).await { - Ok(Some(s)) => s, - Ok(None) => return PatchUploadResponses::NotFound, - Err(e) => return PatchUploadResponses::InternalServerError(eyre::eyre!(e).into()), - }; + // Protocol-surface chunk ceiling. + if bytes.len() as u64 > MAX_CHUNK_SIZE { + return PatchUploadResponses::Error(UploadError::ChunkTooLarge); + } - if session.total_size > 0 { - let total = session.total_size; - let new_offset = session.received_bytes + bytes.len() as u64; - if new_offset != total { - return PatchUploadResponses::BadRequest( - "Chunk size must be 4KB aligned (except for final chunk)".to_string(), - ); - } + // Verify the checksum against the received bytes BEFORE any write: a + // mismatch persists nothing (transit-corruption defense, invariant 12). + let body_hash = hash_bytes(&bytes); + if body_hash != checksum { + return PatchUploadResponses::Error(UploadError::ChunkChecksumMismatch { + header: checksum, + body: body_hash, + }); + } + + // 4 KiB alignment for non-final chunks (invariant 10). + if bytes.len() % 4096 != 0 && session.total_size > 0 { + let new_offset = session.received_bytes + bytes.len() as u64; + if new_offset != session.total_size { + return PatchUploadResponses::Error(UploadError::ChunkNotAligned); } } @@ -229,26 +244,15 @@ pub async fn patch_upload( // Check for completion if session.total_size > 0 && session.received_bytes == session.total_size { // Attempt finalize - match state.upload_service.finalize_upload(&id).await { - Ok(_) => { - // Finalized successfully - } - Err(e) => { - return PatchUploadResponses::InternalServerError(eyre::eyre!(e).into()); - } + if let Err(e) = state.upload_service.finalize_upload(&id).await { + return PatchUploadResponses::Error(e); } } PatchUploadResponses::Success { new_offset: session.received_bytes, } } - Err(e) => { - if e.to_string().contains("Invalid offset") { - PatchUploadResponses::Conflict(e.to_string()) - } else { - PatchUploadResponses::InternalServerError(eyre::eyre!(e).into()) - } - } + Err(e) => PatchUploadResponses::Error(e), } } @@ -288,13 +292,12 @@ pub async fn delete_upload( match state.upload_service.cancel_upload(&id).await { Ok(()) => DeleteUploadResponses::Success, - Err(e) => { - if matches!(e, crate::error::UploadError::SessionNotFound) { - DeleteUploadResponses::NotFound - } else { - DeleteUploadResponses::InternalServerError(eyre::eyre!(e).into()) - } - } + Err(e) => match e { + UploadError::SessionNotFound => DeleteUploadResponses::NotFound, + // Finalization is not interruptible (409). + UploadError::SessionNotActive => DeleteUploadResponses::Error(e), + _ => DeleteUploadResponses::InternalServerError(eyre::eyre!(e).into()), + }, } } diff --git a/capsule-api/upload/src/service/storage.rs b/capsule-api/upload/src/service/storage.rs index 9c77371d..5d6c1969 100644 --- a/capsule-api/upload/src/service/storage.rs +++ b/capsule-api/upload/src/service/storage.rs @@ -1,6 +1,5 @@ -use std::fs::File; -#[cfg(target_os = "linux")] -use std::os::unix::io::{AsRawFd, RawFd}; +use std::fs::OpenOptions; +use std::io::Write; use std::path::PathBuf; use tokio::fs; @@ -8,177 +7,70 @@ use tokio::fs; use crate::config::UploadServerConfig; use crate::error::UploadError; -/// Service responsible for managing the physical storage of upload files and chunks on disk. +/// Service responsible for the physical storage of in-flight uploads on disk. +/// +/// Each session owns exactly one append-only file, `{upload_id}.bin`; accepted +/// chunks are appended in order. Sequential offsets are a hard protocol rule, so +/// append-only storage is correct by construction and there is no per-chunk +/// staging or assembly step (upload-protocol design doc, §Append-Only Storage). #[derive(Clone)] pub(crate) struct StorageService { config: UploadServerConfig, } -// Struct for FICLONERANGE ioctl -// Struct for FICLONERANGE ioctl -#[repr(C)] -#[cfg(target_os = "linux")] -struct FileCloneRange { - src_fd: i64, - src_offset: u64, - src_length: u64, - dest_offset: u64, -} - impl StorageService { pub(crate) fn new(config: UploadServerConfig) -> Self { Self { config } } - /// Generates the filesystem path for a specific chunk of an upload. - /// - /// Chunks are stored with the naming convention: `{upload_id}_{chunk_index}.part` - pub(crate) fn get_chunk_path(&self, upload_id: &str, chunk_index: u64) -> PathBuf { - self.config - .upload_dir - .join(format!("{upload_id}_{chunk_index}.part")) + /// The session's single upload file. + pub(crate) fn get_upload_path(&self, upload_id: &str) -> PathBuf { + self.config.upload_dir.join(format!("{upload_id}.bin")) } - /// Combines all chunk files for an upload into a single destination file. + /// Appends a chunk at `offset`, returning the file's new length. /// - /// This method sequentially reads each chunk (0 to num_chunks-1) and writes it to the - /// destination file defined by `get_upload_path`. After successful combination, the original chunk files are deleted. - /// - /// # Arguments - /// * `upload_id` - The unique identifier of the upload session - /// * `num_chunks` - The total number of chunks to combine - pub(crate) async fn combine_chunks( + /// The file's current length is cross-checked against the expected offset + /// before writing: the file is the on-disk truth and the session counter its + /// cache, so a divergence is a server-side inconsistency + /// (`error.upload.storage_inconsistent`), never silently absorbed. The chunk + /// is flushed to disk before returning — an acknowledged byte is a byte on + /// disk. + pub(crate) async fn append_at( &self, upload_id: &str, - num_chunks: u64, - ) -> Result { - let final_path = self.get_upload_path(upload_id); - - let final_path_clone = final_path.clone(); - let upload_id_str = upload_id.to_string(); - let storage = self.clone(); + offset: u64, + data: bytes::Bytes, + ) -> Result { + let path = self.get_upload_path(upload_id); tokio::task::spawn_blocking(move || { - let mut dest = - File::create(&final_path_clone).map_err(|e| UploadError::Unknown(e.to_string()))?; - - let mut current_offset = 0; - - for i in 0..num_chunks { - let chunk_path = storage.get_chunk_path(&upload_id_str, i); - let mut source = - File::open(&chunk_path).map_err(|e| UploadError::Unknown(e.to_string()))?; - - let written = Self::append_chunk(&mut dest, &mut source, current_offset, false)?; - current_offset += written; + let mut file = OpenOptions::new().create(true).append(true).open(&path)?; + + let on_disk = file.metadata()?.len(); + if on_disk != offset { + return Err(UploadError::StorageInconsistent { + expected: offset, + on_disk, + }); } - Ok::<(), UploadError>(()) + + file.write_all(&data)?; + file.sync_data()?; + Ok::(on_disk + data.len() as u64) }) .await - .map_err(|e| UploadError::Unknown(e.to_string()))??; - - // Clean up chunks after successful merge - self.delete_chunks(upload_id).await?; - - Ok(final_path) + .map_err(|e| UploadError::Unknown(e.to_string()))? } - /// Appends a source file to a destination file, attempting reflink first if available. - /// - /// # Arguments - /// * `dest` - Destination file (opened for writing) - /// * `source` - Source file (opened for reading) - /// * `current_offset` - Current offset in destination file where source should be appended - /// * `force_copy` - If true, skips reflink optimization and forces standard copy. useful for testing. - fn append_chunk( - dest: &mut File, - source: &mut File, - current_offset: u64, - force_copy: bool, - ) -> Result { - let metadata = source - .metadata() - .map_err(|e| UploadError::Unknown(e.to_string()))?; - let size = metadata.len(); - - // `reflink_success` is only reassigned inside the Linux-only reflink block - // below, so on non-Linux targets the `mut` is genuinely unused. - #[cfg_attr(not(target_os = "linux"), allow(unused_mut))] - let mut reflink_success = false; - - if !force_copy { - // Try reflink first. This is a Linux-specific optimization. - #[cfg(target_os = "linux")] - { - let dest_fd = AsRawFd::as_raw_fd(dest); - let src_fd = AsRawFd::as_raw_fd(source); - reflink_success = unsafe { attempt_reflink(dest_fd, src_fd, size, current_offset) }; - } + /// Removes the session's upload file, if present. Used on cancellation, + /// discard, and failed finalization. + pub(crate) async fn remove(&self, upload_id: &str) -> Result<(), UploadError> { + let path = self.get_upload_path(upload_id); + match fs::remove_file(&path).await { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e.into()), } - - if !reflink_success { - // Fallback to copy - use std::io::{Seek, SeekFrom}; - dest.seek(SeekFrom::Start(current_offset)) - .map_err(|e| UploadError::Unknown(e.to_string()))?; - std::io::copy(source, dest).map_err(|e| UploadError::Unknown(e.to_string()))?; - } - - Ok(size) - } - - /// Counts how many sequential chunks exist on disk for a given upload_id. - /// - /// Checks for chunks starting from index 0 and increments until a gap is found. - pub(crate) async fn count_chunks(&self, upload_id: &str) -> Result { - let mut count = 0; - loop { - let path = self.get_chunk_path(upload_id, count); - if fs::metadata(&path).await.is_err() { - break; - } - count += 1; - } - Ok(count) } - - /// Delete all chunks for an upload. Used for cleanup on cancellation or after combining. - /// - /// Returns the number of chunks successfully deleted. - pub(crate) async fn delete_chunks(&self, upload_id: &str) -> Result { - let mut deleted = 0; - loop { - let chunk_path = self.get_chunk_path(upload_id, deleted); - if fs::metadata(&chunk_path).await.is_ok() { - if let Err(e) = fs::remove_file(&chunk_path).await { - tracing::warn!("Failed to delete chunk {}: {}", chunk_path.display(), e); - } - deleted += 1; - } else { - break; - } - } - Ok(deleted) - } - - /// Gets the path for the final combined upload file (.bin). - pub(crate) fn get_upload_path(&self, upload_id: &str) -> PathBuf { - self.config.upload_dir.join(format!("{upload_id}.bin")) - } -} - -#[cfg(target_os = "linux")] -unsafe fn attempt_reflink(dest_fd: RawFd, src_fd: RawFd, len: u64, dest_offset: u64) -> bool { - // FICLONERANGE might not be in the libc crate depending on version/OS - // but on Linux it's standard. - // If it's missing, this will fail to compile, and we'll fix it. - let args = FileCloneRange { - src_fd: src_fd as i64, - src_offset: 0, - src_length: len, - dest_offset, - }; - - let ret = unsafe { libc::ioctl(dest_fd, libc::FICLONERANGE, &args) }; - ret == 0 } diff --git a/capsule-api/upload/src/service/upload.rs b/capsule-api/upload/src/service/upload.rs index e8cdbb7d..6c741baf 100644 --- a/capsule-api/upload/src/service/upload.rs +++ b/capsule-api/upload/src/service/upload.rs @@ -6,10 +6,10 @@ use entity::asset; use nanoid::nanoid; use sea_orm::{DatabaseConnection, TransactionTrait}; use service::{album as AlbumService, asset as AssetService}; -use tokio::fs; use crate::config::UploadServerConfig; use crate::error::UploadError; +use crate::models::requests::CreateUploadRequest; use crate::models::session::{UploadSession, UploadSessionStatus}; use crate::service::processing::ProcessingService; use crate::service::storage::StorageService; @@ -24,7 +24,6 @@ pub(crate) struct UploadService { conn: DatabaseConnection, } -// TODO: Update all methods to use more explicit error types impl UploadService { pub(crate) fn new( config: UploadServerConfig, @@ -41,22 +40,22 @@ impl UploadService { } } - /// Create a new upload session with asset record in Postgres - #[allow(clippy::too_many_arguments)] + /// Create a new upload session with asset record in Postgres. + /// + /// The session record captures the full creation request (crypto/protocol + /// pins, blob role, manifest envelope) so finalization needs no further + /// client input. Deep envelope validation (invariants 1–8) is the envelope + /// gate's job (S-C1); this path enforces shape and access only. pub(crate) async fn create_session( &self, + request: &CreateUploadRequest, owner_id: &str, upload_user_id: &str, - content_type: Option, - total_size: u64, - expected_hash: String, - album_id: Option, - original_filename: String, ) -> Result { let upload_id = nanoid!(); // Validate Album access if provided - if let Some(album_id) = &album_id { + if let Some(album_id) = &request.album_id { match AlbumService::Query::get_album_access(&self.conn, owner_id, album_id).await { Ok(access) => { if !access.is_some_and(|a| a.is_write()) { @@ -71,60 +70,67 @@ impl UploadService { } } - // Check for duplicate hash - asset with same hash already exists for this user + // Duplicate hash: a finalized asset with this hash already exists for the + // user. Per the protocol's idempotency contract this is a 409 carrying the + // existing asset reference — the client's merge trigger, not a 500. + // (Returning an *active* session for the tuple, and closing the TOCTOU + // race with a single SELECT..FOR UPDATE transaction, lands with S-C1.) if let Some(existing) = - AssetService::Query::find_by_hash_for_user(&self.conn, upload_user_id, &expected_hash) + AssetService::Query::find_by_hash_for_user(&self.conn, upload_user_id, &request.hash) .await .map_err(|e| UploadError::Unknown(e.to_string()))? { - return Err(UploadError::InvalidUpload(format!( - "Asset with this hash already exists: {}", - existing.id - ))); + return Err(UploadError::DuplicateBlob { + asset_id: existing.id, + }); } // Determine asset type from content_type - let asset_type = content_type.as_ref().map_or(asset::AssetType::Photo, |ct| { - if ct.starts_with("image/") { - asset::AssetType::Photo - } else if ct.starts_with("video/") { - asset::AssetType::Video - } else { - asset::AssetType::Photo - } - }); + let asset_type = if request.content_type.starts_with("video/") { + asset::AssetType::Video + } else { + asset::AssetType::Photo + }; - // Create pending asset in Postgres with uploaded=false + // Create pending asset in Postgres with uploaded=false. + // LEGACY-PLAINTEXT (frozen, S-C1/S-G3): the plaintext-era `original_filename` + // column gets the opaque upload id — the wire request deliberately carries + // no filename (plaintext metadata rides the encrypted metadata blob). let asset = AssetService::Mutation::create_pending( &self.conn, owner_id.to_string(), upload_user_id.to_string(), - album_id.clone(), + request.album_id.clone(), asset_type, - original_filename, - total_size as i64, - expected_hash.clone(), - content_type - .clone() - .unwrap_or_else(|| "application/octet-stream".to_string()), - None, // date will be extracted on finalize + upload_id.clone(), + request.size as i64, + request.hash.clone(), + request.content_type.clone(), + None, ) .await .map_err(|e| UploadError::Unknown(e.to_string()))?; + let now = Utc::now(); let session = UploadSession { id: upload_id.clone(), asset_id: asset.id.clone(), owner_id: owner_id.to_string(), upload_user_id: upload_user_id.to_string(), - album_id, - content_type, - expected_hash, + album_id: request.album_id.clone(), + content_type: Some(request.content_type.clone()), + expected_hash: request.hash.clone(), + crypto_suite_id: request.crypto_suite_id, + protocol_version: request.protocol_version.clone(), + blob_role: request.blob_role, + intent_id: request.intent_id.clone(), + manifest_envelope: serde_json::to_string(&request.manifest_envelope)?, received_bytes: 0, - total_size, + total_size: request.size, status: UploadSessionStatus::Pending, - created_at: Utc::now(), - expires_at: Utc::now() + chrono::Duration::hours(24), + created_at: now, + last_progress_at: now, + expires_at: now + chrono::Duration::hours(24), }; // Create session in Redis (atomic HSET) @@ -140,7 +146,8 @@ impl UploadService { self.session_manager.get(upload_id).await } - /// List sessions by owner ID + /// List active sessions for a caller (the uploader-scoped index is S-C1; the + /// current index keys by owner, which coincides for self-uploads). pub(crate) async fn list_sessions_by_owner( &self, owner_id: &str, @@ -172,8 +179,10 @@ impl UploadService { .await? .ok_or(UploadError::SessionNotFound)?; - if session.status.is_inactive() { - return Err(UploadError::UploadComplete); + if session.status.is_inactive() + || session.status == UploadSessionStatus::WaitingForProcessing + { + return Err(UploadError::SessionNotActive); } // Validate offset matches current received_bytes @@ -189,30 +198,33 @@ impl UploadService { // Validate size limit before writing let new_size = session.received_bytes + chunk_len; if session.total_size > 0 && new_size > session.total_size { - return Err(UploadError::InvalidUpload( - "Upload exceeds declared total size".to_string(), - )); + return Err(UploadError::SizeExceeded); } if new_size > self.config.max_file_size as u64 { return Err(UploadError::FileTooLarge); } - // Count existing chunks - let chunk_count = self.storage.count_chunks(upload_id).await?; + // Append to the session's single upload file (durability before ACK; the + // file length is cross-checked against the offset inside). + self.storage.append_at(upload_id, offset, data).await?; - // Write chunk to disk - let chunk_path = self.storage.get_chunk_path(upload_id, chunk_count); - fs::write(&chunk_path, &data).await?; + // First accepted chunk transitions Pending -> Uploading (observable via HEAD). + if session.received_bytes == 0 && session.status == UploadSessionStatus::Pending { + self.session_manager + .update_status(upload_id, UploadSessionStatus::Uploading) + .await?; + } - // Atomically increment received_bytes in Redis + // Atomically increment received_bytes and refresh the survival-floor anchor. let new_received_bytes = self .session_manager .increment_received_bytes(upload_id, chunk_len) .await?; + self.session_manager.touch_progress(upload_id).await?; - // Re-fetch session to get updated state let updated_session = UploadSession { received_bytes: new_received_bytes, + status: UploadSessionStatus::Uploading, ..session }; @@ -229,10 +241,10 @@ impl UploadService { .ok_or(UploadError::SessionNotFound)?; match session.status { - status if status.is_inactive() => return Err(UploadError::UploadComplete), - UploadSessionStatus::FailedProcessing => { - return Err(UploadError::UploadInstanceConflict); + UploadSessionStatus::WaitingForProcessing => { + return Err(UploadError::FinalizeInProgress); } + status if status.is_inactive() => return Err(UploadError::SessionNotActive), _ => {} } @@ -243,17 +255,15 @@ impl UploadService { ))); } - // Mark session as processing + // Mark session as processing. (An atomic status compare-and-set replaces + // this read-then-write guard with S-C1.) self.session_manager .update_status(upload_id, UploadSessionStatus::WaitingForProcessing) .await?; - // Combine chunks - let num_chunks = self.storage.count_chunks(upload_id).await?; - - let final_path = self.storage.combine_chunks(upload_id, num_chunks).await?; - - // Verify hash (run sync hash in blocking task) + // The session's single upload file is already the complete blob — no + // assembly step. Verify its hash on the blocking pool. + let final_path = self.storage.get_upload_path(upload_id); let hash_path = final_path.clone(); let actual_hash = tokio::task::spawn_blocking(move || get_file_hash(&hash_path)) .await @@ -261,25 +271,27 @@ impl UploadService { .map_err(|e| UploadError::ProcessingError(e.to_string()))?; if actual_hash != session.expected_hash { - // Hash mismatch - clean up and delete asset - if let Err(e) = fs::remove_file(&final_path).await { + // Hash mismatch — treated as corruption: delete the upload file and + // the pending asset row, fail the session. + if let Err(e) = self.storage.remove(upload_id).await { tracing::warn!("Failed to delete file after hash mismatch: {}", e); } - // Delete the pending asset let _ = AssetService::Mutation::delete(&self.conn, &session.asset_id).await; - // Update session status self.session_manager .update_status(upload_id, UploadSessionStatus::FailedProcessing) .await?; - return Err(UploadError::ChecksumMismatch { + return Err(UploadError::ContentHashMismatch { expected: session.expected_hash, actual: actual_hash, }); } + // Envelope re-validation inside the finalization transaction (invariant 15) + // is wired by the envelope gate slice (S-C1); the envelope is on the session. + // Extract Metadata let metadata = self .processing_service @@ -314,6 +326,13 @@ impl UploadService { // Get session to find asset_id let session = self.get_session(upload_id).await?; + // Finalization is not interruptible. + if let Some(session) = &session + && session.status == UploadSessionStatus::WaitingForProcessing + { + return Err(UploadError::SessionNotActive); + } + // Delete asset from Postgres if session exists if let Some(session) = &session && let Err(e) = AssetService::Mutation::delete(&self.conn, &session.asset_id).await @@ -325,9 +344,13 @@ impl UploadService { ); } - // Delete chunks from disk - if let Err(e) = self.storage.delete_chunks(upload_id).await { - tracing::warn!("Failed to delete chunks for upload {}: {}", upload_id, e); + // Delete the upload file from disk + if let Err(e) = self.storage.remove(upload_id).await { + tracing::warn!( + "Failed to delete upload file for upload {}: {}", + upload_id, + e + ); } // Remove session from Redis diff --git a/capsule-api/upload/src/session/mod.rs b/capsule-api/upload/src/session/mod.rs index a0f3297b..e5252cb2 100644 --- a/capsule-api/upload/src/session/mod.rs +++ b/capsule-api/upload/src/session/mod.rs @@ -7,7 +7,7 @@ use bb8_redis::redis::AsyncCommands; use chrono::{DateTime, Utc}; use crate::error::UploadError; -use crate::models::session::{UploadSession, UploadSessionStatus}; +use crate::models::session::{BlobRole, UploadSession, UploadSessionStatus}; // TODO: Validate this code @@ -53,6 +53,19 @@ impl UploadSessionManager { session.received_bytes.to_string().into_bytes(), ), ("expected_hash", session.expected_hash.clone().into_bytes()), + ( + "crypto_suite_id", + session.crypto_suite_id.to_string().into_bytes(), + ), + ( + "protocol_version", + session.protocol_version.clone().into_bytes(), + ), + ("blob_role", session.blob_role.as_str().as_bytes().to_vec()), + ( + "manifest_envelope", + session.manifest_envelope.clone().into_bytes(), + ), ( "status", serde_json::to_string(&session.status) @@ -60,6 +73,10 @@ impl UploadSessionManager { .into_bytes(), ), ("created_at", session.created_at.to_rfc3339().into_bytes()), + ( + "last_progress_at", + session.last_progress_at.to_rfc3339().into_bytes(), + ), ("expires_at", session.expires_at.to_rfc3339().into_bytes()), ]; @@ -70,6 +87,9 @@ impl UploadSessionManager { if let Some(content_type) = &session.content_type { fields.push(("content_type", content_type.as_bytes().to_vec())); } + if let Some(intent_id) = &session.intent_id { + fields.push(("intent_id", intent_id.as_bytes().to_vec())); + } // Use HSET with multiple fields let mut cmd = bb8_redis::redis::cmd("HSET"); @@ -115,6 +135,17 @@ impl UploadSessionManager { Ok(new_value as u64) } + /// Record chunk progress: refreshes `last_progress_at`, the anchor of the + /// ≥1-hour survival floor (the pressure sweeper itself is S-C1). + pub(crate) async fn touch_progress(&self, upload_id: &str) -> Result<(), UploadError> { + let mut conn = self.pool.get().await?; + let key = self.key(upload_id); + let _: () = conn + .hset(&key, "last_progress_at", Utc::now().to_rfc3339()) + .await?; + Ok(()) + } + /// Atomically update the upload status. pub(crate) async fn update_status( &self, @@ -209,6 +240,20 @@ impl UploadSessionManager { .map_err(|e| UploadError::Unknown(format!("Invalid total_size: {e}")))?; let expected_hash: String = get_string("expected_hash")?; + let crypto_suite_id: u16 = get_string("crypto_suite_id")? + .parse() + .map_err(|e| UploadError::Unknown(format!("Invalid crypto_suite_id: {e}")))?; + let protocol_version = get_string("protocol_version")?; + let blob_role_str = get_string("blob_role")?; + let blob_role: BlobRole = + serde_json::from_str(&format!("\"{blob_role_str}\"")).map_err(|e| { + UploadError::Unknown(format!("Invalid blob_role '{blob_role_str}': {e}")) + })?; + let manifest_envelope = get_string("manifest_envelope")?; + let intent_id = fields + .get("intent_id") + .and_then(|bytes| String::from_utf8(bytes.clone()).ok()); + let status_str = get_string("status")?; let status: UploadSessionStatus = serde_json::from_str(&status_str) .map_err(|e| UploadError::Unknown(format!("Invalid status '{status_str}': {e}")))?; @@ -217,6 +262,10 @@ impl UploadSessionManager { .parse() .map_err(|e| UploadError::Unknown(format!("Invalid created_at: {e}")))?; + let last_progress_at: DateTime = get_string("last_progress_at")? + .parse() + .map_err(|e| UploadError::Unknown(format!("Invalid last_progress_at: {e}")))?; + let expires_at: DateTime = get_string("expires_at")? .parse() .map_err(|e| UploadError::Unknown(format!("Invalid expires_at: {e}")))?; @@ -229,10 +278,16 @@ impl UploadSessionManager { album_id, content_type, expected_hash, + crypto_suite_id, + protocol_version, + blob_role, + intent_id, + manifest_envelope, received_bytes, total_size, status, created_at, + last_progress_at, expires_at, }) } diff --git a/capsule-sdk/src/upload.rs b/capsule-sdk/src/upload.rs index 1b825390..2d554a36 100644 --- a/capsule-sdk/src/upload.rs +++ b/capsule-sdk/src/upload.rs @@ -20,6 +20,11 @@ const CHUNK_SIZE_16MB: u64 = 16 * 1024 * KB; /// All chunks MUST be multiples of 4KB (4096 bytes) const ALIGNMENT: u64 = 4096; +/// Protocol-surface chunk bounds (upload-protocol doc, §Chunk Rules and +/// Strictness). Every adaptive tier range sits inside these; the server rejects +/// outside them (400 / 413). +pub const PROTOCOL_MIN_CHUNK: u64 = 4096; +pub const PROTOCOL_MAX_CHUNK: u64 = CHUNK_SIZE_16MB; /// Window size for throughput measurements const THROUGHPUT_WINDOW_SECS: f64 = 30.0; /// Minimum bytes before scaling chunk size @@ -145,7 +150,12 @@ impl AdaptiveChunkSizeStrategy { } } - /// Get the next chunk size to use + /// Get the next chunk size to use. + /// + /// Alignment is a hard guarantee by construction: every candidate size is a + /// doubling/halving of a 4 KiB-aligned tier bound, so this can never return + /// an unaligned size — the debug_assert is a tripwire for future edits, not + /// a runtime dependency. pub fn next_chunk_size(&self) -> u64 { debug_assert!( self.current_size.is_multiple_of(ALIGNMENT), @@ -171,7 +181,6 @@ impl UploadClient { /// /// # Arguments /// * `file_path` - Path to the file to upload - /// * `filename` - Optional filename to use (defaults to file name from path) /// * `content_type` - Optional content type (defaults to auto-detection) /// /// # Returns @@ -179,27 +188,26 @@ impl UploadClient { pub async fn upload_file( &self, _file_path: &Path, - _filename: Option<&str>, _content_type: Option<&str>, ) -> Result { - // TODO: Implement file upload - // 1. Get file size and metadata - // 2. Create upload session via API - // 3. Create AdaptiveChunkSizeStrategy based on file size - // 4. Upload chunks with adaptive sizing - // 5. Return session ID on completion - todo!("Upload file not yet implemented - waiting for API client generation") + // S-D1: 1. size + ciphertext hash; 2. create session (JSON body per the + // upload-protocol doc — size/hash/content_type/crypto_suite_id/ + // protocol_version/blob_role/manifest_envelope); 3. adaptive strategy; + // 4. chunk loop with the code-driven recovery matrix; 5. session id. + // No filename on the wire: plaintext metadata rides the encrypted + // metadata blob. + todo!("S-D1: upload driver — see SLICES.md") } /// Create an upload session pub async fn create_session( &self, _total_size: u64, - _filename: Option<&str>, _content_type: Option<&str>, ) -> Result { - // TODO: Call POST /upload with X-Capsule-Content-Length header - todo!("Create session not yet implemented") + // S-D1: POST /upload with the JSON body from the upload-protocol doc's + // endpoint table (size is the `size` body field, NOT a header). + todo!("S-D1: create session — see SLICES.md") } /// Upload a single chunk @@ -209,9 +217,10 @@ impl UploadClient { _data: &[u8], _offset: u64, ) -> Result { - // TODO: Call PATCH /upload/{id} with X-Capsule-Offset header - // Returns the new offset on success - todo!("Upload chunk not yet implemented") + // S-D1: PATCH /upload/{id} with Content-Type: application/octet-stream, + // X-Capsule-Offset, and the REQUIRED X-Capsule-Checksum (lowercase-hex + // SHA-256 of the chunk bytes). Returns the new offset on success. + todo!("S-D1: upload chunk — see SLICES.md") } } From 6185151b6cfe17263813f50fa20c3789c268bbe8 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:40:46 -0400 Subject: [PATCH 20/58] refactor(core)!: fold capsule-media into capsule_core::media MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The standalone capsule-media crate had exactly one consumer (capsule-api-upload) and no dependency on capsule-core, so it was a crate boundary without an owner: not independently versioned, not independently shipped, and one more workspace member to wire. It is now the `media` module of capsule-core behind a non-default `media` feature — the default build (crypto data plane, CLI, FFI, future WASM) compiles none of the media stack or its dependencies, and capsule-api-upload opts in explicitly. Mechanical consequences: the media dependencies become optional deps of capsule-core; the crate's internal `fs` sub-feature is folded into `media` (no consumer used media-without-fs); the three examples move to capsule-core gated on the feature; and the PngImage #[ignore] test now carries its slice ID (S-B1) like every other contract-seam test. Module Map and SLICES.md updated to the new path. --- Cargo.lock | 33 +++++---------- Cargo.toml | 2 - README.md | 4 +- SLICES.md | 3 +- capsule-api/upload/Cargo.toml | 3 +- capsule-api/upload/src/service/processing.rs | 2 +- capsule-core/Cargo.toml | 40 +++++++++++++++++++ .../examples/extract_metadata.rs | 6 +-- .../examples/generate_derivative.rs | 10 ++--- .../examples/generate_lqip.rs | 6 +-- capsule-core/src/lib.rs | 2 + .../src/media}/core/mod.rs | 0 .../src/media}/core/transcode.rs | 4 +- .../src/media}/core/types/mod.rs | 4 +- .../src => capsule-core/src/media}/fs/ext.rs | 6 +-- .../src => capsule-core/src/media}/fs/mod.rs | 34 ++++++++-------- .../src/media}/image/buffer.rs | 2 +- .../src/media}/image/formats/avif.rs | 30 +++++++------- .../src/media}/image/formats/bmp.rs | 30 +++++++------- .../src/media}/image/formats/dng.rs | 30 +++++++------- .../src/media}/image/formats/gif.rs | 30 +++++++------- .../src/media}/image/formats/heif.rs | 30 +++++++------- .../src/media}/image/formats/jpeg.rs | 30 +++++++------- .../src/media}/image/formats/jxl.rs | 30 +++++++------- .../src/media}/image/formats/mod.rs | 0 .../src/media}/image/formats/png.rs | 30 +++++++------- .../src/media}/image/formats/raw.rs | 28 ++++++------- .../src/media}/image/formats/tiff.rs | 30 +++++++------- .../src/media}/image/formats/webp.rs | 30 +++++++------- .../src/media}/image/lqip.rs | 6 +-- .../src/media}/image/metadata/exposure.rs | 0 .../src/media}/image/metadata/iptc.rs | 0 .../src/media}/image/metadata/mod.rs | 24 +++++------ .../src/media}/image/metadata/motion.rs | 0 .../src/media}/image/metadata/raw.rs | 0 .../src/media}/image/mod.rs | 20 +++++----- .../src/media}/image/presets.rs | 6 +-- .../src/media}/image/types.rs | 0 .../src/media}/metadata/c2pa.rs | 0 .../src/media}/metadata/exif.rs | 2 +- .../src/media}/metadata/geo.rs | 0 .../src/media}/metadata/icc.rs | 0 .../src/media}/metadata/mod.rs | 0 .../src/media}/metadata/orientation.rs | 0 .../src/media}/metadata/xmp.rs | 0 capsule-core/src/media/mod.rs | 10 +++++ .../src/media}/video/mod.rs | 0 .../src/media}/video/presets.rs | 6 +-- .../src/media}/video/types.rs | 0 .../src/content/docs/design/module-map.md | 7 ++-- capsule-media/Cargo.toml | 24 ----------- capsule-media/src/lib.rs | 6 --- 52 files changed, 304 insertions(+), 296 deletions(-) rename {capsule-media => capsule-core}/examples/extract_metadata.rs (80%) rename {capsule-media => capsule-core}/examples/generate_derivative.rs (78%) rename {capsule-media => capsule-core}/examples/generate_lqip.rs (84%) rename {capsule-media/src => capsule-core/src/media}/core/mod.rs (100%) rename {capsule-media/src => capsule-core/src/media}/core/transcode.rs (82%) rename {capsule-media/src => capsule-core/src/media}/core/types/mod.rs (77%) rename {capsule-media/src => capsule-core/src/media}/fs/ext.rs (91%) rename {capsule-media/src => capsule-core/src/media}/fs/mod.rs (84%) rename {capsule-media/src => capsule-core/src/media}/image/buffer.rs (99%) rename {capsule-media/src => capsule-core/src/media}/image/formats/avif.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/formats/bmp.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/formats/dng.rs (70%) rename {capsule-media/src => capsule-core/src/media}/image/formats/gif.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/formats/heif.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/formats/jpeg.rs (84%) rename {capsule-media/src => capsule-core/src/media}/image/formats/jxl.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/formats/mod.rs (100%) rename {capsule-media/src => capsule-core/src/media}/image/formats/png.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/formats/raw.rs (77%) rename {capsule-media/src => capsule-core/src/media}/image/formats/tiff.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/formats/webp.rs (71%) rename {capsule-media/src => capsule-core/src/media}/image/lqip.rs (97%) rename {capsule-media/src => capsule-core/src/media}/image/metadata/exposure.rs (100%) rename {capsule-media/src => capsule-core/src/media}/image/metadata/iptc.rs (100%) rename {capsule-media/src => capsule-core/src/media}/image/metadata/mod.rs (86%) rename {capsule-media/src => capsule-core/src/media}/image/metadata/motion.rs (100%) rename {capsule-media/src => capsule-core/src/media}/image/metadata/raw.rs (100%) rename {capsule-media/src => capsule-core/src/media}/image/mod.rs (95%) rename {capsule-media/src => capsule-core/src/media}/image/presets.rs (99%) rename {capsule-media/src => capsule-core/src/media}/image/types.rs (100%) rename {capsule-media/src => capsule-core/src/media}/metadata/c2pa.rs (100%) rename {capsule-media/src => capsule-core/src/media}/metadata/exif.rs (97%) rename {capsule-media/src => capsule-core/src/media}/metadata/geo.rs (100%) rename {capsule-media/src => capsule-core/src/media}/metadata/icc.rs (100%) rename {capsule-media/src => capsule-core/src/media}/metadata/mod.rs (100%) rename {capsule-media/src => capsule-core/src/media}/metadata/orientation.rs (100%) rename {capsule-media/src => capsule-core/src/media}/metadata/xmp.rs (100%) create mode 100644 capsule-core/src/media/mod.rs rename {capsule-media/src => capsule-core/src/media}/video/mod.rs (100%) rename {capsule-media/src => capsule-core/src/media}/video/presets.rs (99%) rename {capsule-media/src => capsule-core/src/media}/video/types.rs (100%) delete mode 100644 capsule-media/Cargo.toml delete mode 100644 capsule-media/src/lib.rs diff --git a/Cargo.lock b/Cargo.lock index 5ebb4d49..35e04e49 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1263,13 +1263,12 @@ dependencies = [ "capsule-api-model", "capsule-api-service", "capsule-core", - "capsule-media", + "capsule-i18n", "chrono", "eyre", "futures-util", "indexmap 2.14.0", "jsonwebtoken", - "libc", "nanoid", "ring", "salvo", @@ -1338,9 +1337,11 @@ dependencies = [ "aead", "aes-gcm", "argon2", + "base64 0.22.1", "chrono", "ciborium", "ed25519-dalek", + "file-format", "getrandom 0.3.4", "globset", "half", @@ -1348,10 +1349,13 @@ dependencies = [ "hkdf", "hmac", "indexmap 2.14.0", + "jpeg-encoder", "kamadak-exif", "log", + "memmap2", "ml-dsa", "ml-kem", + "num-rational", "rusqlite", "serde", "serde_bytes", @@ -1362,12 +1366,17 @@ dependencies = [ "tar", "tempfile", "thiserror 2.0.18", + "thumbhash", + "tokio", + "tracing", "tss-esapi", "tzf-rs", "uniffi 0.31.1", "uuid", "walkdir", "x25519-dalek", + "zune-core", + "zune-jpeg", ] [[package]] @@ -1394,26 +1403,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "capsule-media" -version = "0.1.0" -dependencies = [ - "base64 0.22.1", - "chrono", - "file-format", - "indexmap 2.14.0", - "jpeg-encoder", - "memmap2", - "num-rational", - "serde", - "thiserror 2.0.18", - "thumbhash", - "tokio", - "tracing", - "zune-core", - "zune-jpeg", -] - [[package]] name = "capsule-sdk" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index 23df4b99..1e116371 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -16,7 +16,6 @@ members = [ "capsule-core", "capsule-core-ffi", "capsule-i18n", - "capsule-media", "capsule-sdk", "xtask", ] @@ -38,7 +37,6 @@ default-members = [ "capsule-core", "capsule-core-ffi", "capsule-i18n", - "capsule-media", ] resolver = "3" diff --git a/README.md b/README.md index e42eb062..187dc014 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ Components: - [Capsule Desktop](capsule-desktop/README.md) (Planned): Windows/Linux desktop client - [Capsule Android](capsule-android/README.md) (WIP): Jetpack Compose App - [Capsule Swift](capsule-swift/README.md): SwiftUI client for iOS/macOS -- [Capsule Media](capsule-media/README.md) (Beta): C++ library for certain offloading +- Capsule Media: media decode/encode utilities, folded into `capsule-core` as the `media` module (behind the `media` feature) - [Capsule Docs](capsule-docs/README.md): Documentation website in Starlight (Astro) @@ -96,7 +96,7 @@ Considering all the technologies used, you may have to switch between IDEs to de - `capsule-core-kotlin`: Android Studio or IntelliJ IDEA with plugins - `capsule-desktop`: VS Code or similar - `capsule-docs`: VS Code or similar -- `capsule-media`: VS Code or similar +- `capsule-core` (media module): VS Code or similar - `capsule-swift`: Xcode - `capsule-web`: VS Code or similar diff --git a/SLICES.md b/SLICES.md index e1df8c68..265035f0 100644 --- a/SLICES.md +++ b/SLICES.md @@ -220,7 +220,8 @@ graph LR ### S-B1 — Thumbnail/LQIP generation - **Contract:** [Thumbnails](capsule-docs/src/content/docs/design/thumbnails.md). -- **Deliverable:** thumbnail/preview generation over `capsule-media` (which today decodes +- **Deliverable:** thumbnail/preview generation over `capsule_core::media` (the folded + former `capsule-media` crate, behind the non-default `media` feature; today it decodes JPEG only — format decoders grow as needed), chromahash LQIP + `dominant_color` into the sidecar `lqip` field, `DerivativeManifest`-signed outputs. - **Done when:** generation produces the committed formats with signed derivative diff --git a/capsule-api/upload/Cargo.toml b/capsule-api/upload/Cargo.toml index ad9bdede..9e247d32 100644 --- a/capsule-api/upload/Cargo.toml +++ b/capsule-api/upload/Cargo.toml @@ -14,9 +14,8 @@ capsule-api-entity = { path = "../entity" } capsule-api-environment = { path = "../environment", features = ["upload"] } capsule-api-model = { path = "../model" } capsule-api-service = { path = "../service" } -capsule-core = { path = "../../capsule-core" } +capsule-core = { path = "../../capsule-core", features = ["media"] } capsule-i18n = { path = "../../capsule-i18n" } -capsule-media = { path = "../../capsule-media" } bb8-redis = { workspace = true } eyre = { workspace = true } futures-util = { workspace = true } diff --git a/capsule-api/upload/src/service/processing.rs b/capsule-api/upload/src/service/processing.rs index f59dce3c..8836724c 100644 --- a/capsule-api/upload/src/service/processing.rs +++ b/capsule-api/upload/src/service/processing.rs @@ -1,6 +1,6 @@ use std::path::Path; -use capsule_media::fs::{ImageParseError, load_image}; +use capsule_core::media::fs::{ImageParseError, load_image}; use chrono::{DateTime, Utc}; /// Service for processing uploaded assets diff --git a/capsule-core/Cargo.toml b/capsule-core/Cargo.toml index f52535d4..d7ec67a4 100644 --- a/capsule-core/Cargo.toml +++ b/capsule-core/Cargo.toml @@ -22,6 +22,22 @@ required-features = ["ffi-bindgen"] # additionally pulls uniffi's CLI so the `uniffi-bindgen` bin can emit the bindings. ffi = ["dep:uniffi"] ffi-bindgen = ["ffi", "uniffi/cli"] +# `media` exposes the decode/encode + media-metadata utilities (the former standalone +# `capsule-media` crate) as `capsule_core::media`. Non-default so the default library +# build (crypto data plane, CLI, FFI, future WASM) never compiles the media stack; +# `capsule-api-upload` opts in. +media = [ + "dep:base64", + "dep:file-format", + "dep:jpeg-encoder", + "dep:memmap2", + "dep:num-rational", + "dep:thumbhash", + "dep:tokio", + "dep:tracing", + "dep:zune-core", + "dep:zune-jpeg", +] # `tpm` builds the desktop (Linux/Windows) TPM 2.0 reference `HardwareSigner` (see # `crypto::keys::tpm`). Off by default and never built in CI: tss-esapi links the system # `libtss2` and exercising it needs a real TPM or a software TPM (swtpm). Compile on a @@ -63,6 +79,18 @@ getrandom = "0.3" # OS CSPRNG for keys, salts, nonce prefixes tar = "0.4" # deterministic uncompressed POSIX tar backup container sharks = "0.5" # Shamir secret sharing (opt-in 2-of-3 recovery) +# ── Media (optional; `media` feature) — decode/encode + media metadata ────── +base64 = { workspace = true, optional = true } +file-format = { workspace = true, optional = true } +jpeg-encoder = { version = "0.6", optional = true } +memmap2 = { version = "0.9.10", optional = true } +num-rational = { workspace = true, optional = true } +thumbhash = { version = "0.1.0", optional = true } +tokio = { workspace = true, optional = true } +tracing = { workspace = true, optional = true } +zune-core = { version = "0.5.1", optional = true } +zune-jpeg = { version = "0.5.15", optional = true } + # ── FFI (optional; `ffi` feature) ────────────────────────────────────────── # uniffi proc-macro bindings for Kotlin/Swift. Optional so the default library # build (CLI, server, tests) never compiles it. @@ -74,5 +102,17 @@ uniffi = { version = "0.31", optional = true } # build pulls in the TPM stack. See `crypto::keys::tpm`. tss-esapi = { version = "7", optional = true } +[[example]] +name = "extract_metadata" +required-features = ["media"] + +[[example]] +name = "generate_derivative" +required-features = ["media"] + +[[example]] +name = "generate_lqip" +required-features = ["media"] + [dev-dependencies] tempfile = "3" diff --git a/capsule-media/examples/extract_metadata.rs b/capsule-core/examples/extract_metadata.rs similarity index 80% rename from capsule-media/examples/extract_metadata.rs rename to capsule-core/examples/extract_metadata.rs index 7f443b7f..9a8df794 100644 --- a/capsule-media/examples/extract_metadata.rs +++ b/capsule-core/examples/extract_metadata.rs @@ -1,6 +1,6 @@ //! Extract metadata from an image //! -//! This example demonstrates how to use the `capsule_media` crate to read an image file +//! This example demonstrates how to use the `capsule_core::media` module (enable the `media` feature) to read an image file //! and extract its metadata, including EXIF, XMP, IPTC, and technical details. //! //! Usage: @@ -10,7 +10,7 @@ use std::path::PathBuf; -use capsule_media::fs::MediaFile; +use capsule_core::media::fs::MediaFile; #[tokio::main] pub async fn main() { @@ -23,7 +23,7 @@ pub async fn main() { let input_path = PathBuf::from(&args[1]); // Read the media file. This automatically detects the format and decodes it. - let media = capsule_media::fs::read(&input_path) + let media = capsule_core::media::fs::read(&input_path) .await .expect("Failed to read media file"); diff --git a/capsule-media/examples/generate_derivative.rs b/capsule-core/examples/generate_derivative.rs similarity index 78% rename from capsule-media/examples/generate_derivative.rs rename to capsule-core/examples/generate_derivative.rs index a955d0fd..1653f68f 100644 --- a/capsule-media/examples/generate_derivative.rs +++ b/capsule-core/examples/generate_derivative.rs @@ -1,6 +1,6 @@ //! Example of generating a JpegXL derivative from an image file //! -//! This example demonstrates how to use the `capsule_media` crate to read an image file, +//! This example demonstrates how to use the `capsule_core::media` module (enable the `media` feature) to read an image file, //! detect its format, and generate a JpegXL derivative. //! //! Usage: @@ -10,9 +10,9 @@ use std::path::PathBuf; -use capsule_media::fs::MediaFile; -use capsule_media::image::formats::jxl::JxlImage; -use capsule_media::image::{ConvertImage, ImageEncode}; +use capsule_core::media::fs::MediaFile; +use capsule_core::media::image::formats::jxl::JxlImage; +use capsule_core::media::image::{ConvertImage, ImageEncode}; #[tokio::main] pub async fn main() { @@ -29,7 +29,7 @@ pub async fn main() { input_path.with_extension("jxl") }; - let media = capsule_media::fs::read(&input_path) + let media = capsule_core::media::fs::read(&input_path) .await .expect("Failed to create reader"); let MediaFile::Image(file) = media else { diff --git a/capsule-media/examples/generate_lqip.rs b/capsule-core/examples/generate_lqip.rs similarity index 84% rename from capsule-media/examples/generate_lqip.rs rename to capsule-core/examples/generate_lqip.rs index ccb035b2..e000a3cf 100644 --- a/capsule-media/examples/generate_lqip.rs +++ b/capsule-core/examples/generate_lqip.rs @@ -1,8 +1,8 @@ use std::path::PathBuf; -use capsule_media::image::formats::jpeg::JpegImage; -use capsule_media::image::lqip::LQIP; -use capsule_media::image::{Image, ImageReader}; +use capsule_core::media::image::formats::jpeg::JpegImage; +use capsule_core::media::image::lqip::LQIP; +use capsule_core::media::image::{Image, ImageReader}; #[tokio::main] pub async fn main() { diff --git a/capsule-core/src/lib.rs b/capsule-core/src/lib.rs index 3cd6babf..eb910b6f 100644 --- a/capsule-core/src/lib.rs +++ b/capsule-core/src/lib.rs @@ -9,6 +9,8 @@ pub mod exif; pub mod import; pub mod library; pub mod lifecycle; +#[cfg(feature = "media")] +pub mod media; pub mod metadata; pub mod models; pub mod sharing; diff --git a/capsule-media/src/core/mod.rs b/capsule-core/src/media/core/mod.rs similarity index 100% rename from capsule-media/src/core/mod.rs rename to capsule-core/src/media/core/mod.rs diff --git a/capsule-media/src/core/transcode.rs b/capsule-core/src/media/core/transcode.rs similarity index 82% rename from capsule-media/src/core/transcode.rs rename to capsule-core/src/media/core/transcode.rs index 15442464..966a47b1 100644 --- a/capsule-media/src/core/transcode.rs +++ b/capsule-core/src/media/core/transcode.rs @@ -2,8 +2,8 @@ use std::path::PathBuf; use serde::{Deserialize, Serialize}; -use crate::image::types::{ImageFormat, ImageOutputSettings}; -use crate::video::types::{VideoFormat, VideoOutputSettings}; +use crate::media::image::types::{ImageFormat, ImageOutputSettings}; +use crate::media::video::types::{VideoFormat, VideoOutputSettings}; #[derive(Serialize, Deserialize, Debug, Clone)] pub enum TranscodeTask { diff --git a/capsule-media/src/core/types/mod.rs b/capsule-core/src/media/core/types/mod.rs similarity index 77% rename from capsule-media/src/core/types/mod.rs rename to capsule-core/src/media/core/types/mod.rs index 093be995..2e669acb 100644 --- a/capsule-media/src/core/types/mod.rs +++ b/capsule-core/src/media/core/types/mod.rs @@ -1,9 +1,9 @@ use serde::{Deserialize, Serialize}; // Re-export image types for backward compatibility -pub use crate::image::types::*; +pub use crate::media::image::types::*; // Re-export video types for backward compatibility -pub use crate::video::types::*; +pub use crate::media::video::types::*; #[derive(Serialize, Deserialize, Debug, Clone)] pub enum MediaType { diff --git a/capsule-media/src/fs/ext.rs b/capsule-core/src/media/fs/ext.rs similarity index 91% rename from capsule-media/src/fs/ext.rs rename to capsule-core/src/media/fs/ext.rs index d4f05ce0..af8147df 100644 --- a/capsule-media/src/fs/ext.rs +++ b/capsule-core/src/media/fs/ext.rs @@ -2,9 +2,9 @@ use std::path::Path; use file_format::FileFormat; -use crate::core::types::MediaType; -use crate::image::types::ImageFormat; -use crate::video::types::VideoFormat; +use crate::media::core::types::MediaType; +use crate::media::image::types::ImageFormat; +use crate::media::video::types::VideoFormat; macro_rules! img { ($variant:ident) => { diff --git a/capsule-media/src/fs/mod.rs b/capsule-core/src/media/fs/mod.rs similarity index 84% rename from capsule-media/src/fs/mod.rs rename to capsule-core/src/media/fs/mod.rs index 5ff8c86e..2d708b80 100644 --- a/capsule-media/src/fs/mod.rs +++ b/capsule-core/src/media/fs/mod.rs @@ -3,21 +3,21 @@ use std::path::Path; use thiserror::Error; use tokio::fs; -use crate::core::types::MediaType; -use crate::image::formats::avif::AvifImage; -use crate::image::formats::bmp::BmpImage; -use crate::image::formats::gif::GifImage; -use crate::image::formats::heif::HeifImage; -use crate::image::formats::jpeg::JpegImage; -use crate::image::formats::jxl::JxlImage; -use crate::image::formats::png::PngImage; -use crate::image::formats::raw::RawImage; -use crate::image::formats::tiff::TiffImage; -use crate::image::formats::webp::WebpImage as WebPImage; -use crate::image::types::ImageFormat; -use crate::image::{ImageFile, ImageReader, ImageWithMetadata}; -use crate::video::VideoFile; -use crate::video::types::VideoFormat; +use crate::media::core::types::MediaType; +use crate::media::image::formats::avif::AvifImage; +use crate::media::image::formats::bmp::BmpImage; +use crate::media::image::formats::gif::GifImage; +use crate::media::image::formats::heif::HeifImage; +use crate::media::image::formats::jpeg::JpegImage; +use crate::media::image::formats::jxl::JxlImage; +use crate::media::image::formats::png::PngImage; +use crate::media::image::formats::raw::RawImage; +use crate::media::image::formats::tiff::TiffImage; +use crate::media::image::formats::webp::WebpImage as WebPImage; +use crate::media::image::types::ImageFormat; +use crate::media::image::{ImageFile, ImageReader, ImageWithMetadata}; +use crate::media::video::VideoFile; +use crate::media::video::types::VideoFormat; pub mod ext; @@ -83,7 +83,7 @@ pub enum ReadMediaError { #[error("Join error: {0}")] JoinError(#[from] tokio::task::JoinError), #[error("Image error: {0}")] - Image(#[from] crate::image::ImageError), + Image(#[from] crate::media::image::ImageError), } #[derive(Debug)] @@ -128,7 +128,7 @@ pub enum ImageParseError { #[error("Read image error: {0}")] ReadImageError(#[from] ReadImageError), #[error("Image error: {0}")] - ImageError(#[from] crate::image::ImageError), + ImageError(#[from] crate::media::image::ImageError), #[error("Image data error: {0}")] DataError(String), } diff --git a/capsule-media/src/image/buffer.rs b/capsule-core/src/media/image/buffer.rs similarity index 99% rename from capsule-media/src/image/buffer.rs rename to capsule-core/src/media/image/buffer.rs index 5ed1b5e3..a94b8211 100644 --- a/capsule-media/src/image/buffer.rs +++ b/capsule-core/src/media/image/buffer.rs @@ -1,4 +1,4 @@ -use crate::metadata::ColorSpace; +use crate::media::metadata::ColorSpace; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum PixelFormat { diff --git a/capsule-media/src/image/formats/avif.rs b/capsule-core/src/media/image/formats/avif.rs similarity index 71% rename from capsule-media/src/image/formats/avif.rs rename to capsule-core/src/media/image/formats/avif.rs index d5a26b60..f9c05745 100644 --- a/capsule-media/src/image/formats/avif.rs +++ b/capsule-core/src/media/image/formats/avif.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct AvifImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for AvifImage { } impl Image for AvifImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Avif + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Avif } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/bmp.rs b/capsule-core/src/media/image/formats/bmp.rs similarity index 71% rename from capsule-media/src/image/formats/bmp.rs rename to capsule-core/src/media/image/formats/bmp.rs index ad9c9a37..b99b27f1 100644 --- a/capsule-media/src/image/formats/bmp.rs +++ b/capsule-core/src/media/image/formats/bmp.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct BmpImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for BmpImage { } impl Image for BmpImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Bmp + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Bmp } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/dng.rs b/capsule-core/src/media/image/formats/dng.rs similarity index 70% rename from capsule-media/src/image/formats/dng.rs rename to capsule-core/src/media/image/formats/dng.rs index 9dcaccb8..80ac7f8f 100644 --- a/capsule-media/src/image/formats/dng.rs +++ b/capsule-core/src/media/image/formats/dng.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct DngImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for DngImage { } impl Image for DngImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Raw(crate::image::types::RawImageFormat::Dng) + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Raw(crate::media::image::types::RawImageFormat::Dng) } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/gif.rs b/capsule-core/src/media/image/formats/gif.rs similarity index 71% rename from capsule-media/src/image/formats/gif.rs rename to capsule-core/src/media/image/formats/gif.rs index bb5bc76b..a42cd657 100644 --- a/capsule-media/src/image/formats/gif.rs +++ b/capsule-core/src/media/image/formats/gif.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct GifImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for GifImage { } impl Image for GifImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Gif + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Gif } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/heif.rs b/capsule-core/src/media/image/formats/heif.rs similarity index 71% rename from capsule-media/src/image/formats/heif.rs rename to capsule-core/src/media/image/formats/heif.rs index b2267789..b22393ae 100644 --- a/capsule-media/src/image/formats/heif.rs +++ b/capsule-core/src/media/image/formats/heif.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct HeifImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for HeifImage { } impl Image for HeifImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Heic + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Heic } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/jpeg.rs b/capsule-core/src/media/image/formats/jpeg.rs similarity index 84% rename from capsule-media/src/image/formats/jpeg.rs rename to capsule-core/src/media/image/formats/jpeg.rs index 3a9268e5..50c090f8 100644 --- a/capsule-media/src/image/formats/jpeg.rs +++ b/capsule-core/src/media/image/formats/jpeg.rs @@ -5,19 +5,19 @@ use zune_core::colorspace::ColorSpace as ZuneColorSpace; use zune_core::options::DecoderOptions; use zune_jpeg::JpegDecoder; -use crate::image::buffer::{ComponentType, ImageBuffer, PixelFormat}; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::{ComponentType, ImageBuffer, PixelFormat}; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct JpegImage { @@ -85,8 +85,8 @@ impl ImageMetadataExtractor for JpegImage { } impl Image for JpegImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Jpeg + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Jpeg } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/jxl.rs b/capsule-core/src/media/image/formats/jxl.rs similarity index 71% rename from capsule-media/src/image/formats/jxl.rs rename to capsule-core/src/media/image/formats/jxl.rs index 0ec9400d..f734b7d8 100644 --- a/capsule-media/src/image/formats/jxl.rs +++ b/capsule-core/src/media/image/formats/jxl.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct JxlImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for JxlImage { } impl Image for JxlImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Jxl + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Jxl } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/mod.rs b/capsule-core/src/media/image/formats/mod.rs similarity index 100% rename from capsule-media/src/image/formats/mod.rs rename to capsule-core/src/media/image/formats/mod.rs diff --git a/capsule-media/src/image/formats/png.rs b/capsule-core/src/media/image/formats/png.rs similarity index 71% rename from capsule-media/src/image/formats/png.rs rename to capsule-core/src/media/image/formats/png.rs index 7ff99a3e..46cad4da 100644 --- a/capsule-media/src/image/formats/png.rs +++ b/capsule-core/src/media/image/formats/png.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct PngImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for PngImage { } impl Image for PngImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Png + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Png } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/raw.rs b/capsule-core/src/media/image/formats/raw.rs similarity index 77% rename from capsule-media/src/image/formats/raw.rs rename to capsule-core/src/media/image/formats/raw.rs index 2dc01395..ba47f1bc 100644 --- a/capsule-media/src/image/formats/raw.rs +++ b/capsule-core/src/media/image/formats/raw.rs @@ -1,20 +1,20 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::{ComponentType, ImageBuffer, PixelFormat}; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadata, ImageMetadataExtractor}; -use crate::image::types::{ImageFormat, RawImageFormat}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::{ComponentType, ImageBuffer, PixelFormat}; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadata, ImageMetadataExtractor}; +use crate::media::image::types::{ImageFormat, RawImageFormat}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct RawImage { diff --git a/capsule-media/src/image/formats/tiff.rs b/capsule-core/src/media/image/formats/tiff.rs similarity index 71% rename from capsule-media/src/image/formats/tiff.rs rename to capsule-core/src/media/image/formats/tiff.rs index fc7cafeb..364fb0ae 100644 --- a/capsule-media/src/image/formats/tiff.rs +++ b/capsule-core/src/media/image/formats/tiff.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct TiffImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for TiffImage { } impl Image for TiffImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::Tiff + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::Tiff } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/formats/webp.rs b/capsule-core/src/media/image/formats/webp.rs similarity index 71% rename from capsule-media/src/image/formats/webp.rs rename to capsule-core/src/media/image/formats/webp.rs index d3d2db9d..c853da9c 100644 --- a/capsule-media/src/image/formats/webp.rs +++ b/capsule-core/src/media/image/formats/webp.rs @@ -1,19 +1,19 @@ use std::io::Write; use std::path::Path; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::image::metadata::{ContentMetadata, ImageMetadataExtractor}; -use crate::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::image::metadata::{ContentMetadata, ImageMetadataExtractor}; +use crate::media::image::{Image, ImageDecode, ImageEncode, ImageError, ImageMetadata}; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; #[derive(Debug, Clone)] pub struct WebpImage {} @@ -73,8 +73,8 @@ impl ImageMetadataExtractor for WebpImage { } impl Image for WebpImage { - fn get_format(&self) -> crate::core::types::ImageFormat { - crate::core::types::ImageFormat::WebP + fn get_format(&self) -> crate::media::core::types::ImageFormat { + crate::media::core::types::ImageFormat::WebP } fn get_buffer(&self) -> ImageBuffer { diff --git a/capsule-media/src/image/lqip.rs b/capsule-core/src/media/image/lqip.rs similarity index 97% rename from capsule-media/src/image/lqip.rs rename to capsule-core/src/media/image/lqip.rs index cc696189..861feaf1 100644 --- a/capsule-media/src/image/lqip.rs +++ b/capsule-core/src/media/image/lqip.rs @@ -1,8 +1,8 @@ use thiserror::Error; -use crate::image::buffer::{ComponentType, ImageBuffer, ImageBufferError, PixelFormat}; -use crate::image::resize_to_max_dimension; -use crate::metadata::ColorSpace; +use crate::media::image::buffer::{ComponentType, ImageBuffer, ImageBufferError, PixelFormat}; +use crate::media::image::resize_to_max_dimension; +use crate::media::metadata::ColorSpace; /// LQIP (thumbhash) struct pub struct LQIP(Vec); diff --git a/capsule-media/src/image/metadata/exposure.rs b/capsule-core/src/media/image/metadata/exposure.rs similarity index 100% rename from capsule-media/src/image/metadata/exposure.rs rename to capsule-core/src/media/image/metadata/exposure.rs diff --git a/capsule-media/src/image/metadata/iptc.rs b/capsule-core/src/media/image/metadata/iptc.rs similarity index 100% rename from capsule-media/src/image/metadata/iptc.rs rename to capsule-core/src/media/image/metadata/iptc.rs diff --git a/capsule-media/src/image/metadata/mod.rs b/capsule-core/src/media/image/metadata/mod.rs similarity index 86% rename from capsule-media/src/image/metadata/mod.rs rename to capsule-core/src/media/image/metadata/mod.rs index 4cdc8781..f1fbfc79 100644 --- a/capsule-media/src/image/metadata/mod.rs +++ b/capsule-core/src/media/image/metadata/mod.rs @@ -1,18 +1,18 @@ use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; -use crate::core::types::ImageFormat; -use crate::image::Image; -use crate::image::metadata::exposure::CaptureSettings; -use crate::image::metadata::iptc::IptcData; -use crate::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; -use crate::image::metadata::raw::RawSensorInfo; -use crate::metadata::c2pa::C2PAManifest; -use crate::metadata::exif::ExifData; -use crate::metadata::geo::GpsLocation; -use crate::metadata::icc::IccProfile; -use crate::metadata::xmp::XmpData; -use crate::metadata::{ColorSpace, DeviceMetadata}; +use crate::media::core::types::ImageFormat; +use crate::media::image::Image; +use crate::media::image::metadata::exposure::CaptureSettings; +use crate::media::image::metadata::iptc::IptcData; +use crate::media::image::metadata::motion::{AuxiliaryImage, MotionPhotoInfo}; +use crate::media::image::metadata::raw::RawSensorInfo; +use crate::media::metadata::c2pa::C2PAManifest; +use crate::media::metadata::exif::ExifData; +use crate::media::metadata::geo::GpsLocation; +use crate::media::metadata::icc::IccProfile; +use crate::media::metadata::xmp::XmpData; +use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub mod exposure; pub mod iptc; diff --git a/capsule-media/src/image/metadata/motion.rs b/capsule-core/src/media/image/metadata/motion.rs similarity index 100% rename from capsule-media/src/image/metadata/motion.rs rename to capsule-core/src/media/image/metadata/motion.rs diff --git a/capsule-media/src/image/metadata/raw.rs b/capsule-core/src/media/image/metadata/raw.rs similarity index 100% rename from capsule-media/src/image/metadata/raw.rs rename to capsule-core/src/media/image/metadata/raw.rs diff --git a/capsule-media/src/image/mod.rs b/capsule-core/src/media/image/mod.rs similarity index 95% rename from capsule-media/src/image/mod.rs rename to capsule-core/src/media/image/mod.rs index 6629b218..88d0e9cb 100644 --- a/capsule-media/src/image/mod.rs +++ b/capsule-core/src/media/image/mod.rs @@ -3,9 +3,9 @@ use std::path::{Path, PathBuf}; use thiserror::Error; -use crate::image::buffer::ImageBuffer; -use crate::image::metadata::{ImageMetadata, ImageMetadataProvider}; -use crate::image::types::ImageFormat; +use crate::media::image::buffer::ImageBuffer; +use crate::media::image::metadata::{ImageMetadata, ImageMetadataProvider}; +use crate::media::image::types::ImageFormat; pub mod buffer; pub mod formats; @@ -304,10 +304,10 @@ mod tests { } #[test] - #[ignore = "PngImage not yet implemented"] + #[ignore = "S-B1: PngImage decoder not yet implemented — see SLICES.md"] fn test_image_conversion() { - use crate::image::formats::jpeg::JpegImage; - use crate::image::formats::png::PngImage; + use crate::media::image::formats::jpeg::JpegImage; + use crate::media::image::formats::png::PngImage; // Create a sample JPEG image buffer let width = 100; @@ -320,7 +320,7 @@ mod tests { height, buffer::PixelFormat::Rgb, buffer::ComponentType::U8, - crate::metadata::ColorSpace::Srgb, + crate::media::metadata::ColorSpace::Srgb, ) .unwrap(); @@ -330,7 +330,7 @@ mod tests { width: width as u32, height: height as u32, bit_depth: 8, - color_space: crate::metadata::ColorSpace::Srgb, + color_space: crate::media::metadata::ColorSpace::Srgb, ..Default::default() }; @@ -353,7 +353,7 @@ mod tests { height, buffer::PixelFormat::Rgb, buffer::ComponentType::U8, - crate::metadata::ColorSpace::Srgb, + crate::media::metadata::ColorSpace::Srgb, ) .unwrap(), ImageMetadata { @@ -362,7 +362,7 @@ mod tests { width: width as u32, height: height as u32, bit_depth: 8, - color_space: crate::metadata::ColorSpace::Srgb, + color_space: crate::media::metadata::ColorSpace::Srgb, ..Default::default() }, ) diff --git a/capsule-media/src/image/presets.rs b/capsule-core/src/media/image/presets.rs similarity index 99% rename from capsule-media/src/image/presets.rs rename to capsule-core/src/media/image/presets.rs index fdd88478..e27e494b 100644 --- a/capsule-media/src/image/presets.rs +++ b/capsule-core/src/media/image/presets.rs @@ -14,14 +14,14 @@ //! # Example //! //! ```rust -//! use capsule_media::image::presets::ImagePresets; -//! use capsule_media::image::types::ImageOutputSettings; +//! use capsule_core::media::image::presets::ImagePresets; +//! use capsule_core::media::image::types::ImageOutputSettings; //! //! // Get a preset for web streaming in JXL format //! let settings = ImagePresets::web_streaming_jxl(); //! ``` -use crate::image::types::{ +use crate::media::image::types::{ AvifSettings, ChromaSubsampling, ImageOutputSettings, ImageResolution, JpegSettings, JxlSettings, PngSettings, StandardImageSize, TiffCompression, TiffSettings, WebPSettings, }; diff --git a/capsule-media/src/image/types.rs b/capsule-core/src/media/image/types.rs similarity index 100% rename from capsule-media/src/image/types.rs rename to capsule-core/src/media/image/types.rs diff --git a/capsule-media/src/metadata/c2pa.rs b/capsule-core/src/media/metadata/c2pa.rs similarity index 100% rename from capsule-media/src/metadata/c2pa.rs rename to capsule-core/src/media/metadata/c2pa.rs diff --git a/capsule-media/src/metadata/exif.rs b/capsule-core/src/media/metadata/exif.rs similarity index 97% rename from capsule-media/src/metadata/exif.rs rename to capsule-core/src/media/metadata/exif.rs index 5b0d8f58..6e3089ed 100644 --- a/capsule-media/src/metadata/exif.rs +++ b/capsule-core/src/media/metadata/exif.rs @@ -1,7 +1,7 @@ use num_rational::Ratio; use serde::{Deserialize, Serialize}; -use crate::metadata::orientation::Orientation; +use crate::media::metadata::orientation::Orientation; #[derive(Serialize, Deserialize, Debug, Clone)] pub struct ExifData { diff --git a/capsule-media/src/metadata/geo.rs b/capsule-core/src/media/metadata/geo.rs similarity index 100% rename from capsule-media/src/metadata/geo.rs rename to capsule-core/src/media/metadata/geo.rs diff --git a/capsule-media/src/metadata/icc.rs b/capsule-core/src/media/metadata/icc.rs similarity index 100% rename from capsule-media/src/metadata/icc.rs rename to capsule-core/src/media/metadata/icc.rs diff --git a/capsule-media/src/metadata/mod.rs b/capsule-core/src/media/metadata/mod.rs similarity index 100% rename from capsule-media/src/metadata/mod.rs rename to capsule-core/src/media/metadata/mod.rs diff --git a/capsule-media/src/metadata/orientation.rs b/capsule-core/src/media/metadata/orientation.rs similarity index 100% rename from capsule-media/src/metadata/orientation.rs rename to capsule-core/src/media/metadata/orientation.rs diff --git a/capsule-media/src/metadata/xmp.rs b/capsule-core/src/media/metadata/xmp.rs similarity index 100% rename from capsule-media/src/metadata/xmp.rs rename to capsule-core/src/media/metadata/xmp.rs diff --git a/capsule-core/src/media/mod.rs b/capsule-core/src/media/mod.rs new file mode 100644 index 00000000..44c8efd5 --- /dev/null +++ b/capsule-core/src/media/mod.rs @@ -0,0 +1,10 @@ +//! Media decode/encode and metadata utilities (formerly the standalone +//! `capsule-media` crate), behind the non-default `media` feature. Owner doc: +//! Thumbnails and Previews; thumbnail/LQIP generation over these utilities is +//! slice `S-B1` in the repo-root `SLICES.md`. + +pub mod core; +pub mod fs; +pub mod image; +pub mod metadata; +pub mod video; diff --git a/capsule-media/src/video/mod.rs b/capsule-core/src/media/video/mod.rs similarity index 100% rename from capsule-media/src/video/mod.rs rename to capsule-core/src/media/video/mod.rs diff --git a/capsule-media/src/video/presets.rs b/capsule-core/src/media/video/presets.rs similarity index 99% rename from capsule-media/src/video/presets.rs rename to capsule-core/src/media/video/presets.rs index 6e579242..26192128 100644 --- a/capsule-media/src/video/presets.rs +++ b/capsule-core/src/media/video/presets.rs @@ -22,14 +22,14 @@ //! # Example //! //! ```rust -//! use capsule_media::video::presets::VideoPresets; -//! use capsule_media::video::types::VideoOutputSettings; +//! use capsule_core::media::video::presets::VideoPresets; +//! use capsule_core::media::video::types::VideoOutputSettings; //! //! // Get a preset for 1080p streaming //! let settings = VideoPresets::streaming_1080p_h264(); //! ``` -use crate::video::types::{ +use crate::media::video::types::{ AudioSettings, Mp4Codec, Mp4Settings, RateControl, StandardResolution, VideoOutputSettings, VideoResolution, VpxDeadline, WebmCodec, WebmSettings, X264Preset, }; diff --git a/capsule-media/src/video/types.rs b/capsule-core/src/media/video/types.rs similarity index 100% rename from capsule-media/src/video/types.rs rename to capsule-core/src/media/video/types.rs diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 04ca5391..f7d240f5 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -28,7 +28,6 @@ The mapping reflects the *design intent*. Modules not yet implemented are annota | `capsule-api-environment` | Configuration, env vars, feature flags | | `capsule-api-testing` | Shared test utilities (testcontainer setup, schema fixtures) | | `capsule-cli` | Command-line client | -| `capsule-media` | Standalone media utility crate | | `capsule-i18n` | Runtime localization (locale negotiation + ICU message formatting) for the server and CLI | | `capsule-web` | Browser/WASM web-upload client: guest drop sealing + upload (planned) | | `capsule-vision` | Python research package for on-device ML experimentation (not a cargo crate) | @@ -99,12 +98,12 @@ Hardware-key adapters do **not** live in `capsule-sdk`: the `Signer`/`HardwareSi | `capsule-api-environment` | (configuration; no design owner) | Unit | | `capsule-api-testing` | (test utilities; no design owner) | n/a | -### `capsule-cli`, `capsule-media`, `capsule-i18n`, `capsule-web` +### `capsule-cli`, `capsule-i18n`, `capsule-web` | Crate | Owning design doc | Validation tier | | --------------- | ----------------------------------------------------------------- | -------------------- | | `capsule-cli` | [Clients](/design/clients/) (treats CLI as a client) | Smoke | -| `capsule-media` | [Thumbnails and Previews](/design/thumbnails/) (decode/encode utilities; only JPEG decode is implemented today) | Unit | +| `capsule-core::media` (behind the non-default `media` feature) | [Thumbnails and Previews](/design/thumbnails/) (decode/encode utilities; only JPEG decode is implemented today) | Unit | | `capsule-i18n` | [Internationalization](/design/i18n/) | Unit + Smoke | | `capsule-web` | [Web Upload](/design/web-upload/), [Clients](/design/clients/) | Smoke (browser/WASM) | @@ -142,7 +141,7 @@ Navigation from a design doc back to where the code lives. | [Peering](/design/peering/) | `capsule-sdk::peering` (planned) + `capsule-core::backup` (artifact format) | | [Organization](/design/organization/) | `capsule-core::domain::stack_type`, `capsule-api-service::{album,stack}` | | [AI/ML Integrations](/design/ai/) | `capsule-core::ml` (planned), `capsule-vision` (Python research), model registry + per-platform inference runners | -| [Thumbnails](/design/thumbnails/) | Client-side gen in `capsule-sdk` (planned) over `capsule-media` decode/encode utilities + serving in `capsule-api-media` | +| [Thumbnails](/design/thumbnails/) | Client-side gen in `capsule-sdk` (planned) over `capsule_core::media` decode/encode utilities + serving in `capsule-api-media` | | [Share Links](/design/share-links/) | `capsule-core::sharing` (planned), `capsule-api-media::shares` (planned) | | [Web Upload](/design/web-upload/) | `capsule-core::drop` (planned), `capsule-api-media::drops` (planned), `capsule-web` (planned); reuses `capsule-api-upload` for drop chunks | | [Moderation](/design/moderation/) | `capsule-api::moderation` (planned) | diff --git a/capsule-media/Cargo.toml b/capsule-media/Cargo.toml deleted file mode 100644 index cb54fbf2..00000000 --- a/capsule-media/Cargo.toml +++ /dev/null @@ -1,24 +0,0 @@ -[package] -name = "capsule-media" -version.workspace = true -edition = "2024" - -[dependencies] -base64 = { workspace = true } -chrono = { workspace = true } -file-format = { workspace = true, optional = true } -indexmap = { workspace = true } -serde = { workspace = true } -thiserror = { workspace = true } -thumbhash = "0.1.0" -tokio = { workspace = true } -zune-core = "0.5.1" -zune-jpeg = "0.5.15" -jpeg-encoder = "0.6" -tracing = { workspace = true } -memmap2 = "0.9.10" -num-rational = { workspace = true } - -[features] -default = ["fs"] -fs = ["file-format"] diff --git a/capsule-media/src/lib.rs b/capsule-media/src/lib.rs deleted file mode 100644 index 9ad33d42..00000000 --- a/capsule-media/src/lib.rs +++ /dev/null @@ -1,6 +0,0 @@ -pub mod core; -#[cfg(feature = "fs")] -pub mod fs; -pub mod image; -pub mod metadata; -pub mod video; From 2996a1343683516c11e6fef5cee341d44037c121 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:45:40 -0400 Subject: [PATCH 21/58] refactor(sdk): drop progenitor, stage the spargen transition MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit progenitor consumes OpenAPI 3.0 only, so the SDK's client generation ran through a lossy 3.1->3.0 schema down-conversion script against a live server — a standing source of drift, and the reason the crate could not compile standalone (the macro's openapi.json input was gitignored and absent, which in turn forced '--exclude capsule-sdk' on every Rust gate). We do not downgrade schemas: the typed REST client will be generated from the OpenAPI 3.1 schema by spargen, our in-house generator (in development; slice S-D8). - progenitor, the generate_api! macro, the downconvert script, and the stale auth_flow example are gone; AuthenticatedClient and UploadClient are commented out (not deleted) as the shape S-D8 revives, per the staged-transition decision. - capsule-sdk now compiles standalone and re-enters every Rust gate (clippy/build/nextest/coverage — exclusions removed from mise.toml). Running its tests for the first time surfaced a broken test the exclusion had been hiding: test_adaptive_scaling_up asserted scaling after 3 chunks, but the warm-up rule (now normative in the upload doc) requires >=5 chunks or >=8 MiB — the test now exercises both sides of the warm-up boundary. Three never-linted format-string warnings fixed. - api-surfaces.md and module-map.md now name spargen (with the no-schema-downgrade rationale) instead of progenitor. --- Cargo.lock | 203 ------------------ .../src/content/docs/design/api-surfaces.md | 2 +- .../src/content/docs/design/module-map.md | 2 +- capsule-sdk/.gitignore | 1 - capsule-sdk/Cargo.toml | 1 - capsule-sdk/README.md | 9 +- capsule-sdk/examples/auth_flow.rs | 38 ---- capsule-sdk/generate_openapi.sh | 21 -- capsule-sdk/src/lib.rs | 136 +++++++----- capsule-sdk/src/upload.rs | 141 ++++++------ mise.toml | 10 +- 11 files changed, 164 insertions(+), 400 deletions(-) delete mode 100644 capsule-sdk/.gitignore delete mode 100644 capsule-sdk/examples/auth_flow.rs delete mode 100755 capsule-sdk/generate_openapi.sh diff --git a/Cargo.lock b/Cargo.lock index 35e04e49..2d007a76 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1409,7 +1409,6 @@ version = "0.1.0" dependencies = [ "chrono", "log", - "progenitor", "reqwest", "serde", "thiserror 2.0.18", @@ -4034,17 +4033,6 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" -[[package]] -name = "openapiv3" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c8d427828b22ae1fff2833a03d8486c2c881367f1c336349f307f321e7f4d05" -dependencies = [ - "indexmap 2.14.0", - "serde", - "serde_json", -] - [[package]] name = "openssl" version = "0.10.80" @@ -4593,72 +4581,6 @@ dependencies = [ "yansi", ] -[[package]] -name = "progenitor" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2326f73d5326257514712436680ef8da4543ee47c0e9e0d501545c8909ee12e4" -dependencies = [ - "progenitor-client", - "progenitor-impl", - "progenitor-macro", -] - -[[package]] -name = "progenitor-client" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71a0beb939758f229cbae70a4889c7c76a4ac0e90f0b1e7ae9b4636a927d1018" -dependencies = [ - "bytes", - "futures-core", - "percent-encoding", - "reqwest", - "serde", - "serde_json", - "serde_urlencoded", -] - -[[package]] -name = "progenitor-impl" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90f6d9109b04e005bbdec84cacec7e81cc15533f2b5dc505f0defc212d270c15" -dependencies = [ - "heck 0.5.0", - "http", - "indexmap 2.14.0", - "openapiv3", - "proc-macro2", - "quote", - "regex", - "schemars 0.8.22", - "serde", - "serde_json", - "syn 2.0.117", - "thiserror 2.0.18", - "typify", - "unicode-ident", -] - -[[package]] -name = "progenitor-macro" -version = "0.11.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "46596c574831739c661f22923fe587399c61f5e3e79b73cc9a93644c72248d84" -dependencies = [ - "openapiv3", - "proc-macro2", - "progenitor-impl", - "quote", - "schemars 0.8.22", - "serde", - "serde_json", - "serde_tokenstream", - "serde_yaml", - "syn 2.0.117", -] - [[package]] name = "prost" version = "0.13.5" @@ -5057,16 +4979,6 @@ version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" -[[package]] -name = "regress" -version = "0.10.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2057b2325e68a893284d1538021ab90279adac1139957ca2a74426c6f118fb48" -dependencies = [ - "hashbrown 0.16.1", - "memchr", -] - [[package]] name = "rend" version = "0.4.2" @@ -5604,20 +5516,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "schemars" -version = "0.8.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3fbf2ae1b8bc8e02df939598064d22402220cd5bbcca1c76f7d6a310974d5615" -dependencies = [ - "chrono", - "dyn-clone", - "schemars_derive", - "serde", - "serde_json", - "uuid", -] - [[package]] name = "schemars" version = "0.9.0" @@ -5642,18 +5540,6 @@ dependencies = [ "serde_json", ] -[[package]] -name = "schemars_derive" -version = "0.8.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32e265784ad618884abaea0600a9adf15393368d840e0222d101a072f3f7534d" -dependencies = [ - "proc-macro2", - "quote", - "serde_derive_internals", - "syn 2.0.117", -] - [[package]] name = "scopeguard" version = "1.2.0" @@ -5970,17 +5856,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "serde_derive_internals" -version = "0.29.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "serde_json" version = "1.0.150" @@ -6027,18 +5902,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "serde_tokenstream" -version = "0.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7c49585c52c01f13c5c2ebb333f14f6885d76daa768d8a037d28017ec538c69" -dependencies = [ - "proc-macro2", - "quote", - "serde", - "syn 2.0.117", -] - [[package]] name = "serde_urlencoded" version = "0.7.1" @@ -6083,19 +5946,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "serde_yaml" -version = "0.9.34+deprecated" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" -dependencies = [ - "indexmap 2.14.0", - "itoa", - "ryu", - "serde", - "unsafe-libyaml", -] - [[package]] name = "sha1" version = "0.10.6" @@ -7396,53 +7246,6 @@ version = "1.20.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" -[[package]] -name = "typify" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7144144e97e987c94758a3017c920a027feac0799df325d6df4fc8f08d02068e" -dependencies = [ - "typify-impl", - "typify-macro", -] - -[[package]] -name = "typify-impl" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "062879d46aa4c9dfe0d33b035bbaf512da192131645d05deacb7033ec8581a09" -dependencies = [ - "heck 0.5.0", - "log", - "proc-macro2", - "quote", - "regress", - "schemars 0.8.22", - "semver", - "serde", - "serde_json", - "syn 2.0.117", - "thiserror 2.0.18", - "unicode-ident", -] - -[[package]] -name = "typify-macro" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9708a3ceb6660ba3f8d2b8f0567e7d4b8b198e2b94d093b8a6077a751425de9e" -dependencies = [ - "proc-macro2", - "quote", - "schemars 0.8.22", - "semver", - "serde", - "serde_json", - "serde_tokenstream", - "syn 2.0.117", - "typify-impl", -] - [[package]] name = "tzf-rel" version = "0.0.2025-c" @@ -7777,12 +7580,6 @@ dependencies = [ "subtle", ] -[[package]] -name = "unsafe-libyaml" -version = "0.2.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" - [[package]] name = "unsafe-libyaml-norway" version = "0.2.15" diff --git a/capsule-docs/src/content/docs/design/api-surfaces.md b/capsule-docs/src/content/docs/design/api-surfaces.md index 837da7bf..ccbdf887 100644 --- a/capsule-docs/src/content/docs/design/api-surfaces.md +++ b/capsule-docs/src/content/docs/design/api-surfaces.md @@ -29,7 +29,7 @@ Two consequences worth stating plainly: ## Why Two Transports -- **REST + OpenAPI** for every request/response surface: the OpenAPI schema is the machine-readable contract that generates `capsule-sdk` (progenitor), and a plain HTTP surface stays debuggable with nothing but `curl` — which matters for a self-hosted product. +- **REST + OpenAPI** for every request/response surface: the OpenAPI **3.1** schema is the machine-readable contract that generates `capsule-sdk`'s typed REST client via `spargen`, our in-house generator (in development; SLICES.md `S-D8`). Progenitor was dropped because it consumes OpenAPI 3.0 only, forcing a lossy 3.1→3.0 down-conversion — we do not downgrade schemas. A plain HTTP surface stays debuggable with nothing but `curl` — which matters for a self-hosted product. - **gRPC** only where a typed, paged feed contract earns it: the sync feed and its federation twin are one proto contract consumed by every client and every peer server, with the signed manifest traveling as opaque canonical CBOR inside it (never re-modeled as proto fields — re-encoding would detach it from its signatures). ## Legacy: GraphQL (retiring) diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index f7d240f5..6193a718 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -13,7 +13,7 @@ The mapping reflects the *design intent*. Modules not yet implemented are annota | Crate | Purpose | | ------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `capsule-core` | Shared logic across server and clients: cryptography, library layout, import pipeline, metadata, ML orchestration | -| `capsule-sdk` | Client SDK: auto-generated OpenAPI client, upload protocol, LAN-peering glue | +| `capsule-sdk` | Client SDK: the sanctioned network path (auth/session + token refresh, upload protocol, sync, LAN-peering glue); typed REST client generated by `spargen` from OpenAPI 3.1 (in development) | | `capsule-core-ffi` | uniffi bindings crate for native Swift/Kotlin consumers (sidecar `Catalog` surface; the `HardwareSigner` foreign trait ships under `capsule-core`'s `ffi` feature) | | `capsule-api` | Server entry-point + routing | | `capsule-api-auth` | Authentication, sessions, OIDC, device directory | diff --git a/capsule-sdk/.gitignore b/capsule-sdk/.gitignore deleted file mode 100644 index 17998065..00000000 --- a/capsule-sdk/.gitignore +++ /dev/null @@ -1 +0,0 @@ -openapi.json diff --git a/capsule-sdk/Cargo.toml b/capsule-sdk/Cargo.toml index e39b8642..1669073b 100644 --- a/capsule-sdk/Cargo.toml +++ b/capsule-sdk/Cargo.toml @@ -10,7 +10,6 @@ serde = { workspace = true } thiserror = { workspace = true } uuid = { version = "1.23.3", features = ["v7", "serde"] } -progenitor = { version = "0.11.2" } reqwest = { version = "0.12", default-features = false, features = [ "json", "rustls-tls", diff --git a/capsule-sdk/README.md b/capsule-sdk/README.md index 265de3e2..968e5e0a 100644 --- a/capsule-sdk/README.md +++ b/capsule-sdk/README.md @@ -11,10 +11,11 @@ SDK for Capsule API. Assess all Capsule APIs statelessly via one library only. N ## Development -- Generate OpenAPI spec: - - Start up external dependencies with `podman compose up` in [capsule-api](../capsule-api). - - Run `./generate_openapi.sh` in [capsule-sdk](./). - - Note: [Progenitor](https://github.com/oxidecomputer/progenitor) is used to generate the SDK as long as the OpenAPI spec is provided at the assumed path. +- The typed REST client is generated from the server's OpenAPI **3.1** schema by `spargen`, + our in-house generator (in development; slice `S-D8` in the repo-root `SLICES.md`). + The previous progenitor pipeline required a lossy 3.1→3.0 schema down-conversion and is + gone — we do not downgrade schemas. Until spargen lands, `AuthenticatedClient` is parked + (commented out) in `src/lib.rs` and the hand-written surfaces stand alone. ## Usage diff --git a/capsule-sdk/examples/auth_flow.rs b/capsule-sdk/examples/auth_flow.rs deleted file mode 100644 index 943baba1..00000000 --- a/capsule-sdk/examples/auth_flow.rs +++ /dev/null @@ -1,38 +0,0 @@ -use std::env; - -use capsule_sdk::types::AuthModelsRequestsLoginRequest; -use capsule_sdk::{AuthenticatedClient, Client}; - -#[tokio::main] -async fn main() -> Result<(), Box> { - let base_url = - env::var("CAPSULE_API_URL").unwrap_or_else(|_| "http://localhost:3000".to_string()); - - let client = Client::new(&base_url); - - let email = String::from("johndoe@email.com"); - let password = String::from("password"); - - println!("Logging in with: {}; {}", email, password); - - let login_req = AuthModelsRequestsLoginRequest { email, password }; - let res = match client.login_user(&login_req).await { - Ok(res) => { - println!("Login successful!"); - res - } - Err(e) => { - eprintln!("Failed to login: {:?}", e); - std::process::exit(1); - } - }; - println!("Access Token: {}", res.access_token); - - // Verify profile - println!("Fetching profile..."); - let authenticated_client = AuthenticatedClient::new(&base_url, &res.access_token); - let profile = authenticated_client.get_user_profile().await?; - println!("Profile: {:#?}", profile); - - Ok(()) -} diff --git a/capsule-sdk/generate_openapi.sh b/capsule-sdk/generate_openapi.sh deleted file mode 100755 index cf4731fc..00000000 --- a/capsule-sdk/generate_openapi.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd) -API_DIR="$SCRIPT_DIR/../capsule-api" -SDK_DIR="$SCRIPT_DIR/../capsule-sdk" - -cd "$API_DIR" - -echo "Generating OpenAPI 3.1 spec..." -cargo run --bin gen_openapi --features full -- ./openapi.json - -if [ ! -s "./openapi.json" ]; then - echo "Error: ./openapi.json was not generated or is empty." >&2 - exit 1 -fi - -# Downgrading to OpenAPI 3.0... - -npx @apiture/openapi-down-convert --input ./openapi.json > $SDK_DIR/openapi.json diff --git a/capsule-sdk/src/lib.rs b/capsule-sdk/src/lib.rs index 898126f4..01ec0021 100644 --- a/capsule-sdk/src/lib.rs +++ b/capsule-sdk/src/lib.rs @@ -1,62 +1,80 @@ -progenitor::generate_api!("openapi.json"); +//! Capsule client SDK: the sanctioned network path for every Capsule client +//! (upload protocol, sync/download, auth flows). +//! +//! # REST client generation (spargen; slice `S-D8` in the repo-root `SLICES.md`) +//! +//! The typed REST client is generated from the server's **OpenAPI 3.1** schema by +//! `spargen`, our in-house generator (in development). The previous progenitor +//! pipeline is gone deliberately: progenitor consumes OpenAPI 3.0 only, which +//! forced a lossy 3.1→3.0 schema down-conversion — a standing source of drift and +//! failures. We do not downgrade schemas. Until spargen lands, the hand-written +//! surfaces below (the adaptive upload strategy in [`upload`]) stand alone, and +//! the generated-client wrapper is parked below. pub mod upload; -/// Authenticated OpenAPI client -pub struct AuthenticatedClient { - /// Base URL of the API - base_url: String, - /// Current access token - access_token: String, - /// OpenAPI client - client: Client, -} - -impl AuthenticatedClient { - pub fn new(base_url: &str, access_token: &str) -> AuthenticatedClient { - AuthenticatedClient { - base_url: base_url.to_string(), - access_token: access_token.to_string(), - client: Self::get_authenticated_client(base_url, access_token), - } - } - - fn get_authenticated_client(base_url: &str, access_token: &str) -> Client { - let authorization_header = format!("Bearer {}", access_token); - - let mut headers = reqwest::header::HeaderMap::new(); - headers.insert( - reqwest::header::AUTHORIZATION, - authorization_header.parse().unwrap(), - ); - - let client_with_custom_defaults = reqwest::ClientBuilder::new() - .default_headers(headers) - .build() - .unwrap(); - - Client::new_with_client(base_url, client_with_custom_defaults) - } - - /// Returns same instance with new base URL - pub fn with_base_url(&mut self, base_url: &str) -> &mut Self { - self.base_url = base_url.to_string(); - self.client = Self::get_authenticated_client(base_url, &self.access_token); - self - } - - /// Returns same instance with new access token - pub fn with_access_token(&mut self, access_token: &str) -> &mut Self { - self.access_token = access_token.to_string(); - self.client = Self::get_authenticated_client(&self.base_url, access_token); - self - } -} - -impl std::ops::Deref for AuthenticatedClient { - type Target = Client; - - fn deref(&self) -> &Self::Target { - &self.client - } -} +// ─── Parked until spargen (S-D8) ──────────────────────────────────────────── +// `AuthenticatedClient` wraps the generated `Client` type, which does not exist +// without a generator. It is commented out — not deleted — because its shape +// (default bearer header, base-url/token swap, Deref to the typed client) is the +// contract S-D8 revives; S-D7 additionally gives it a token store with async +// refresh-expiry pre-flight so callers never juggle raw access tokens. +// +// /// Authenticated OpenAPI client +// pub struct AuthenticatedClient { +// /// Base URL of the API +// base_url: String, +// /// Current access token +// access_token: String, +// /// OpenAPI client +// client: Client, +// } +// +// impl AuthenticatedClient { +// pub fn new(base_url: &str, access_token: &str) -> AuthenticatedClient { +// AuthenticatedClient { +// base_url: base_url.to_string(), +// access_token: access_token.to_string(), +// client: Self::get_authenticated_client(base_url, access_token), +// } +// } +// +// fn get_authenticated_client(base_url: &str, access_token: &str) -> Client { +// let authorization_header = format!("Bearer {}", access_token); +// +// let mut headers = reqwest::header::HeaderMap::new(); +// headers.insert( +// reqwest::header::AUTHORIZATION, +// authorization_header.parse().unwrap(), +// ); +// +// let client_with_custom_defaults = reqwest::ClientBuilder::new() +// .default_headers(headers) +// .build() +// .unwrap(); +// +// Client::new_with_client(base_url, client_with_custom_defaults) +// } +// +// /// Returns same instance with new base URL +// pub fn with_base_url(&mut self, base_url: &str) -> &mut Self { +// self.base_url = base_url.to_string(); +// self.client = Self::get_authenticated_client(base_url, &self.access_token); +// self +// } +// +// /// Returns same instance with new access token +// pub fn with_access_token(&mut self, access_token: &str) -> &mut Self { +// self.access_token = access_token.to_string(); +// self.client = Self::get_authenticated_client(&self.base_url, access_token); +// self +// } +// } +// +// impl std::ops::Deref for AuthenticatedClient { +// type Target = Client; +// +// fn deref(&self) -> &Self::Target { +// &self.client +// } +// } diff --git a/capsule-sdk/src/upload.rs b/capsule-sdk/src/upload.rs index 2d554a36..46dce91c 100644 --- a/capsule-sdk/src/upload.rs +++ b/capsule-sdk/src/upload.rs @@ -6,11 +6,8 @@ // TODO: Use this module in SDK use std::collections::VecDeque; -use std::path::Path; use std::time::Duration; -use crate::AuthenticatedClient; - // Constants for chunk sizes (4KB aligned) const KB: u64 = 1024; const CHUNK_SIZE_256KB: u64 = 256 * KB; @@ -165,64 +162,70 @@ impl AdaptiveChunkSizeStrategy { } } -/// Upload client for managing chunked uploads to Capsule API -pub struct UploadClient { - /// Capsule API Client - client: AuthenticatedClient, -} - -impl UploadClient { - /// Create a new upload client - pub fn new(client: AuthenticatedClient) -> Self { - Self { client } - } - - /// Upload a file using adaptive chunked upload - /// - /// # Arguments - /// * `file_path` - Path to the file to upload - /// * `content_type` - Optional content type (defaults to auto-detection) - /// - /// # Returns - /// The upload session ID on success - pub async fn upload_file( - &self, - _file_path: &Path, - _content_type: Option<&str>, - ) -> Result { - // S-D1: 1. size + ciphertext hash; 2. create session (JSON body per the - // upload-protocol doc — size/hash/content_type/crypto_suite_id/ - // protocol_version/blob_role/manifest_envelope); 3. adaptive strategy; - // 4. chunk loop with the code-driven recovery matrix; 5. session id. - // No filename on the wire: plaintext metadata rides the encrypted - // metadata blob. - todo!("S-D1: upload driver — see SLICES.md") - } - - /// Create an upload session - pub async fn create_session( - &self, - _total_size: u64, - _content_type: Option<&str>, - ) -> Result { - // S-D1: POST /upload with the JSON body from the upload-protocol doc's - // endpoint table (size is the `size` body field, NOT a header). - todo!("S-D1: create session — see SLICES.md") - } - - /// Upload a single chunk - pub async fn upload_chunk( - &self, - _session_id: &str, - _data: &[u8], - _offset: u64, - ) -> Result { - // S-D1: PATCH /upload/{id} with Content-Type: application/octet-stream, - // X-Capsule-Offset, and the REQUIRED X-Capsule-Checksum (lowercase-hex - // SHA-256 of the chunk bytes). Returns the new offset on success. - todo!("S-D1: upload chunk — see SLICES.md") - } -} +// ─── Parked until spargen (S-D8) + the S-D1 upload driver ────────────────── +// `UploadClient` transports over `AuthenticatedClient`, which is parked in +// lib.rs until the spargen-generated client exists. The adaptive strategy and +// error taxonomy below are live and tested; only the transport is parked. +// +// /// Upload client for managing chunked uploads to Capsule API +// pub struct UploadClient { +// /// Capsule API Client +// client: AuthenticatedClient, +// } +// +// impl UploadClient { +// /// Create a new upload client +// pub fn new(client: AuthenticatedClient) -> Self { +// Self { client } +// } +// +// /// Upload a file using adaptive chunked upload +// /// +// /// # Arguments +// /// * `file_path` - Path to the file to upload +// /// * `content_type` - Optional content type (defaults to auto-detection) +// /// +// /// # Returns +// /// The upload session ID on success +// pub async fn upload_file( +// &self, +// _file_path: &Path, +// _content_type: Option<&str>, +// ) -> Result { +// // S-D1: 1. size + ciphertext hash; 2. create session (JSON body per the +// // upload-protocol doc — size/hash/content_type/crypto_suite_id/ +// // protocol_version/blob_role/manifest_envelope); 3. adaptive strategy; +// // 4. chunk loop with the code-driven recovery matrix; 5. session id. +// // No filename on the wire: plaintext metadata rides the encrypted +// // metadata blob. +// todo!("S-D1: upload driver — see SLICES.md") +// } +// +// /// Create an upload session +// pub async fn create_session( +// &self, +// _total_size: u64, +// _content_type: Option<&str>, +// ) -> Result { +// // S-D1: POST /upload with the JSON body from the upload-protocol doc's +// // endpoint table (size is the `size` body field, NOT a header). +// todo!("S-D1: create session — see SLICES.md") +// } +// +// /// Upload a single chunk +// pub async fn upload_chunk( +// &self, +// _session_id: &str, +// _data: &[u8], +// _offset: u64, +// ) -> Result { +// // S-D1: PATCH /upload/{id} with Content-Type: application/octet-stream, +// // X-Capsule-Offset, and the REQUIRED X-Capsule-Checksum (lowercase-hex +// // SHA-256 of the chunk bytes). Returns the new offset on success. +// todo!("S-D1: upload chunk — see SLICES.md") +// } +// } +// /// Response from creating an upload session #[derive(Debug)] @@ -253,10 +256,10 @@ pub enum UploadError { impl std::fmt::Display for UploadError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { - UploadError::IoError(e) => write!(f, "IO error: {}", e), - UploadError::NetworkError(msg) => write!(f, "Network error: {}", msg), + UploadError::IoError(e) => write!(f, "IO error: {e}"), + UploadError::NetworkError(msg) => write!(f, "Network error: {msg}"), UploadError::ServerError { status, message } => { - write!(f, "Server error ({}): {}", status, message) + write!(f, "Server error ({status}): {message}") } UploadError::ChecksumMismatch => write!(f, "Checksum mismatch"), UploadError::SessionNotFound => write!(f, "Session not found or expired"), @@ -303,11 +306,17 @@ mod tests { #[test] fn test_adaptive_scaling_up() { let mut strategy = AdaptiveChunkSizeStrategy::for_file_size(50 * 1024 * 1024); - // Simulate 3 fast uploads (high throughput) + // Warm-up rule (upload-protocol doc, §Adaptive Chunk Sizing): no + // adjustment until ≥5 chunks or ≥8 MiB at the current size. Three fast + // chunks must NOT scale yet… for _ in 0..3 { strategy.record_chunk(CHUNK_SIZE_1MB, Duration::from_millis(100)); } - // After 3 fast chunks, should scale up + assert_eq!(strategy.current_size, CHUNK_SIZE_1MB); + // …but sustained >5 MB/s past the warm-up doubles the size. + for _ in 0..2 { + strategy.record_chunk(CHUNK_SIZE_1MB, Duration::from_millis(100)); + } assert!(strategy.current_size > CHUNK_SIZE_1MB); } } diff --git a/mise.toml b/mise.toml index 3aca22d9..6487bf14 100644 --- a/mise.toml +++ b/mise.toml @@ -112,10 +112,10 @@ run = "cargo fmt" run = "cargo fmt --check" [tasks.lint-rust] -run = "cargo clippy --workspace --exclude capsule-sdk --fix --allow-dirty -- $CLIPPY_FLAGS" +run = "cargo clippy --workspace --fix --allow-dirty -- $CLIPPY_FLAGS" [tasks.lint-check-rust] -run = "cargo clippy --workspace --exclude capsule-sdk -- $CLIPPY_FLAGS" +run = "cargo clippy --workspace -- $CLIPPY_FLAGS" # nextest: cross-binary parallel scheduling + process-per-test isolation. Two # invocations — the workspace (default features), then capsule-core's FFI surface. @@ -123,15 +123,15 @@ run = "cargo clippy --workspace --exclude capsule-sdk -- $CLIPPY_FLAGS" # `cargo test --doc` step). CI selects the `ci` profile via NEXTEST_PROFILE=ci. [tasks.test-rust] run = [ - "cargo nextest run --workspace --exclude capsule-sdk", + "cargo nextest run --workspace", "cargo nextest run -p capsule-core --features ffi", ] [tasks.test-coverage-rust] -run = "cargo llvm-cov nextest --workspace --exclude capsule-sdk --fail-under-lines 0" +run = "cargo llvm-cov nextest --workspace --fail-under-lines 0" [tasks.build-rust] -run = "cargo build --workspace --exclude capsule-sdk" +run = "cargo build --workspace" # Compile the canonical locales/ catalogs into each platform's native i18n format # (Rust bundle, web JSON, Android strings.xml, iOS .xcstrings). See xtask/src/i18n.rs. From 022c6c2db4ac6a12d13521341537240bc3d38a7f Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:47:07 -0400 Subject: [PATCH 22/58] docs(design): add the Local Gallery owner contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 'Every native app works as a local gallery even with no server' was implied across four docs (principles' offline/online divide, api-surfaces' client- side queries, filesystem/client's local index, clients' duties) but owned by none — so it wasn't testable and couldn't accrete requirements. The new local-gallery.md is the owner: FRs (full gallery offline; never-signed-in is a valid mode; import offline; degradation visible never structural; rebuild without a server), security requirements (Recently-Deleted and Hidden views gated behind fresh biometric/credential re-auth with a short grace window; an honest per-platform at-rest posture matrix — mobile sandbox+FBE protects library bytes from other apps and casual file access, desktop is OS user permissions and we say so; no plaintext spillage outside the library root), and assertable NFRs (zero network I/O on read paths; an airplane-mode E2E case). Mechanism docs keep their content and link here for the contract. --- capsule-docs/astro.config.mjs | 1 + .../src/content/docs/design/clients.md | 2 + .../content/docs/design/filesystem/client.md | 2 + .../src/content/docs/design/local-gallery.md | 42 +++++++++++++++++++ 4 files changed, 47 insertions(+) create mode 100644 capsule-docs/src/content/docs/design/local-gallery.md diff --git a/capsule-docs/astro.config.mjs b/capsule-docs/astro.config.mjs index 4eff2135..1394b558 100644 --- a/capsule-docs/astro.config.mjs +++ b/capsule-docs/astro.config.mjs @@ -115,6 +115,7 @@ export default defineConfig({ items: [ { slug: 'design/organization' }, { slug: 'design/clients' }, + { slug: 'design/local-gallery' }, { slug: 'design/i18n' }, { slug: 'design/ai' }, ], diff --git a/capsule-docs/src/content/docs/design/clients.md b/capsule-docs/src/content/docs/design/clients.md index cd26f1b2..47447f49 100644 --- a/capsule-docs/src/content/docs/design/clients.md +++ b/capsule-docs/src/content/docs/design/clients.md @@ -8,6 +8,8 @@ Capsule's clients are native per platform, with as little divergence as possible The boundary this doc owns is **what every client must do** — the client-class duties that, if skipped, put the client in the *faulty* class (see [Threat Model — Client Class Taxonomy](/design/threat-model/#client-class-taxonomy)). Plus the sandboxed-decoder pattern, which is the largest remaining attack surface on the client. +The offline-first requirement set every native client must satisfy — the local-gallery FRs/NFRs, the gated Recently-Deleted/Hidden views, and the at-rest posture matrix — is owned by [Local Gallery](/design/local-gallery/). + ## Design Priorities - **Native.** Native implementations per platform ensure familiar usability and enable platform-specific optimizations. diff --git a/capsule-docs/src/content/docs/design/filesystem/client.md b/capsule-docs/src/content/docs/design/filesystem/client.md index 75fb2482..1e9b0b3c 100644 --- a/capsule-docs/src/content/docs/design/filesystem/client.md +++ b/capsule-docs/src/content/docs/design/filesystem/client.md @@ -10,6 +10,8 @@ What a client keeps locally depends on its sync setting — *metadata only*, *me The directory layout below is itself a contract — the recovery-first rebuild assumes exactly these filenames and sharding rules. +This doc owns the on-disk mechanisms; the offline-first requirement contract they serve (FRs/NFRs, gated views, at-rest posture) is owned by [Local Gallery](/design/local-gallery/). + ## Desktop Library Layout ```text diff --git a/capsule-docs/src/content/docs/design/local-gallery.md b/capsule-docs/src/content/docs/design/local-gallery.md new file mode 100644 index 00000000..995afb67 --- /dev/null +++ b/capsule-docs/src/content/docs/design/local-gallery.md @@ -0,0 +1,42 @@ +--- +title: Local Gallery +description: The owner contract for offline-first native clients — every native app is a complete local gallery with no server +status: draft +--- + +Every native Capsule app is a **complete local gallery first** and a synced client second. This doc owns that requirement set — the functional and non-functional requirements that make "works with no server" a testable contract rather than a slogan. The mechanisms live where they always did — the local index in [Filesystem — Client](/design/filesystem/client/), the client-side query surface in [API Surfaces](/design/api-surfaces/#surface--transport-map), client duties in [Clients](/design/clients/) — this doc owns the *requirements* and links the mechanisms. + +The [offline/online divide principle](/design/principles/#principles) is the root: a feature works either solely online or solely offline, never differently by connectivity. This doc pins down which side of the divide the gallery sits on: **all of it is offline**. + +## Functional Requirements + +- **FR1 — Full gallery offline.** Browse (timeline, albums, stacks), search, filter, view full-resolution locally-held assets, and organize (albums, tags, ratings, [culling flags](/design/organization/), hidden) with zero connectivity. Every one of these reads and writes `library.sqlite` + the local blob store ([Filesystem — Client](/design/filesystem/client/)); none has a server dependency by construction ([rich queries have no server surface at all](/design/api-surfaces/#surface--transport-map)). +- **FR2 — Never-signed-in is a valid mode.** A user who installs a native app and never connects a server has a fully functional local photo library: import, organize, search, export. Sync is an *addition* to a working product, not its precondition. (The crypto data plane operates locally; enrollment happens when — if ever — a server is added.) +- **FR3 — Import works offline.** The scan → plan → execute [import pipeline](/design/import/pipeline/) is entirely local; upload is a separate, later concern. (The one deliberate exception: [streaming import-upload mode](/design/import/pipeline/#import-upload-streaming-mode) exists *because* the device cannot hold the bytes — it requires a server by definition.) +- **FR4 — Degradation is visible, never structural.** When connectivity exists but a *remote-only* representation is unavailable, the asset stays listed with its best local representation and a non-destructive unavailability state ([the degrade ladder](/design/import/download-sync/#tiered-on-demand-fetch)). No offline condition ever removes an asset, an album, or an index entry. +- **FR5 — Recovery without a server.** A damaged local index is rebuilt from local sidecars alone ([recovery-first rebuild](/design/filesystem/maintenance/)); the rebuild requires no network. + +## Security Requirements + +- **SR1 — Gated views require fresh local auth.** Opening the **Recently Deleted** (trash) view or the **Hidden** view requires fresh local authentication — biometric where enrolled (Face ID / Touch ID / BiometricPrompt), else the device or account credential. One grant covers a short grace window (default 5 minutes, per-view), after which re-auth is required. The gate is view-time UX protection against a borrowed-unlocked-phone snoop; it is **not** a cryptographic boundary (the same data is reachable through the filesystem by anyone who can defeat the platform sandbox — see SR2 for what actually protects bytes at rest). +- **SR2 — At-rest posture per platform, stated honestly.** + | Platform | Store | Protection from other apps / the user's file browser | + | --- | --- | --- | + | iOS | App-private container | Sandbox denies other apps; not visible in Files unless exported; file-level Data Protection (FBE) at rest | + | Android | `Context` private storage (`filesDir`) | Sandbox + SELinux deny other apps and `content://` browsing; File-Based Encryption at rest | + | Desktop (macOS/Windows/Linux) | User-directory library ([layout](/design/filesystem/client/#desktop-library-layout)) | OS user permissions only — any process of the same user can read it. Full-disk encryption is the at-rest story; we do not pretend otherwise. | + + On mobile, library content therefore **is** protected from the user's own casual file access and from other apps — by platform sandbox, which is the strongest guarantee actually available. Rooted/jailbroken devices void it; we do not claim otherwise. +- **SR3 — No plaintext spillage outside the library root.** Decoded previews, caches, and temp files live inside the app-private library root ([placement rules](/design/filesystem/client/#mobile-clients)), never in shared/world-readable locations. Export is the only sanctioned way bytes leave the root, and it applies the [privacy-on-export strip](/design/metadata/#privacy-on-export). + +## Non-Functional Requirements + +- **NFR1 — No network on read paths (assertable).** Gallery read paths (timeline, album, search, asset view of locally-held tiers) perform zero network I/O — not "tolerate failure," but *do not attempt*. This is testable by construction: run the read-path suite with networking disabled and assert no socket is ever opened. +- **NFR2 — Airplane-mode E2E.** The bounded E2E surface includes one case: import → browse → search → organize → export on a device in airplane mode from first launch (never enrolled). It must behave identically to the connected case minus sync surfaces. +- **NFR3 — Offline startup is not a degraded startup.** Cold start offline takes the same code path as cold start online; there is no blocking connectivity probe on the startup path. + +## Validation + +- **Unit:** gated-view policy (grace window expiry, per-view independence, biometric-fallback-to-credential); placement rules for every cache/temp write (path is under the library root). +- **Smoke:** read-path network assertion (NFR1) with a socket-refusing harness; index rebuild offline (FR5); never-enrolled library round-trip (FR2). +- **E2E:** the airplane-mode case (NFR2), listed in the [Module Map E2E surface](/design/module-map/#e2e-test-surface). From e7b714efb6487f58cdc04c15ba41de27abe1edc0 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:51:11 -0400 Subject: [PATCH 23/58] feat(core)!: make grouping convergent by construction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The architect requirement is that every grouping operation — manual and automatic/AI — be idempotent and order-independent. Tags (OR-set) and caption/rating (LWW) already were; stack_membership was the one grouping structure with no defined merge rule (a plain optional field, silently last-write-wins by whole-sidecar rewrite). Since v1 has not shipped, the schema is fixed at the root: stack_membership becomes an LWW register over Option — join, move, and leave (a stamped None) are all the same (ts, device_id)-ordered write, using the exact machinery caption and rating already use. A never-written register stays wire-absent, so existing stackless sidecars encode byte-identically (regression-tested); convergence and replay-idempotency have dedicated tests. Docs now state the requirement once as a normative table (metadata.md 'Grouping Convergence'): OR-set / LWW / computed-view / single-home-plus- ordered-lifecycle, each with why it converges. ai.md pins AI grouping as a pure function of (inputs, model_id, model_version) with sorted inputs and fixed seeds — clustering is otherwise order-sensitive — and fixes the sequencing contract for the deferred quality evaluations (best shot/ framing/exposure): they run strictly after grouping, keyed by (group_id, membership_hash, model_id, model_version), so any regroup invalidates them by key construction and the recompute is deterministic. That deferral gets a slice (S-H4) in the SLICES reconciliation commit. --- capsule-core/src/lifecycle.rs | 6 +- capsule-core/src/metadata/export_policy.rs | 2 +- capsule-core/src/sidecar/sidecar_v1.rs | 89 +++++++++++++++++-- capsule-docs/src/content/docs/design/ai.md | 4 + .../src/content/docs/design/metadata.md | 23 +++-- .../src/content/docs/design/organization.md | 5 +- 6 files changed, 113 insertions(+), 16 deletions(-) diff --git a/capsule-core/src/lifecycle.rs b/capsule-core/src/lifecycle.rs index 9b1d4bd8..4fe03890 100644 --- a/capsule-core/src/lifecycle.rs +++ b/capsule-core/src/lifecycle.rs @@ -48,7 +48,7 @@ use crate::crypto::provenance::{AssetManifest, ProvenanceChain, ProvenanceRecord use crate::crypto::verify_asset::{VerifyOutcome, verify_asset}; use crate::db::{AssetRow, CachedRepresentationRow, DatabaseDriver}; use crate::library::Library; -use crate::metadata::crdt::{AddId, Counter}; +use crate::metadata::crdt::{AddId, Counter, Lww}; use crate::sidecar::sidecar_v1::{SIDECAR_SCHEMA_V1, SidecarV1}; /// A device is treated as added far in the past so any import timestamp postdates it. @@ -531,7 +531,7 @@ impl Workspace { tags_ai: Default::default(), caption: Default::default(), rating: Default::default(), - stack_membership: None, + stack_membership: Lww::new(), camera_id: None, device_id: self.account.device.device_id, session_id: Uuid::now_v7(), @@ -817,7 +817,7 @@ impl Workspace { tags_ai: Default::default(), caption: Default::default(), rating: Default::default(), - stack_membership: None, + stack_membership: Lww::new(), camera_id: None, device_id: head.core.created_by_device, session_id: Uuid::now_v7(), diff --git a/capsule-core/src/metadata/export_policy.rs b/capsule-core/src/metadata/export_policy.rs index 48a8ca58..97905623 100644 --- a/capsule-core/src/metadata/export_policy.rs +++ b/capsule-core/src/metadata/export_policy.rs @@ -81,7 +81,7 @@ mod tests { tags_ai: Default::default(), caption: Default::default(), rating: Default::default(), - stack_membership: None, + stack_membership: Default::default(), camera_id: Some(CameraId { model: "iPhone 15 Pro".into(), serial: "SECRET-SERIAL".into(), diff --git a/capsule-core/src/sidecar/sidecar_v1.rs b/capsule-core/src/sidecar/sidecar_v1.rs index 8f25be9c..a3932c2d 100644 --- a/capsule-core/src/sidecar/sidecar_v1.rs +++ b/capsule-core/src/sidecar/sidecar_v1.rs @@ -141,8 +141,12 @@ pub struct SidecarV1 { pub caption: Lww, /// Rating LWW register. pub rating: Lww, - /// Stack membership, if any. - pub stack_membership: Option, + /// Stack membership: an LWW register over `Option` so stack + /// edits (join, move, leave = `None`) converge order-independently under the + /// same `(ts, device_id)` rule as caption/rating — grouping is convergent by + /// construction, not convention. Wire-absent default = the never-written + /// register (distinct from an explicit removal, which is a stamped `None`). + pub stack_membership: Lww>, /// Camera identifier (export-stripped). pub camera_id: Option, /// Importing device id (UUIDv4; export-stripped). @@ -207,7 +211,11 @@ impl SidecarV1 { put!("tags_ai", self.tags_ai); put!("caption", self.caption); put!("rating", self.rating); - put_opt!("stack_membership", self.stack_membership); + // Wire-absent default: a never-written register is omitted so pre-existing + // sidecars (and stackless assets) encode byte-identically. + if self.stack_membership != Lww::default() { + put!("stack_membership", self.stack_membership); + } put_opt!("camera_id", self.camera_id); put!("device_id", self.device_id); put!("session_id", self.session_id); @@ -304,7 +312,10 @@ impl SidecarV1 { let tags_ai = req!("tags_ai", OrSet); let caption = req!("caption", Lww); let rating = req!("rating", Lww); - let stack_membership = opt!("stack_membership", StackMembership); + let stack_membership = match text.remove("stack_membership") { + None | Some(Value::Null) => Lww::new(), + Some(v) => from_value::>>(v)?, + }; let camera_id = opt!("camera_id", CameraId); let device_id = req!("device_id", Uuid); let session_id = req!("session_id", Uuid); @@ -361,7 +372,7 @@ mod tests { tags_ai: OrSet::new(), caption: Lww::new(), rating: Lww::new(), - stack_membership: None, + stack_membership: Lww::new(), camera_id: Some(CameraId { model: "iPhone 15 Pro".into(), serial: "ABC123".into(), @@ -453,4 +464,72 @@ mod tests { let s = minimal(); assert_eq!(s.to_canonical_vec(), s.to_canonical_vec()); } + + #[test] + fn absent_stack_membership_is_wire_absent_and_byte_stable() { + // A never-written register is omitted from the wire, so pre-existing + // sidecars (and stackless assets) encode byte-identically to before the + // Lww wrap. + let s = minimal(); + let bytes = s.to_canonical_vec(); + let needle = b"stack_membership"; + assert!(!bytes.windows(needle.len()).any(|w| w == needle)); + let back = SidecarV1::from_canonical_slice(&bytes, SIDECAR_SCHEMA_V1).unwrap(); + assert_eq!(back.stack_membership, Lww::new()); + } + + #[test] + fn stack_membership_round_trips_and_survives_signing() { + let ik = HybridSigningKey::from_seed_bytes(&[1; 32], &[2; 32]); + let mut s = minimal(); + s.stack_membership.set( + Some(StackMembership { + stack_id: Uuid::from_u128(0x51), + stack_type: StackType::Burst, + role: StackRole::Primary, + member_index: Some(0), + }), + "2026-05-31T12:00:00Z", + Uuid::from_u128(0xD1), + ); + s.sign(&ik); + let back = + SidecarV1::from_canonical_slice(&s.to_canonical_vec(), SIDECAR_SCHEMA_V1).unwrap(); + assert_eq!(back, s); + assert!(back.verify(&ik.verifying_key())); + } + + #[test] + fn concurrent_stack_edits_converge_order_independently() { + // Replica A joins a stack at 10:00; replica B removes membership at + // 11:00. Merging in either order — or replaying either op again — + // converges on the removal: grouping is idempotent and + // order-independent by construction (metadata doc, Convergence table). + let membership = Some(StackMembership { + stack_id: Uuid::from_u128(0x51), + stack_type: StackType::Burst, + role: StackRole::Member, + member_index: Some(3), + }); + let mut a: Lww> = Lww::new(); + a.set( + membership.clone(), + "2026-05-31T10:00:00Z", + Uuid::from_u128(0xA), + ); + let mut b: Lww> = Lww::new(); + b.set(None, "2026-05-31T11:00:00Z", Uuid::from_u128(0xB)); + + let mut ab = a.clone(); + ab.merge(&b); + let mut ba = b.clone(); + ba.merge(&a); + assert_eq!(ab.get(), ba.get()); + assert_eq!(ab.get(), Some(&None)); // the removal wins + + // Idempotent: replaying the stale join changes nothing. + let mut replay = ab.clone(); + replay.merge(&a); + assert_eq!(replay.get(), ab.get()); + } } diff --git a/capsule-docs/src/content/docs/design/ai.md b/capsule-docs/src/content/docs/design/ai.md index 15d77881..a0d0d820 100644 --- a/capsule-docs/src/content/docs/design/ai.md +++ b/capsule-docs/src/content/docs/design/ai.md @@ -25,6 +25,8 @@ AI inference can be wrong, biased, or hallucinatory. A core rule prevents it fro A hallucinating model can pollute its own namespace, never user intent. This is the structural defense against the "AI mistake silently overwrites user data" damage class — see [Threat Model — Forbidden Client Behaviors](/design/threat-model/schema-rules/#forbidden-client-behaviors). +**AI grouping is a pure function.** Every automatic grouping output (clusters, similarity groups, scene groupings) is a deterministic function of `(input asset set, model_id, model_version)` — inputs are processed in sorted asset-id order and any stochastic step (cluster seeding, initialization) uses fixed seeds, because most clustering algorithms are otherwise order-sensitive. Re-running a grouping over the same inputs yields byte-identical output; growing the input set and re-running is the only way results change. This is what makes automatic grouping idempotent and order-independent under the [grouping-convergence requirement](/design/metadata/#grouping-convergence-requirement). + ## Semantic Indexing Semantic search converts an image and a text query into vectors and measures their distance. Because embeddings are generated client-side, every device must run the same canonical model along a deterministic path so vectors are comparable — the constraint and its platform-partition fallback are specified in [Embedding Provenance](#embedding-provenance). @@ -41,6 +43,8 @@ Face Detection & Matching (clustering) runs the **Face Detection** and **Face Re Deferred to post-v1. The category and its sidecar fields are reserved in the [containment model](#ai-output-containment) so it can land later without a schema change; the Quality candidate models in the [inventory](#models-and-algorithms) are not part of the v1 pipeline. +**Sequencing contract (fixed now so the deferral can't drift):** group-scoped evaluations — best shot, best framing, best exposure within a stack, burst, or similarity group — run strictly **after** grouping, never interleaved with it. Each evaluation result is keyed by `(group_id, membership_hash, model_id, model_version)`, where `membership_hash` is the hash of the group's sorted member ids. Any membership change therefore invalidates the group's evaluations *by key construction* — no invalidation bookkeeping to forget — and the recompute is deterministic (same members, same model → same result; ties broken by asset id). Evaluation outputs are derived, AI-namespaced state: the durable, user-visible outcome remains the [`role = primary` pointer](/design/organization/#asset-stacking) a user or the client sets from them. Implementation is slice `S-H4` (after semantic/face features). + ## Model Batching On-device inference is memory- and power-bound, so execution mode is chosen per device: diff --git a/capsule-docs/src/content/docs/design/metadata.md b/capsule-docs/src/content/docs/design/metadata.md index 6286cfde..71592ad4 100644 --- a/capsule-docs/src/content/docs/design/metadata.md +++ b/capsule-docs/src/content/docs/design/metadata.md @@ -33,8 +33,10 @@ SidecarV1 { superseded_captions: Vec<{ value: String, written_by: device_id, ts: RFC3339 }>, // bounded ≤ 16 rating_lww: Option<{ value: u8, ts: RFC3339, by: device_id }>, - // organization — stack grouping; StackMembership shape owned by Asset Organization - stack_membership: Option, + // organization — stack grouping; StackMembership shape owned by Asset Organization. + // An LWW register over Option (leave = a stamped None), wire-absent + // when never written, so stack edits converge like caption/rating. + stack_membership: Lww>, // identifiers (see Identifiers below; privacy-on-export rules apply) camera_id: Option<{ model: String, serial: String }>, @@ -152,7 +154,7 @@ Capsule's *own* devices syncing the *same user's* library do **not** trigger thi User-editable metadata on a shared album — tags, captions, ratings — can be edited concurrently on different devices, including offline. To make these merges deterministic, such fields are modelled as CRDTs: - **Tags:** an OR-set (observed-remove set) with explicit [`add_id` binding](#add-id-binding), so a tag added on one device and removed on another converge predictably, and a remove that targets an unknown `add_id` is rejected rather than treated as a no-op. -- **Single-value fields** (`caption_lww`, `rating_lww`): last-writer-wins registers keyed by a signed timestamp and the writing `device_id` as the lexicographic tiebreaker. +- **Single-value fields** (`caption_lww`, `rating_lww`, `stack_membership`): last-writer-wins registers keyed by a signed timestamp and the writing `device_id` as the lexicographic tiebreaker. `stack_membership`'s value domain includes "no membership" (a stamped `None`), so joining, moving between, and leaving stacks are all the same LWW write and converge identically. ### Surfacing Concurrent Edits @@ -169,10 +171,21 @@ We encrypt the **operations**, not the resulting state. Merges are then commutat Each operation carries the same `prior_provenance_hash` chain link as any [lifecycle action](/design/authorization/#the-closed-action-set), so a metadata-update is provenance-tracked exactly like a create or delete. -Album *membership* is deliberately **not** a CRDT here — it is driven by MLS proposals and commits (see [Cryptography — MLS](/design/cryptography/mls/)), which already resolve concurrent changes. - The same encrypted-operation path also carries the per-owner **library-settings document** — [smart-album](/design/organization/#system--smart-albums-views) definitions (predicate + display name) and similar client-authored organizational state — synced and merged across devices like any other collaborative metadata, and never legible to the server. (The [default-album](/design/organization/#the-default-album) *designation* is separate: a non-secret server-side owner pointer, not part of this encrypted document.) +### Grouping Convergence (Requirement) + +**Every grouping operation — manual or automatic/AI — is idempotent and order-independent.** Applying the same operation twice, or applying a set of operations in any arrival order, yields the same state. This is a requirement satisfied *by construction*, not by convention; each grouping structure names its mechanism: + +| Structure | Mechanism | Why it converges | +| --- | --- | --- | +| Tags (`tags_user` / `tags_ai`) | OR-set | Add/remove keyed by `add_id`; merges commutative, associative, idempotent | +| Caption / rating / `stack_membership` | LWW register | Total order on `(ts, device_id)`; replay of any op is a no-op | +| Smart albums, people clusters, [aggregated federated albums](/design/federation/) | Computed views | Nothing stored — membership is a deterministic function of inputs; recomputation is idempotent by definition ([views](/design/organization/#system--smart-albums-views), [AI determinism](/design/ai/#ai-output-containment)) | +| Container-album membership | Single home + ordered lifecycle ops | Exactly one container per asset; a move is a signed lifecycle action whose replay finds the target state already in place and no-ops ([Organization](/design/organization/#container-albums)); concurrency is resolved by MLS commit order, below | + +Album *membership* is deliberately **not** a CRDT here — it is driven by MLS proposals and commits (see [Cryptography — MLS](/design/cryptography/mls/)), which already resolve concurrent changes into one total order. + This LWW/OR-set approach is intentionally simpler than a full event-graph with state resolution: photo metadata does not need it, and the extra machinery would not be functionally justified. ## Tag Provenance and Namespacing diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index 52597f62..759adefa 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -19,6 +19,7 @@ The UI calls two different things "albums," and the design keeps them strictly s A container album is Capsule's primary organizational unit and its primary **sharing and access-control boundary**. An album *is* an MLS group: its cryptographic identity (the per-epoch [AMK](/design/cryptography/keys/#album-master-keys-amks)) and membership operations are owned by [Cryptography — Keys](/design/cryptography/keys/) and [MLS](/design/cryptography/mls/), and its server-side storage shape (rows, blob references, `protocol_version` pin) lives in the [Filesystem — Server](/design/filesystem/server/) Postgres schema. This section owns the *interaction surface* over that machinery. +- **Moves are idempotent.** Every asset lives in exactly one container; moving it is a signed lifecycle action naming `(asset, target album, epoch)`. Replaying a move finds the target state already in place and no-ops; concurrent moves resolve through the MLS commit order — see the [grouping-convergence requirement](/design/metadata/#grouping-convergence-requirement). - **Membership and roles.** Each member holds one of the album's three capabilities — read (AMK only), write (AMK + write-tier key), or admin (also the admin-tier key) — delivered over MLS to that member's devices ([Keys — Album Master Keys](/design/cryptography/keys/#album-master-keys-amks)). A role change is an MLS commit and bumps the AMK epoch. - **Invitation and join.** An admin invites a user by fetching and verifying their [device directory](/design/cryptography/keys/#device-directory) and issuing an MLS `Add` for all their devices; the `Welcome` delivers the AMK range set by the album's `history_policy` ([MLS — History Delivery](/design/cryptography/mls/#history-delivery-for-new-joiners)). Inviting a user on another home server also issues a [federation capability](/design/federation/#federation-capabilities); inviting a non-account recipient uses a [share link](/design/share-links/). Joining is acceptance of the `Welcome`; leaving or removal is an MLS `Remove` + epoch bump. (The MLS membership operations this surface invokes are blocked upstream — see the [MLS status note](/design/cryptography/mls/).) - **Album-level policy** — `history_policy`, the `protocol_version` pin, and the default `retention_until` — is fixed at creation and changed only through an [album upgrade ceremony](/design/versioning/#album-upgrade-ceremony), never ad hoc. @@ -46,11 +47,11 @@ View albums are organizational surfaces computed entirely client-side over the a Related files often belong together — RAW+JPEG pairs, bursts, a video and its external audio track. Rather than clutter the library with near-identical entries, Capsule groups them into one stack via best-effort auto-detection. -**Stacking is metadata-only.** A stack edit modifies the `stack_membership` field of each member asset's sidecar and emits a `metadata-update` provenance record per affected asset. It **never** deletes, rewrites, or merges the underlying asset bytes — even a "best photo" choice within a burst is just the `role = primary` pointer in metadata, not a destructive operation. A buggy or malicious stack edit therefore cannot lose original bytes. The full atomicity rule (stage all `.tmp` files, rename together, discard on any rename failure) lives in [Filesystem — Atomic Writes](/design/filesystem/maintenance/#atomic-writes-and-crash-recovery) and [Threat Model — Atomicity Invariants](/design/threat-model/validation/#atomicity-invariants). +**Stacking is metadata-only.** A stack edit modifies the `stack_membership` field of each member asset's sidecar — an LWW register over `Option` (leaving a stack is a stamped `None`), so concurrent stack edits from different devices converge order-independently under the [grouping-convergence requirement](/design/metadata/#grouping-convergence-requirement) — and emits a `metadata-update` provenance record per affected asset. It **never** deletes, rewrites, or merges the underlying asset bytes — even a "best photo" choice within a burst is just the `role = primary` pointer in metadata, not a destructive operation. A buggy or malicious stack edit therefore cannot lose original bytes. The full atomicity rule (stage all `.tmp` files, rename together, discard on any rename failure) lives in [Filesystem — Atomic Writes](/design/filesystem/maintenance/#atomic-writes-and-crash-recovery) and [Threat Model — Atomicity Invariants](/design/threat-model/validation/#atomicity-invariants). ### Stack Membership Schema -The `stack_membership` field on each member sidecar carries: +The `stack_membership` register's value on each member sidecar carries: ```rust StackMembership { From 693e862905ccd4b005eea178e8b2e87af17657e3 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:53:26 -0400 Subject: [PATCH 24/58] feat(core): add cull and hidden flags to the signed sidecar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two asset attributes the schema could not express: - cull: the photographer's trinary review flag (pick | neutral | reject), deliberately separate from the numeric star rating — a reject can carry three stars, and tools that conflate the two force lossy workflows. The culling workflow (flag -> filter -> batch-act) is documented in organization.md; the review UX is indexed as a slice, but the schema and semantics freeze now so sidecars written today survive it unchanged. - hidden: excluded from default views (timeline/search/system views), visible only in a re-auth-gated Hidden view. View-layer only — never deletion or access control. Companion files (RAW+JPEG's JPEG, a Live Photo's video) stay handled by stack roles; hidden covers what stacking cannot express. Both are LWW registers with wire-absent defaults, following the exact key_mode/stack_membership discipline: existing sidecars encode byte-identically (regression-tested), concurrent edits converge under the grouping-convergence requirement, and round-trip + signing tests cover the set case. --- capsule-core/src/lifecycle.rs | 4 + capsule-core/src/metadata/export_policy.rs | 2 + capsule-core/src/sidecar/sidecar_v1.rs | 95 +++++++++++++++++++ .../src/content/docs/design/local-gallery.md | 2 +- .../src/content/docs/design/metadata.md | 9 +- .../src/content/docs/design/organization.md | 16 ++++ 6 files changed, 125 insertions(+), 3 deletions(-) diff --git a/capsule-core/src/lifecycle.rs b/capsule-core/src/lifecycle.rs index 4fe03890..61de9585 100644 --- a/capsule-core/src/lifecycle.rs +++ b/capsule-core/src/lifecycle.rs @@ -532,6 +532,8 @@ impl Workspace { caption: Default::default(), rating: Default::default(), stack_membership: Lww::new(), + cull: Lww::new(), + hidden: Lww::new(), camera_id: None, device_id: self.account.device.device_id, session_id: Uuid::now_v7(), @@ -818,6 +820,8 @@ impl Workspace { caption: Default::default(), rating: Default::default(), stack_membership: Lww::new(), + cull: Lww::new(), + hidden: Lww::new(), camera_id: None, device_id: head.core.created_by_device, session_id: Uuid::now_v7(), diff --git a/capsule-core/src/metadata/export_policy.rs b/capsule-core/src/metadata/export_policy.rs index 97905623..fe0bcfe3 100644 --- a/capsule-core/src/metadata/export_policy.rs +++ b/capsule-core/src/metadata/export_policy.rs @@ -82,6 +82,8 @@ mod tests { caption: Default::default(), rating: Default::default(), stack_membership: Default::default(), + cull: Default::default(), + hidden: Default::default(), camera_id: Some(CameraId { model: "iPhone 15 Pro".into(), serial: "SECRET-SERIAL".into(), diff --git a/capsule-core/src/sidecar/sidecar_v1.rs b/capsule-core/src/sidecar/sidecar_v1.rs index a3932c2d..a108f62e 100644 --- a/capsule-core/src/sidecar/sidecar_v1.rs +++ b/capsule-core/src/sidecar/sidecar_v1.rs @@ -112,6 +112,20 @@ pub struct StackMembership { pub member_index: Option, } +/// Trinary culling flag (owner: Asset Organization — Culling). `Neutral` is the +/// never-flagged state and the wire-absent default. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum CullFlag { + /// Marked as a keeper. + Pick, + /// Not yet culled either way (default). + #[default] + Neutral, + /// Marked for rejection (filtered out, candidate for batch delete). + Reject, +} + /// The signed CBOR sidecar v1. #[derive(Debug, Clone, PartialEq)] pub struct SidecarV1 { @@ -147,6 +161,12 @@ pub struct SidecarV1 { /// construction, not convention. Wire-absent default = the never-written /// register (distinct from an explicit removal, which is a stamped `None`). pub stack_membership: Lww>, + /// Culling flag: LWW register over the trinary [`CullFlag`]. Wire-absent + /// default = never flagged (`Neutral`). + pub cull: Lww, + /// Hidden flag: LWW register; hidden assets are excluded from default views + /// (owner: Asset Organization — Hidden Assets). Wire-absent default = visible. + pub hidden: Lww, /// Camera identifier (export-stripped). pub camera_id: Option, /// Importing device id (UUIDv4; export-stripped). @@ -216,6 +236,12 @@ impl SidecarV1 { if self.stack_membership != Lww::default() { put!("stack_membership", self.stack_membership); } + if self.cull != Lww::default() { + put!("cull", self.cull); + } + if self.hidden != Lww::default() { + put!("hidden", self.hidden); + } put_opt!("camera_id", self.camera_id); put!("device_id", self.device_id); put!("session_id", self.session_id); @@ -316,6 +342,14 @@ impl SidecarV1 { None | Some(Value::Null) => Lww::new(), Some(v) => from_value::>>(v)?, }; + let cull = match text.remove("cull") { + None | Some(Value::Null) => Lww::new(), + Some(v) => from_value::>(v)?, + }; + let hidden = match text.remove("hidden") { + None | Some(Value::Null) => Lww::new(), + Some(v) => from_value::>(v)?, + }; let camera_id = opt!("camera_id", CameraId); let device_id = req!("device_id", Uuid); let session_id = req!("session_id", Uuid); @@ -338,6 +372,8 @@ impl SidecarV1 { caption, rating, stack_membership, + cull, + hidden, camera_id, device_id, session_id, @@ -373,6 +409,8 @@ mod tests { caption: Lww::new(), rating: Lww::new(), stack_membership: Lww::new(), + cull: Lww::new(), + hidden: Lww::new(), camera_id: Some(CameraId { model: "iPhone 15 Pro".into(), serial: "ABC123".into(), @@ -532,4 +570,61 @@ mod tests { replay.merge(&a); assert_eq!(replay.get(), ab.get()); } + + #[test] + fn absent_cull_and_hidden_are_wire_absent_and_byte_stable() { + // Never-flagged assets omit both registers, so pre-existing sidecars + // encode byte-identically to before the fields existed. + let s = minimal(); + let bytes = s.to_canonical_vec(); + for needle in [b"cull".as_slice(), b"hidden".as_slice()] { + assert!( + !bytes.windows(needle.len()).any(|w| w == needle), + "unexpected wire presence" + ); + } + let back = SidecarV1::from_canonical_slice(&bytes, SIDECAR_SCHEMA_V1).unwrap(); + assert_eq!(back.cull, Lww::new()); + assert_eq!(back.hidden, Lww::new()); + } + + #[test] + fn cull_and_hidden_round_trip_and_survive_signing() { + let ik = HybridSigningKey::from_seed_bytes(&[1; 32], &[2; 32]); + let mut s = minimal(); + s.cull.set( + CullFlag::Reject, + "2026-05-31T12:00:00Z", + Uuid::from_u128(0xD1), + ); + s.hidden + .set(true, "2026-05-31T12:00:01Z", Uuid::from_u128(0xD1)); + s.sign(&ik); + let back = + SidecarV1::from_canonical_slice(&s.to_canonical_vec(), SIDECAR_SCHEMA_V1).unwrap(); + assert_eq!(back, s); + assert_eq!(back.cull.get(), Some(&CullFlag::Reject)); + assert_eq!(back.hidden.get(), Some(&true)); + assert!(back.verify(&ik.verifying_key())); + } + + #[test] + fn concurrent_cull_flags_converge() { + // Two devices flag the same asset concurrently: pick at 10:00, reject at + // 11:00 — both merge orders converge on the later reject. + let mut a: Lww = Lww::new(); + a.set(CullFlag::Pick, "2026-05-31T10:00:00Z", Uuid::from_u128(0xA)); + let mut b: Lww = Lww::new(); + b.set( + CullFlag::Reject, + "2026-05-31T11:00:00Z", + Uuid::from_u128(0xB), + ); + let mut ab = a.clone(); + ab.merge(&b); + let mut ba = b.clone(); + ba.merge(&a); + assert_eq!(ab.get(), ba.get()); + assert_eq!(ab.get(), Some(&CullFlag::Reject)); + } } diff --git a/capsule-docs/src/content/docs/design/local-gallery.md b/capsule-docs/src/content/docs/design/local-gallery.md index 995afb67..0c07a04b 100644 --- a/capsule-docs/src/content/docs/design/local-gallery.md +++ b/capsule-docs/src/content/docs/design/local-gallery.md @@ -10,7 +10,7 @@ The [offline/online divide principle](/design/principles/#principles) is the roo ## Functional Requirements -- **FR1 — Full gallery offline.** Browse (timeline, albums, stacks), search, filter, view full-resolution locally-held assets, and organize (albums, tags, ratings, [culling flags](/design/organization/), hidden) with zero connectivity. Every one of these reads and writes `library.sqlite` + the local blob store ([Filesystem — Client](/design/filesystem/client/)); none has a server dependency by construction ([rich queries have no server surface at all](/design/api-surfaces/#surface--transport-map)). +- **FR1 — Full gallery offline.** Browse (timeline, albums, stacks), search, filter, view full-resolution locally-held assets, and organize (albums, tags, ratings, [culling flags](/design/organization/#culling), hidden) with zero connectivity. Every one of these reads and writes `library.sqlite` + the local blob store ([Filesystem — Client](/design/filesystem/client/)); none has a server dependency by construction ([rich queries have no server surface at all](/design/api-surfaces/#surface--transport-map)). - **FR2 — Never-signed-in is a valid mode.** A user who installs a native app and never connects a server has a fully functional local photo library: import, organize, search, export. Sync is an *addition* to a working product, not its precondition. (The crypto data plane operates locally; enrollment happens when — if ever — a server is added.) - **FR3 — Import works offline.** The scan → plan → execute [import pipeline](/design/import/pipeline/) is entirely local; upload is a separate, later concern. (The one deliberate exception: [streaming import-upload mode](/design/import/pipeline/#import-upload-streaming-mode) exists *because* the device cannot hold the bytes — it requires a server by definition.) - **FR4 — Degradation is visible, never structural.** When connectivity exists but a *remote-only* representation is unavailable, the asset stays listed with its best local representation and a non-destructive unavailability state ([the degrade ladder](/design/import/download-sync/#tiered-on-demand-fetch)). No offline condition ever removes an asset, an album, or an index entry. diff --git a/capsule-docs/src/content/docs/design/metadata.md b/capsule-docs/src/content/docs/design/metadata.md index 71592ad4..563f9e5e 100644 --- a/capsule-docs/src/content/docs/design/metadata.md +++ b/capsule-docs/src/content/docs/design/metadata.md @@ -38,6 +38,11 @@ SidecarV1 { // when never written, so stack edits converge like caption/rating. stack_membership: Lww>, + // organization — culling + visibility (semantics owned by Asset Organization). + // LWW registers, wire-absent when never written (never-flagged / visible). + cull: Lww, // pick | neutral | reject + hidden: Lww, + // identifiers (see Identifiers below; privacy-on-export rules apply) camera_id: Option<{ model: String, serial: String }>, device_id: UUIDv4, @@ -154,7 +159,7 @@ Capsule's *own* devices syncing the *same user's* library do **not** trigger thi User-editable metadata on a shared album — tags, captions, ratings — can be edited concurrently on different devices, including offline. To make these merges deterministic, such fields are modelled as CRDTs: - **Tags:** an OR-set (observed-remove set) with explicit [`add_id` binding](#add-id-binding), so a tag added on one device and removed on another converge predictably, and a remove that targets an unknown `add_id` is rejected rather than treated as a no-op. -- **Single-value fields** (`caption_lww`, `rating_lww`, `stack_membership`): last-writer-wins registers keyed by a signed timestamp and the writing `device_id` as the lexicographic tiebreaker. `stack_membership`'s value domain includes "no membership" (a stamped `None`), so joining, moving between, and leaving stacks are all the same LWW write and converge identically. +- **Single-value fields** (`caption_lww`, `rating_lww`, `stack_membership`, `cull`, `hidden`): last-writer-wins registers keyed by a signed timestamp and the writing `device_id` as the lexicographic tiebreaker. `stack_membership`'s value domain includes "no membership" (a stamped `None`), so joining, moving between, and leaving stacks are all the same LWW write and converge identically. ### Surfacing Concurrent Edits @@ -180,7 +185,7 @@ The same encrypted-operation path also carries the per-owner **library-settings | Structure | Mechanism | Why it converges | | --- | --- | --- | | Tags (`tags_user` / `tags_ai`) | OR-set | Add/remove keyed by `add_id`; merges commutative, associative, idempotent | -| Caption / rating / `stack_membership` | LWW register | Total order on `(ts, device_id)`; replay of any op is a no-op | +| Caption / rating / `stack_membership` / `cull` / `hidden` | LWW register | Total order on `(ts, device_id)`; replay of any op is a no-op | | Smart albums, people clusters, [aggregated federated albums](/design/federation/) | Computed views | Nothing stored — membership is a deterministic function of inputs; recomputation is idempotent by definition ([views](/design/organization/#system--smart-albums-views), [AI determinism](/design/ai/#ai-output-containment)) | | Container-album membership | Single home + ordered lifecycle ops | Exactly one container per asset; a move is a signed lifecycle action whose replay finds the target state already in place and no-ops ([Organization](/design/organization/#container-albums)); concurrency is resolved by MLS commit order, below | diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index 759adefa..eab21305 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -87,6 +87,22 @@ StackMembership { - **Chaptered Video:** Action cameras (like GoPro) often split long recordings into 4GB chunks. Files like `GOPR001.mp4` and `GOPR002.mp4` are stacked so they appear as one continuous video. - **Dual-System Audio:** Groups video files with high-quality external audio (WAV/AIFF) using timecode or waveform matching. +## Culling + +Culling is the review pass photographers make after a shoot: keep, undecided, toss. Capsule models it as a **trinary flag per asset** — `pick | neutral | reject` — stored in the sidecar's `cull` LWW register ([schema](/design/metadata/#sidecar-schema-v1)); `neutral` is the never-flagged default and is wire-absent. The flag is orthogonal to the numeric star `rating` (a reject can carry three stars; tools that conflate them force lossy workflows). + +- **Workflow.** Flag during review (single keystroke/swipe per asset), filter the view by flag, then act: batch-move rejects to [trash](#recycling) (the only destructive step, and it is soft-per-retention like any delete), promote picks into albums or shares. Flagging itself never touches bytes and is fully reversible. +- **Groups.** A stack or burst has no stored flag of its own — a group's cull state is **derived** from its members (all-rejected, any-pick, else mixed), so there is no second source of truth to diverge. Flagging a collapsed stack applies the flag to each member (one `metadata-update` per member, atomically staged like any [stack edit](#asset-stacking)). +- **Sync.** Like every LWW field, concurrent flags converge under the [grouping-convergence requirement](/design/metadata/#grouping-convergence-requirement). + +The dedicated culling UX (keyboard-driven review mode, reject-sweep) is client work tracked as its own slice in the repo-root `SLICES.md`; the schema and semantics above are frozen now so sidecars written today survive it unchanged. + +## Hidden Assets + +Every asset carries a `hidden` flag (sidecar LWW register, wire-absent default = visible). A hidden asset is **excluded from default views** — timeline, search results, and system views — and appears only in the dedicated Hidden view, which sits behind the same fresh-local-auth gate as Recently Deleted ([Local Gallery — SR1](/design/local-gallery/)). Hiding is view-layer only: the asset stays in its container album, keeps syncing, and remains reachable from contexts that reference it directly (its stack, a share it was already part of). + +Hiding is for "don't surface this" (a document photo, an awkward duplicate that must stay); it is not deletion and not access control. Sidecar-style companion files (the JPEG half of a RAW+JPEG pair, a Live Photo's video) are already suppressed from default views by their stack `role` — collapsed stacks show only the `primary` — so `hidden` is not needed for them; it exists for the cases stacking cannot express. + ## Recycling When you delete an asset, it defaults to trash (i.e. soft delete). On sync, new items in trash are essentially a metadata update rather than removal. A true "delete" operation is only performed when the user explicitly empties the trash, the asset has been in the trash for its full retention period, or the user requests immediate deletion. From 88199a8009eb2d982b5cf3f08bb85cf83d6c0be0 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 19:58:51 -0400 Subject: [PATCH 25/58] docs(design): staged uploads, network resilience, and background-sync contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three sync-domain gaps closed, one commit because they interlock: - Staged uploads (download-sync.md): upload-direction tiering for low-data situations — traveling on a metered plan, the index of what exists escapes first (manifest+metadata w/ LQIP), then previews, then originals on Wi-Fi. If the phone drowns, the user knows exactly what was lost and holds a preview of it. The policy is deliberately CLIENT-SIDE SESSION ORDERING ONLY: same sessions, same bundle mechanics, zero server mode branches — the enumerated mode exists in the structure while both policies share one code path. The one contract change: visibility flips on manifest+metadata (was original+metadata), with original_held carried per sync entry and the derived awaiting-original state (transient error.blob.pending_upload, never 410; GC carve-out; verify-before-destroy untouched — nothing releases a local original until T2 is durable; staged x streaming rejected by the planner). 'Backup' is deliberately not the name — that term stays reserved for the export artifact. - networking.md (new owner doc): the connection-class taxonomy the docs kept naming but never defined (unmetered/metered/constrained/adverse/ offline, with behavioral promotion to adverse — no OS API reports the networks that need it most), three retry policy classes that existing per-doc ladders become instances of, and the adverse-network posture: assume mid-transfer resets and black-holing (explicitly the steady state for paths crossing China's GFW toward servers outside the mainland, HK included), short independent resumable requests, the sync feed stays contractually unary short-poll, stall-detection over total timeouts, bounded windows under adverse, LAN peering as the degraded-mode alternative, and in-region self-hosting named as the structural remedy — resilience, not circumvention. - Background execution (download-sync.md): the OS contracts auto-sync actually runs under (BGAppRefreshTask/BGProcessingTask budgets, WorkManager constraints + Doze, desktop self-throttling) with one uniform rule: every window is preemptible, losing at most one chunk, because resume-from-server-truth makes 'must finish' state structurally absent. Contract seams: original_held on the sync proto (S-B4-tagged), UploadPolicy/UploadTier in capsule-core, ConnectionClass/RetryClass in capsule-sdk. S-C1/S-C2 contract-text amendments ride the SLICES commit. --- .../sync/proto/capsule/sync/v1/sync.proto | 6 ++ capsule-core/src/import/upload.rs | 31 ++++++++++ capsule-docs/astro.config.mjs | 1 + .../content/docs/design/filesystem/client.md | 2 +- .../docs/design/import/download-sync.md | 57 +++++++++++++++++-- .../content/docs/design/import/pipeline.md | 2 +- .../docs/design/import/upload-protocol.md | 4 +- .../src/content/docs/design/networking.md | 50 ++++++++++++++++ .../src/content/docs/design/principles.md | 2 + capsule-docs/src/content/docs/design/quota.md | 1 + capsule-sdk/src/lib.rs | 1 + capsule-sdk/src/net.rs | 39 +++++++++++++ 12 files changed, 188 insertions(+), 8 deletions(-) create mode 100644 capsule-docs/src/content/docs/design/networking.md create mode 100644 capsule-sdk/src/net.rs diff --git a/capsule-api/sync/proto/capsule/sync/v1/sync.proto b/capsule-api/sync/proto/capsule/sync/v1/sync.proto index 7f938469..61b99b4f 100644 --- a/capsule-api/sync/proto/capsule/sync/v1/sync.proto +++ b/capsule-api/sync/proto/capsule/sync/v1/sync.proto @@ -64,6 +64,12 @@ message SyncEntry { bytes metadata_blob = 7; // Content addresses of the asset's blobs by role — never blob bytes. BlobManifest blobs = 8; + // Whether the original blob is finalized on the server (S-B4, staged uploads: + // visibility flips on manifest+metadata, so an entry can precede its original — + // the derived `awaiting-original` state; download-sync design doc). proto3 + // absent-default is false, so clients read this only at/after the protocol + // date that introduces it. + bool original_held = 9; } message BlobManifest { diff --git a/capsule-core/src/import/upload.rs b/capsule-core/src/import/upload.rs index 99bd973b..dca17bad 100644 --- a/capsule-core/src/import/upload.rs +++ b/capsule-core/src/import/upload.rs @@ -8,6 +8,37 @@ use std::{ use crate::import::ImportExecutionPlan; +/// Per-device upload policy (contract type for slice `S-B4`, staged uploads; +/// SSoT: download-sync design doc, "Upload Tiering (Staged Uploads)"). +/// +/// The policy is client-side **session ordering only** — the server has zero +/// mode branches. Under [`UploadPolicy::Staged`] the scheduler opens each +/// asset's sessions in [`UploadTier`] order, gating each tier on the sync +/// connection criteria; under [`UploadPolicy::Full`] all sessions open eagerly. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum UploadPolicy { + /// Every session of an asset's bundle opens eagerly (default). + #[default] + Full, + /// Sessions open in tier order (index → preview → original), each tier + /// gated by connection class. Mutually exclusive with streaming import. + Staged, +} + +/// The upload tier ladder, mirroring the download ladder. Tiers map onto +/// existing blob roles — no new blob kind exists for staging. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum UploadTier { + /// T0: signed manifest + metadata blob (embedded LQIP) — the index that + /// makes the asset visible (`awaiting-original`) on other devices. + Index, + /// T1: thumbnail + preview derivative blobs. + Preview, + /// T2: the original blob; its finalization flips `original_held` on the + /// sync feed and unlocks every release path (verify-before-destroy). + Original, +} + pub struct UploadExecutionPlan(pub Vec); pub struct UploadPriorityConfig { diff --git a/capsule-docs/astro.config.mjs b/capsule-docs/astro.config.mjs index 1394b558..a8f4404c 100644 --- a/capsule-docs/astro.config.mjs +++ b/capsule-docs/astro.config.mjs @@ -53,6 +53,7 @@ export default defineConfig({ { slug: 'design/principles' }, { slug: 'design/module-map' }, { slug: 'design/api-surfaces' }, + { slug: 'design/networking' }, ], }, { diff --git a/capsule-docs/src/content/docs/design/filesystem/client.md b/capsule-docs/src/content/docs/design/filesystem/client.md index 1e9b0b3c..20eedbd2 100644 --- a/capsule-docs/src/content/docs/design/filesystem/client.md +++ b/capsule-docs/src/content/docs/design/filesystem/client.md @@ -61,7 +61,7 @@ What is eligible for reclamation is exactly the rebuildable-or-refetchable set: ### Automatic cache management -Implemented (issue #23): last-access tracking lives in the `cached_representations` table in `capsule-core::db`, and the sweep is `capsule-core::library::cache_sweep`. The connection-class detection that would *drive* the byte budget from `capsule-sdk` is planned — the budget is a plain parameter today. +Implemented (issue #23): last-access tracking lives in the `cached_representations` table in `capsule-core::db`, and the sweep is `capsule-core::library::cache_sweep`. The [connection-class detection](/design/networking/#connection-classes) that would *drive* the byte budget from `capsule-sdk` is planned — the budget is a plain parameter today. The reclaimable set is held within a **user-configurable cache budget**. When it grows past budget — typically while browsing a large library on a device that cannot hold everything — Capsule reclaims space itself rather than waiting for the user or letting the OS decide: diff --git a/capsule-docs/src/content/docs/design/import/download-sync.md b/capsule-docs/src/content/docs/design/import/download-sync.md index ca11c018..74ae1884 100644 --- a/capsule-docs/src/content/docs/design/import/download-sync.md +++ b/capsule-docs/src/content/docs/design/import/download-sync.md @@ -6,7 +6,7 @@ status: draft Download is the inverse of [upload](/design/import/upload-protocol/), and rests on the same two foundations: blobs are **content-addressed by ciphertext hash**, and the server never holds a key, so it serves only opaque ciphertext. Where the upload path optimises for correctness under interruption, the download path optimises for **bandwidth and storage frugality** — a client fetches the smallest representation that satisfies the user's current intent, and nothing more. -The download client lives in `capsule-sdk` (per-platform glue handles cache placement and connection-class detection; planned). Server-side, the two surfaces ride different transports per the [API surface map](/design/api-surfaces/#surface--transport-map): the **sync feed** is the gRPC `capsule.sync.v1.SyncService` in `capsule-api-sync` (a stub today), and **blob fetch** is a REST ranged `GET` served by `capsule-api-media`. The sync feed format is the **contract** other modules consume; its versioning and per-album monotonic ordering are what defeats the stale-rewind attack class. +The download client lives in `capsule-sdk` (per-platform glue handles cache placement and [connection-class detection](/design/networking/#connection-classes); planned). Server-side, the two surfaces ride different transports per the [API surface map](/design/api-surfaces/#surface--transport-map): the **sync feed** is the gRPC `capsule.sync.v1.SyncService` in `capsule-api-sync` (a stub today), and **blob fetch** is a REST ranged `GET` served by `capsule-api-media`. The sync feed format is the **contract** other modules consume; its versioning and per-album monotonic ordering are what defeats the stale-rewind attack class. ## Discovering What Changed @@ -69,26 +69,69 @@ Sync is checked conservatively. When a check fires, the client reconciles everyt The actual synchronization criteria are strict and scale with the reconciliation amount (i.e. total upload + download transfer): -- **Small reconciliation** — a handful of new assets, or metadata-only deltas: synced proactively whenever the device has any non-metered connection. +- **Small reconciliation** — a handful of new assets, or metadata-only deltas: synced proactively whenever the device has any non-metered [connection class](/design/networking/#connection-classes). - **Large reconciliation** — bulk uploads, or original-tier downloads: deferred until the device is connected to unmetered Wi-Fi. A storage-constrained [streaming import](/design/import/pipeline/#import-upload-streaming-mode) is a large reconciliation and obeys these same rules, pausing if the connection drops or becomes metered. ### Platform Limitations Auto sync is implemented **only** if it can be guaranteed to behave appropriately under all scenarios. It is explicitly not implemented on platforms that lack the APIs we need (e.g., detecting metered connections), to avoid surprises. +### Background Execution + +Mobile OSes do not let an app sync whenever it likes; the scheduler is written against the platform contracts, not around them: + +- **iOS.** Change detection and small reconciliations ride `BGAppRefreshTask` (short, OS-budgeted, no guaranteed cadence); large reconciliations request `BGProcessingTask` with `requiresNetworkConnectivity` (+ `requiresExternalPower` for very large batches). The OS may grant nothing for days — that is exactly the case the [two-week staleness notification](#notifications) exists for, and why it is a product surface rather than a bug. +- **Android.** All background sync rides `WorkManager` with explicit constraints (`UNMETERED` for large reconciliations, `CONNECTED` for small; battery-not-low), surviving Doze and app restarts. A *user-initiated* force-sync of a large reconciliation may run as a user-visible foreground service with progress; background work never does. +- **Desktop.** No OS budget applies; the scheduler self-throttles (idle + on-power gating for bulk work), reusing the [maintenance gating rules](/design/filesystem/maintenance/#content-validation-expensive-scheduled). +- **Uniform rules, all platforms.** Every background window is treated as *preemptible*: work is chunked so an OS kill mid-window loses at most one chunk (uploads resume via [HEAD offsets](/design/import/upload-protocol/#idempotency-and-resumption), downloads via `Range`); the scheduler never holds a wake lock across a transfer; and a window that ends mid-reconciliation simply leaves the remainder for the next window — there is no "must finish" state, by construction. Retry/backoff inside a window follows the [bulk-transfer policy class](/design/networking/#retry-policy-classes). + ### Notifications When the auto sync criteria have not been met for a prolonged period — **two weeks** specifically — the library falls silently out of date, which defeats the purpose of a backup. The client surfaces this rather than letting it pass unnoticed: -- After two weeks without a completed sync *while changes remain un-synced*, the user is notified that the library is behind and offered a one-tap **force sync now**, which proceeds regardless of the metered/Wi-Fi criteria with their explicit consent. (A device idle long enough to trigger this may also be approaching the session's [sliding inactivity expiry](/design/authentication/#sliding-inactivity-expiry); the force-sync flow routes through re-authentication when the session has lapsed rather than failing the sync.) +- After two weeks without a completed sync *while changes remain un-synced* — including originals still pending under a [staged upload policy](#upload-tiering-staged-uploads) — the user is notified that the library is behind and offered a one-tap **force sync now**, which proceeds regardless of the metered/Wi-Fi criteria with their explicit consent. (A device idle long enough to trigger this may also be approaching the session's [sliding inactivity expiry](/design/authentication/#sliding-inactivity-expiry); the force-sync flow routes through re-authentication when the session has lapsed rather than failing the sync.) - The notification can be **snoozed** until a later date (e.g. another two weeks) or **disabled** outright. Snoozing only suppresses the warning; disabling opts out of the warning entirely and does not affect auto sync itself. ## Synchronization Scope -- **Uploadable new content:** the source (original) asset is uploaded along with all associated metadata and derivatives. +- **Uploadable new content:** the source (original) asset is uploaded along with all associated metadata and derivatives — in the session order chosen by the [upload policy](#upload-tiering-staged-uploads) (`full` today's behavior, `staged` the low-data ladder). - **Modified/deleted content:** associated metadata is updated. - **Fetch new content:** depending on setting, metadata only / metadata + thumbnails / metadata + thumbnails + original is fetched for all new assets. Unless the original already exists locally (e.g., if the device was the original uploader), the original is only fetched on demand (e.g. the user explicitly views the original or shares the original with others). This is to save bandwidth and storage on client devices. Metadata includes LQIP which can be used as a preview before even thumbnails are fetched. +## Upload Tiering (Staged Uploads) + +Downloads have always been tiered; uploads were all-or-nothing. **Staged uploads** add the upload-direction ladder for low-data situations — traveling with a metered plan, weeks away from Wi-Fi — where what matters most is that the *index* of what exists escapes the device: if the phone drowns, the user knows exactly what was lost and holds a preview of it. (The name avoids "backup" deliberately — that term is reserved for the [encrypted export artifact](/design/backup-recovery/).) + +The per-device **upload policy** is a closed enum: + +- **`full`** (default) — every session of an asset's bundle opens eagerly, in any order (today's behavior). +- **`staged`** — sessions open in tier order per asset, each tier gated by the connection rules below. + +The **upload tier ladder** mirrors the download ladder and maps directly onto existing [blob roles](/design/import/upload-protocol/#what-gets-uploaded) — no new blob kind, no new wire surface: + +| Tier | Blobs (by role) | Opens when | +| --- | --- | --- | +| **T0 — index** | signed manifest + metadata blob (with embedded LQIP) | any usable connection, even `constrained`/`adverse` — a few KB per asset | +| **T1 — preview** | thumbnail + preview derivative blobs | any non-metered connection (small-reconciliation rule) | +| **T2 — original** | original blob | unmetered Wi-Fi (large-reconciliation rule) or explicit force-sync | + +**The policy is client-side session ordering only.** The server has zero mode branches: the same `POST /upload` sessions, the same bundle mechanics, the same finalization — under `staged` the client simply hasn't opened the T2 session yet. This is what keeps the two policies on one code path: the scheduler takes the tier ladder as an ordering input; nothing else in the pipeline knows the policy exists. + +**The `awaiting-original` state.** Visibility already flips on manifest + metadata finalization ([upload protocol](/design/import/upload-protocol/#what-gets-uploaded)); whether the original has landed travels as the derived per-asset fact `original_held` on each sync feed entry. An asset with `original_held = false` is in the derived state **awaiting-original**: + +- Other devices see it in the timeline immediately (LQIP, then T1 tiers as they land) with an "original still on *device*" badge. +- Fetching its original returns the transient `error.blob.pending_upload` — explicitly distinct from `410 Gone`; the client shows the badge, never a failure, and re-fetches when the feed flips `original_held`. +- Server GC and the index-rebuild rule treat a missing original on an `awaiting-original` asset as expected state, not a dangling reference ([Filesystem — Server](/design/filesystem/server/)). +- The state is always **derived** from the blob-role rows / feed field — never stored as a second source of truth. + +**What staged uploads never change:** + +- **Verify-before-destroy is untouched.** No release path — cache eviction of a device-owned original, Move-import source deletion, streaming release — may fire until the **original** is uploaded and [`/storage/verify`](/design/import/storage-verification/#verify-before-destroy) returns durable. A staged asset pins its local original until T2 completes, by the same gate that always governed release. +- **Staged and [streaming import](/design/import/pipeline/#import-upload-streaming-mode) are mutually exclusive per import.** Streaming exists to release local bytes quickly; staged defers exactly the upload that release depends on. The planner rejects the combination outright. +- **Quota** charges each tier's session at its own creation — the existing enforcement point, just later in time for T2. Deleting an `awaiting-original` asset cancels its pending tiers and tombstones normally. + +Resume needs no new client state: the tier queue is re-derived from server truth (held roles on the feed entry + `GET /upload/sessions` for in-flight tiers); `library.sqlite`'s work queue stays a rebuildable cache. + ## Validation - **Sync feed monotonicity (unit).** Server-side unit tests assert that every `sync_seq` advance over a given album is strictly increasing; concurrent writes are linearised by the same Postgres transaction that mints the new `sync_seq`. @@ -100,5 +143,11 @@ When the auto sync criteria have not been met for a prolonged period — **two w - **Resume after interrupt (smoke).** Start a large original fetch; interrupt mid-Range; resume; assert byte-identical result with no re-fetched bytes. - **Auto-sync state machine (smoke).** Simulate connectivity changes (Wi-Fi → metered → offline → Wi-Fi); assert the scheduler pauses, resumes, and respects the small/large threshold. - **Cross-asset dedup hit (unit).** Two assets with the same thumbnail hash; the second viewing must not refetch. +- **Staged ladder order (unit).** Under `staged`, sessions open strictly T0 → T1 → T2 per asset, T2 only under the large-reconciliation criteria; under `full`, eagerly. +- **awaiting-original semantics (unit).** Visibility flips at metadata finalization with `original_held = false`; flips true inside the T2 finalization transaction; a skeleton fetch surfaces the transient pending state and never removes metadata or the index entry; `pending` is distinguishable from `410`. +- **Staged release gate (unit).** Under `staged`, every release path refuses while T2 is not durable. +- **Staged resume from server truth (smoke).** Kill the client mid-ladder; on restart the tier queue re-derives from the feed + session list and re-uploads only missing tiers. +- **Planner staged × streaming exclusion (unit).** A plan configured with both is rejected at confirmation. +- **Background window preemption (smoke).** Kill a sync window mid-transfer; assert at most one chunk of progress is lost and the next window resumes from server truth. The cross-module case — server emits a sync entry → client applies and fetches blob — is bounded E2E surface listed in [Module Map](/design/module-map/#e2e-test-surface). diff --git a/capsule-docs/src/content/docs/design/import/pipeline.md b/capsule-docs/src/content/docs/design/import/pipeline.md index 5b0b1857..6eaae669 100644 --- a/capsule-docs/src/content/docs/design/import/pipeline.md +++ b/capsule-docs/src/content/docs/design/import/pipeline.md @@ -59,7 +59,7 @@ Step 1–3 can be parallelized across files. The executor is cancellation-aware: The default pipeline imports every file into the local library *before* upload, so the device temporarily holds the whole import on disk. That is impossible on a storage-constrained device — a laptop with a near-full SSD importing a library larger than its free space. **Streaming mode** removes the requirement that the full import ever land locally at once. -It is **auto-detected**: when the [free-space probe](#plan--confirm) reports `total_size` is near or over available space, the plan is marked `streaming_recommended` and the user confirms a streaming import. Instead of executing all files and then uploading, the executor runs a bounded **sliding window**: +It is **auto-detected**: when the [free-space probe](#plan--confirm) reports `total_size` is near or over available space, the plan is marked `streaming_recommended` and the user confirms a streaming import. Streaming is **mutually exclusive with a [staged upload policy](/design/import/download-sync/#upload-tiering-staged-uploads)** for the same import — streaming exists to release local bytes quickly, staged defers exactly the upload that release depends on — and the planner rejects the combination at confirmation. Instead of executing all files and then uploading, the executor runs a bounded **sliding window**: 1. Import the next file (or a small window of files) into the library — encrypt, sign the [manifest](/design/cryptography/provenance/#asset-manifest), generate derivatives — exactly as the normal [Execute](#execute) step. 2. Upload its bundle via the [upload protocol](/design/import/upload-protocol/). diff --git a/capsule-docs/src/content/docs/design/import/upload-protocol.md b/capsule-docs/src/content/docs/design/import/upload-protocol.md index abc60956..3dcc6a6a 100644 --- a/capsule-docs/src/content/docs/design/import/upload-protocol.md +++ b/capsule-docs/src/content/docs/design/import/upload-protocol.md @@ -20,7 +20,7 @@ An asset is never uploaded as a single plaintext file. Because Capsule is end-to - **Derivative blobs** — thumbnails and previews, generated client-side during import (see [Thumbnails](/design/thumbnails/)), each encrypted independently. - The **metadata blob** — the CBOR metadata document (capture date, dimensions, EXIF-derived fields, the [LQIP](/design/thumbnails/#lqip), provenance), encrypted under the [bulk AEAD](/design/cryptography/primitives/#bulk-aead) (see [Metadata](/design/metadata/)). -Each blob is its own upload with its own upload ID; the protocol does not couple them and imposes **no wire ordering**. Every session declares its blob's **role** at creation — a closed enum `original | derivative | metadata | provenance | backup` in the `POST /upload` body, recorded on the pending row — so the server can reason about bundle completeness without reading plaintext. The client may transfer the bundle in any order — decoupling lets small derivatives land while a large original is still in flight — and the server **gates visibility** on the pending-asset row: the asset becomes visible to other devices once its **manifest and metadata blob are finalized**. Whether the original is also held yet travels as a per-asset completeness fact on the sync feed (`original_held`), which is what makes [staged uploads](/design/import/download-sync/) possible; an asset whose original has not landed is in the derived `awaiting-original` state, and fetching its original returns the transient `error.blob.pending_upload` rather than `410`. Every blob in the bundle — original, derivatives, metadata, provenance — counts toward the uploader's storage [quota](/design/quota/#accounting-model). +Each blob is its own upload with its own upload ID; the protocol does not couple them and imposes **no wire ordering**. Every session declares its blob's **role** at creation — a closed enum `original | derivative | metadata | provenance | backup` in the `POST /upload` body, recorded on the pending row — so the server can reason about bundle completeness without reading plaintext. The client may transfer the bundle in any order — decoupling lets small derivatives land while a large original is still in flight — and the server **gates visibility** on the pending-asset row: the asset becomes visible to other devices once its **manifest and metadata blob are finalized**. Whether the original is also held yet travels as a per-asset completeness fact on the sync feed (`original_held`), which is what makes [staged uploads](/design/import/download-sync/#upload-tiering-staged-uploads) possible; an asset whose original has not landed is in the derived `awaiting-original` state, and fetching its original returns the transient `error.blob.pending_upload` rather than `410`. Every blob in the bundle — original, derivatives, metadata, provenance — counts toward the uploader's storage [quota](/design/quota/#accounting-model). "Blob" is defined once, in [Filesystem — Server: Uniform, Opaque Blobs](/design/filesystem/server/#uniform-opaque-blobs); this protocol is its transport, not its definition. Every asset and derivative blob carries a signed [manifest envelope](/design/cryptography/provenance/#asset-manifest): at `POST /upload` the server validates the envelope's `created_by_device` against the uploader's [device directory](/design/cryptography/keys/#device-directory) (invariant 7), and the client verifies the full write-tier signature on download via [`verify_asset`](/design/cryptography/keys/#write-authorization). **Backup artifacts are the one exception** — they carry no per-asset provenance of their own (the exporting device is not the original author); their integrity rides the library-level backup MANIFEST instead (see [Backup and Recovery](/design/backup-recovery/)). @@ -192,7 +192,7 @@ The client algorithm is normative for `capsule-sdk` (another client may adapt di - Clamp to the tier's range; every tier range sits inside the protocol bounds `[256 KiB, 16 MiB]` for non-final chunks. - Every candidate size is a 4 KiB multiple **by construction** (the candidate set is doublings/halvings of 4 KiB-aligned tier bounds), so alignment never depends on a runtime check. -The rationale is a direct trade-off — chunks that are too small waste round-trips, while chunks that are too large waste re-transmission on a flaky link and pin more memory per in-flight request. Under an [adverse connection](/design/import/download-sync/), the conservative choice is the tier minimum; the client must never let adaptation regress effective throughput. +The rationale is a direct trade-off — chunks that are too small waste round-trips, while chunks that are too large waste re-transmission on a flaky link and pin more memory per in-flight request. Under an [adverse connection](/design/networking/#connection-classes), the conservative choice is the tier minimum; the client must never let adaptation regress effective throughput. We deliberately do **not** expose per-blob upload *ordering* as a protocol concern. Concurrent sessions plus the OS and TCP stack settle ordering naturally; see [Pipeline — Upload Prioritization](/design/import/pipeline/#upload-prioritization) for the client-side heuristics that decide which assets to *start*. diff --git a/capsule-docs/src/content/docs/design/networking.md b/capsule-docs/src/content/docs/design/networking.md new file mode 100644 index 00000000..2f44bafe --- /dev/null +++ b/capsule-docs/src/content/docs/design/networking.md @@ -0,0 +1,50 @@ +--- +title: Network Resilience +description: The connection-class taxonomy, retry policy classes, and the adverse-network posture every transfer surface builds on +status: draft +--- + +Capsule clients run on phones with metered plans, hotel Wi-Fi, and networks that silently drop long-lived connections. This doc owns the three cross-cutting primitives every transfer surface consumes: the **connection-class taxonomy** (the input to sync criteria and cache budgets), the **retry policy classes** (so per-doc retry ladders are instances of a shared shape, not reinventions), and the **adverse-network posture** (the design assumptions that keep Capsule usable on hostile networks). Protocol mechanics stay owned by their protocol docs — [upload](/design/import/upload-protocol/), [download & sync](/design/import/download-sync/), [peering](/design/peering/), [federation](/design/federation/); this doc owns only the shared taxonomy and posture. + +## Connection Classes + +A closed enum, evaluated continuously on-device: + +| Class | Meaning | Detection inputs | +| --- | --- | --- | +| `unmetered` | Bulk transfer is acceptable | iOS `NWPathMonitor` (`isExpensive == false`); Android `NET_CAPABILITY_NOT_METERED`; desktop default | +| `metered` | Byte-counted link; bulk deferred | `isExpensive`; metered capability absent | +| `constrained` | OS-level data saver active | iOS `isConstrained` (Low Data Mode); Android Data Saver | +| `adverse` | Connectivity is present but unreliable | **Behavioral promotion**: ≥ N mid-transfer resets, stalls, or black-holes within a sliding window (thresholds are client-tunable, not protocol surface); demoted after a clean period | +| `offline` | No usable path | OS path state | + +Consumers: the [sync criteria](/design/import/download-sync/#synchronization-criteria) (small vs. large reconciliation), the [staged-upload tier gates](/design/import/download-sync/#upload-tiering-staged-uploads), the [cache-eviction byte budget](/design/filesystem/client/#automatic-cache-management), [prefetch](/design/import/download-sync/#prefetch-and-frugality), and [adaptive chunk sizing](/design/import/upload-protocol/#adaptive-chunk-sizing) (under `adverse`, the conservative choice is the tier minimum). `adverse` is deliberately behavioral — no OS reports it, and the networks that need it most (see below) look "connected" to every OS API. + +## Retry Policy Classes + +Three named classes; every retrying surface declares which one it instantiates. Universal rules: exponential backoff **with full jitter**, always; honor server backoff signals (`Retry-After`, 429/503); every retry loop has a bounded give-up that surfaces a user-visible state; no surface ever hot-loops. + +| Class | Shape | Instances (owned where they are) | +| --- | --- | --- | +| `interactive` | short timeout, ≤ 2 retries, then a visible failure state | auth flows, on-demand tier fetches | +| `bulk-transfer` | resume-first (HEAD offset / `Range`), patient within the session's lifetime, backoff between attempts | [upload resume](/design/import/upload-protocol/#idempotency-and-resumption), [download resume](/design/import/download-sync/#resumption-and-verification), [transient fetch retry](/design/import/download-sync/#tiered-on-demand-fetch) | +| `control/ceremony` | slow ladder, long horizon, never abandons silently | [MLS recovery ladder](/design/mls-resilience/) (30 s → 2 min → 10 min), [federation circuit breaker](/design/federation/) (5/30/60 min) | + +## Adverse-Network Posture + +Some networks — congested carrier links, captive-portal hotels, and notably paths crossing China's Great Firewall — exhibit connectivity that no error code explains: connections reset mid-transfer, long-lived streams die silently after minutes, throughput collapses unpredictably, and the same request succeeds on retry. A server outside mainland China (Hong Kong included) serving a user inside it should expect all of the above as the steady state, not an incident. Capsule's answer is **resilience by shape, not workarounds**: the protocols are built so that hostile paths degrade throughput, never correctness. + +- **Assume mid-transfer resets and silent black-holing.** Every transfer is short, independent, and resumable; losing any single request loses at most one chunk/window of progress. This already falls out of the upload protocol (chunked, offset-resumable) and ranged blob fetch — stated here as a *requirement* on any future surface. +- **No long-lived stream is ever required for correctness.** The sync feed is a unary, paged `Sync(cursor, page_size)` call — short-poll by construction. Any future push/streaming optimization MUST keep the unary equivalent as the fallback path, because long-lived connections are precisely what unreliable middleboxes kill. +- **Stall detection over total timeouts.** Bulk transfers cut on *no-bytes-for-T-seconds*, not on a total-duration timeout (which either kills slow-but-live transfers or waits forever on dead ones), then resume from the offset. A stall counts toward `adverse` promotion. +- **Bounded transfer windows under `adverse`.** Blob fetches shrink to bounded `Range` windows and uploads sit at the chunk-size tier minimum, so each request is small enough to usually complete between resets. +- **Connection racing at dial only.** Happy Eyeballs v2 (parallel IPv6/IPv4 dial) is cheap and standard. Racing whole *requests* across paths is not done in v1 — it doubles server load for marginal gain. +- **LAN peering is the degraded-mode alternative.** When the WAN path is persistently adverse, [peering](/design/peering/) moves originals between same-user devices without touching it; the staged-upload T0 index still escapes over the thin link. +- **The structural remedy is locality.** Resilience keeps Capsule *usable* across a hostile path; it cannot make one fast. For a persistently degraded region, the answer is self-hosting in-region — Capsule is built for that — not circumvention machinery in the client, which is out of scope. + +## Validation + +- **Class detection (unit).** Mocked OS signals per platform map to the expected class; `constrained` and `metered` are distinguished. +- **Adverse promotion/demotion (unit).** An injected reset/stall pattern promotes to `adverse` and switches fetches to bounded windows; a clean period demotes. +- **Stall-cut-resume (smoke).** Black-hole a `Range` fetch mid-window; the stall detector cuts within its bound and the resume transfers zero duplicate bytes. +- **Backoff discipline (unit).** Every policy class's retry sequence stays within its bounds, jitters, and honors `Retry-After`; no configuration can produce an unbounded hot loop. diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 1d717a53..77608633 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -48,6 +48,8 @@ The owner docs are: | Backup artifact container + escrow + recovery mechanisms | [Backup and Recovery](/design/backup-recovery/) | | CRDT scheme, identifiers, geolocation, sidecar schema | [Metadata](/design/metadata/) | | Import pipeline (scan, plan, execute) | [Import — Pipeline](/design/import/pipeline/) | +| Connection classes + retry policy classes + adverse-network posture | [Network Resilience](/design/networking/) | +| Offline-first requirement contract (local-gallery FRs/NFRs) | [Local Gallery](/design/local-gallery/) | | Upload protocol (wire, sessions, finalization) | [Import — Upload Protocol](/design/import/upload-protocol/) | | Download, sync feed, tiered fetch, auto-sync | [Import — Download & Sync](/design/import/download-sync/) | | Federation trust model, capability tokens, soft-fail | [Federation](/design/federation/) | diff --git a/capsule-docs/src/content/docs/design/quota.md b/capsule-docs/src/content/docs/design/quota.md index 7f36cc7a..5f903c38 100644 --- a/capsule-docs/src/content/docs/design/quota.md +++ b/capsule-docs/src/content/docs/design/quota.md @@ -46,6 +46,7 @@ Where the quota check actually runs: - **At [`POST /upload`](/design/import/upload-protocol/#endpoints) session creation.** The server computes `quota_used(upload_user_id) + declared_size` and rejects with `403 Quota Exceeded` (or similar structural code) if it crosses the hard limit. This is the *only* hard enforcement point — once a session is open, the declared size is the cap, and the session is allowed to complete. - **At session cancellation.** When a session is cancelled or expires, the reserved-but-uncommitted bytes are released; the next quota check sees the new (lower) usage. - **At [finalization](/design/import/upload-protocol/#finalization-and-integrity).** Cumulative size is bounded by the declared size at chunk acceptance; no separate quota check at finalization is needed because the declared size was already approved at session creation. +- **[Staged uploads](/design/import/download-sync/#upload-tiering-staged-uploads) need no special case.** Each tier's session is charged at its own creation — the same single enforcement point, just later in time for the original's session. - **At metadata-update writes.** A metadata-update creates a new encrypted metadata blob; the size delta is checked against quota. Tiny but non-zero. ## Scope Decisions diff --git a/capsule-sdk/src/lib.rs b/capsule-sdk/src/lib.rs index 01ec0021..5ee8ef64 100644 --- a/capsule-sdk/src/lib.rs +++ b/capsule-sdk/src/lib.rs @@ -11,6 +11,7 @@ //! surfaces below (the adaptive upload strategy in [`upload`]) stand alone, and //! the generated-client wrapper is parked below. +pub mod net; pub mod upload; // ─── Parked until spargen (S-D8) ──────────────────────────────────────────── diff --git a/capsule-sdk/src/net.rs b/capsule-sdk/src/net.rs new file mode 100644 index 00000000..ab679091 --- /dev/null +++ b/capsule-sdk/src/net.rs @@ -0,0 +1,39 @@ +//! Connection-class and retry-policy contract types (SSoT: the Network +//! Resilience design doc). Detection (OS signals + behavioral promotion to +//! [`ConnectionClass::Adverse`]) and the per-class retry engines land with +//! slices `S-D2` (class detection feeding sync/cache budgets) and `S-D10` +//! (adverse-network hardening) in the repo-root `SLICES.md`. + +/// The closed connection-class enum, evaluated continuously on-device. +/// Consumers: sync criteria (small/large reconciliation), staged-upload tier +/// gates, the cache-eviction byte budget, prefetch, adaptive chunk sizing. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ConnectionClass { + /// Bulk transfer is acceptable. + Unmetered, + /// Byte-counted link; bulk work is deferred. + Metered, + /// OS-level data saver (iOS Low Data Mode / Android Data Saver). + Constrained, + /// Connectivity present but unreliable — behavioral promotion on repeated + /// mid-transfer resets/stalls within a sliding window; demoted after a + /// clean period. No OS reports this class; it is derived. + Adverse, + /// No usable path. + Offline, +} + +/// The three retry policy classes every retrying surface instantiates. +/// Universal rules (owned by the Network Resilience doc): exponential backoff +/// with full jitter, honor server backoff signals, bounded give-up surfaced to +/// the user, never hot-loop. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RetryClass { + /// Short timeout, ≤ 2 retries, then a visible failure state. + Interactive, + /// Resume-first (HEAD offset / Range), patient within the session lifetime. + BulkTransfer, + /// Slow ladder, long horizon, never abandons silently (MLS recovery, + /// federation circuit breaker are instances). + ControlCeremony, +} From 28aeb4d4c130d80153d6c30cdab72243c6ad9e7e Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 20:00:00 -0400 Subject: [PATCH 26/58] docs(design): federated sharing via aggregated albums MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The killer sharing feature — accounts on different servers contributing to one album — previously had no shipping story: true multi-writer cross- server albums are deferred to v2 (open question #1) and v1 federation is read-direction only. The aggregated album closes the gap without touching either constraint: N ordinary container albums (one per contributor, each single-writer on its own home server) rendered client-side as one logical album. Group membership travels as an encrypted album-group assertion inside each contributor's OWN album metadata — no shared mutable object, so no cross-server consensus — and inclusion is injection-proof by construction (a constituent renders only if you were invited to it AND it asserts the group id). Zero new server surface; servers never learn a group exists. Ordering is computed (capture time, asset-id tiebreak), consistent with the container-vs-view split and the grouping-convergence requirement. Honest limitation stated: no group-level kick — you can only unshare your own photos; shared governance is exactly the v2 problem, and this design collapses cleanly into a v2 multi-writer album if it ships. --- .../src/content/docs/design/federation.md | 40 ++++++++++++++++++- .../src/content/docs/design/metadata.md | 2 +- .../src/content/docs/design/organization.md | 2 +- .../docs/design/threat-model/schema-rules.md | 2 +- 4 files changed, 42 insertions(+), 4 deletions(-) diff --git a/capsule-docs/src/content/docs/design/federation.md b/capsule-docs/src/content/docs/design/federation.md index 4ba2d9ed..beeb45e5 100644 --- a/capsule-docs/src/content/docs/design/federation.md +++ b/capsule-docs/src/content/docs/design/federation.md @@ -42,7 +42,45 @@ For v1, **each album has exactly one home server** — the server that issued th This rule keeps the v1 federation API surface small (no replication, no cross-server commit ordering) and forecloses several damage classes — split-brain ownership, two-server delete races, conflicting AMK-epoch advances — that would otherwise need explicit cross-server consensus to prevent. -Cross-server replication of a *single* album (where two users on different home servers each want to write the same album) is **out of scope for v1** and deferred to v2. v1 supports cross-server sharing in the read direction (Alice on `home.tld` shares an album to Bob on `other.tld`; Bob reads via federation; Bob's writes either remain on `home.tld` via a registered or sponsored account, or are out of scope). The v2 design space is flagged in [Threat Model — Open Questions](/design/threat-model/schema-rules/#open-questions). +Cross-server replication of a *single* album (where two users on different home servers each want to write the same album) is **out of scope for v1** and deferred to v2. v1 supports cross-server sharing in the read direction (Alice on `home.tld` shares an album to Bob on `other.tld`; Bob reads via federation; Bob's writes either remain on `home.tld` via a registered or sponsored account, or are out of scope). The v2 design space is flagged in [Threat Model — Open Questions](/design/threat-model/schema-rules/#open-questions). What ships *before* v2 is the aggregated album, next — multi-party sharing without multi-writer albums. + +## Federated Shared Albums (Aggregated Albums) + +The thing people actually want from federated sharing — "everyone on the trip puts their photos in the album, whichever server their account lives on" — does not require a multi-writer album. It requires the *view* of one. An **aggregated album** is N ordinary container albums, one per contributor, each homed on its contributor's own server and single-writer-domain as always, presented by clients as **one logical album**. Every rule above stays intact: one home server per constituent, pull-only replication, no cross-server commit ordering. There is **zero new server surface** — servers never learn that a group exists. + +### The Album-Group Assertion + +The group is client-side metadata, asserted per constituent: + +- The creator mints a `group_id` (UUIDv7) and shares it in the invite. +- Each contributor's client writes an **album-group assertion** into *their own* container album's encrypted collaborative-metadata stream (the [operation path](/design/metadata/#how-operations-travel); this doc owns the assertion's schema): + +```rust +AlbumGroupAssertion { + group_id: UUIDv7, + group_name: Lww, // converges across participants like caption_lww + member_hint: Vec<(album_id, home_server)>, // advisory discovery only, never trusted +} +``` + +There is deliberately **no shared mutable group object** — that would be exactly the multi-writer cross-server state v1 defers. A union of per-album assertions needs no cross-server consensus: each assertion lives in its author's own single-writer album and travels to readers through the existing feed/pull machinery. + +### Membership and Rendering + +- **Everyone writes their own, reads the others.** A participant contributes by writing to their own constituent (ordinary album writes on their home server) and reads the other constituents via ordinary per-album invites — same-server membership or [federation capabilities](#federation-capabilities) — with blobs pulled from each origin. +- **Inclusion is injection-proof by construction.** A constituent appears in the local aggregate only if the local user is a *member* of that album (holds its AMK) **and** it asserts the `group_id`. A stranger's album cannot inject itself into anyone's view: without an invite, its assertion is never even decryptable. `member_hint` only tells the client where to *ask*; membership does the admitting. +- **The aggregate is a computed view.** Merged ordering is `capture_timestamp` with `asset_id` as the tiebreak — computed at render, nothing stored, so it is idempotent under the [grouping-convergence requirement](/design/metadata/#grouping-convergence-requirement) by definition. It holds no keys and is no access-control boundary, exactly like every [view album](/design/organization/#system--smart-albums-views). Group name converges by LWW across assertions; the cover is a per-viewer preference in the library-settings document (falling back to newest asset) — deliberately not shared state. +- **Partial views degrade visibly.** One origin unreachable → that constituent's entries render from the local index with the existing per-origin [degraded state](#robustness-against-connectivity-loss) ("photos from `other.tld` currently unavailable"); nothing is removed. [Search](#federated-breadcrumb-index) spans constituents through the breadcrumb index unchanged. + +### Leaving, Revocation, and Moderation + +- **Leaving** = removing your assertion — your constituent drops out of every participant's aggregate on their next sync. Optionally also unshare your container (AMK epoch bump + capability revocation) to cut read access to the historical photos. +- **There is no group-level kick.** Each contributor is sovereign over their own constituent; you can stop someone from seeing *your* photos (unshare), but nobody can remove someone else's constituent from the group for other viewers. Stated as an honest limitation of the aggregation model rather than papered over — a true shared-governance album is precisely the v2 problem. +- **Moderation is per-origin.** Blocking a server or user ([Moderation](/design/moderation/)) drops their constituent from your aggregate; the per-peer containment rules above apply to each origin independently. + +### Relationship to v2 Multi-Writer + +The aggregate neither depends on nor precludes true multi-writer albums. Nothing server-side encodes the group, so there is nothing to migrate: if v2 ships a genuinely co-written album, it joins a group as one more constituent and the group collapses to it when participants consolidate. [Open question #1](/design/threat-model/schema-rules/#open-questions) stays open; aggregated albums are the shipping answer until it closes. ## Federation Capabilities diff --git a/capsule-docs/src/content/docs/design/metadata.md b/capsule-docs/src/content/docs/design/metadata.md index 563f9e5e..8d540e02 100644 --- a/capsule-docs/src/content/docs/design/metadata.md +++ b/capsule-docs/src/content/docs/design/metadata.md @@ -176,7 +176,7 @@ We encrypt the **operations**, not the resulting state. Merges are then commutat Each operation carries the same `prior_provenance_hash` chain link as any [lifecycle action](/design/authorization/#the-closed-action-set), so a metadata-update is provenance-tracked exactly like a create or delete. -The same encrypted-operation path also carries the per-owner **library-settings document** — [smart-album](/design/organization/#system--smart-albums-views) definitions (predicate + display name) and similar client-authored organizational state — synced and merged across devices like any other collaborative metadata, and never legible to the server. (The [default-album](/design/organization/#the-default-album) *designation* is separate: a non-secret server-side owner pointer, not part of this encrypted document.) +The same encrypted-operation path also carries each album's [album-group assertion](/design/federation/#the-album-group-assertion) (schema owned by Federation) and the per-owner **library-settings document** — [smart-album](/design/organization/#system--smart-albums-views) definitions (predicate + display name) and similar client-authored organizational state — synced and merged across devices like any other collaborative metadata, and never legible to the server. (The [default-album](/design/organization/#the-default-album) *designation* is separate: a non-secret server-side owner pointer, not part of this encrypted document.) ### Grouping Convergence (Requirement) diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index eab21305..2e729f1e 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -38,7 +38,7 @@ A container album must be explicitly created, but a brand-new account has none ### System & Smart Albums (Views) -View albums are organizational surfaces computed entirely client-side over the assets the user can already decrypt (the union of their container-album memberships), materialized by querying the [local index](/design/filesystem/client/#local-index-staleness). A view is **not** an MLS group, holds **no** AMK, **owns no assets**, and is **not** a sharing or access-control boundary — sharing happens only at the container tier. Two kinds: +View albums are organizational surfaces computed entirely client-side over the assets the user can already decrypt (the union of their container-album memberships), materialized by querying the [local index](/design/filesystem/client/#local-index-staleness). The [aggregated federated album](/design/federation/#federated-shared-albums-aggregated-albums) is a view in exactly this sense — its predicate is an album-group id spanning constituents on different home servers. A view is **not** an MLS group, holds **no** AMK, **owns no assets**, and is **not** a sharing or access-control boundary — sharing happens only at the container tier. Two kinds: - **System albums** — built-in and implicit. The canonical one is **All** — every asset the user can see; because that is the union over their containers, every asset appears in it (which is exactly why the [default album](#the-default-album) matters: an import always enters *some* container and so shows up in All). [Trash](#recycling) is another system view, over lifecycle state. - **Smart / dynamic albums** — user-defined filtered views whose membership is a predicate over sidecar fields and AI-derived attributes ([Metadata](/design/metadata/#sidecar-schema-v1), [AI](/design/ai/)). Membership is **computed**, never stored: editing a smart album, or an asset's attributes, never moves or re-encrypts an asset. A definition (predicate + display name) is user content — stored in the per-owner, E2E-encrypted **library-settings document** declared in [Metadata — How Operations Travel](/design/metadata/#how-operations-travel), synced across the user's devices with the same [CRDT semantics](/design/metadata/#collaborative-metadata) as other collaborative metadata, so the server never learns it. (That document's concrete schema is a design follow-up, tracked as its own slice in the repo-root `SLICES.md`; this doc owns only what it must carry — smart-album definitions and the scope-override map.) diff --git a/capsule-docs/src/content/docs/design/threat-model/schema-rules.md b/capsule-docs/src/content/docs/design/threat-model/schema-rules.md index f226a769..11930b79 100644 --- a/capsule-docs/src/content/docs/design/threat-model/schema-rules.md +++ b/capsule-docs/src/content/docs/design/threat-model/schema-rules.md @@ -62,7 +62,7 @@ The deprecation surface is **never** retroactive against historical state. Old a One design question remains open — and it is **deliberately deferred to v2**, not a v1 blocker: -1. **Cross-server album replication (v2).** v1 pins each album to a single home server; v2 will need a story for cross-server MLS state and federated commit ordering. +1. **Cross-server album replication (v2).** v1 pins each album to a single home server; v2 will need a story for cross-server MLS state and federated commit ordering. The interim shipping answer is the [aggregated album](/design/federation/#federated-shared-albums-aggregated-albums) — client-side aggregation of per-contributor single-writer albums — which neither depends on nor precludes the v2 design. The following questions have since been **resolved** and now live in their owner docs, not here: From 7faf15699ab10a1fba01b6ff1c4fa181e2a2d8ee Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 20:02:02 -0400 Subject: [PATCH 27/58] feat(core): device cohorts for session-ledger grouping MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reinstalled device re-enrolls with a new device_id by design (hardware- bound keys), so one physical phone accumulates indistinguishable session- ledger entries. The device cohort hash groups them: ONE primary identifier per platform (iOS Keychain-persisted non-synchronized seed — IDFV rejected, it dies exactly when cohorts matter; Android SSAID; macOS IOPlatformUUID; Windows MachineGuid; Linux hashed machine-id), folded through domain- separated canonical CBOR — never naive concatenation — as SHA-256(CBOR([label, user_id, platform_tag, primary_id])). user_id is in the preimage so the same device under two accounts is unlinkable. Honest scope, stated in-doc: factory-reset stability is impossible on iOS/Android by OS design (attestation returns verdicts, not identifiers), so the promise is reinstall-stable everywhere, reset-stable only on macOS — a scoped guarantee rather than a false one. The value is structurally advisory: client-asserted, unverifiable, and no authorization decision may read it; sent only in the session-creation body, never in signed artifacts or to peers. UX asserts rather than litigates ('a device you've used before'), with a one-tap support bundle of the hash + device-id map as the dispute path. Pure function + label land in capsule_core::cohort (tested); server storage and client emission are slices S-C13/S-D11. --- capsule-core/src/cohort.rs | 103 ++++++++++++++++++ capsule-core/src/lib.rs | 1 + .../src/content/docs/design/authentication.md | 44 ++++++++ .../docs/design/threat-model/validation.md | 2 +- 4 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 capsule-core/src/cohort.rs diff --git a/capsule-core/src/cohort.rs b/capsule-core/src/cohort.rs new file mode 100644 index 00000000..aa211438 --- /dev/null +++ b/capsule-core/src/cohort.rs @@ -0,0 +1,103 @@ +//! Device-cohort hash — the advisory session-grouping aid (SSoT: +//! [Authentication — Device Cohorts]; server storage + client emission are +//! slices `S-C13`/`S-D11` in the repo-root `SLICES.md`). +//! +//! One deterministic hash groups a physical device's sessions across app +//! reinstalls (reset-stability is platform-limited and documented honestly in +//! the owner doc). The construction is domain-separated canonical CBOR — never +//! naive concatenation — and folds the `user_id` in so the same device under +//! two accounts yields unlinkable hashes. +//! +//! **Advisory-only invariant:** the value is client-asserted and unverifiable; +//! no authorization or capability decision may ever read it. +//! +//! [Authentication — Device Cohorts]: https://docs/design/authentication/#device-cohorts + +use ciborium::value::Value; +use uuid::Uuid; + +use crate::cbor; +use crate::crypto::hash::{Hash32, hash_bytes}; + +/// Domain-separation label (versioned; a construction change bumps the suffix). +pub const DEVICE_COHORT_V1: &str = "capsule-device-cohort/v1"; + +/// Closed platform tag (wire value is the lowercase name; a new platform is an +/// additive protocol change). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PlatformTag { + Ios, + Android, + Macos, + Windows, + Linux, +} + +impl PlatformTag { + pub fn as_str(self) -> &'static str { + match self { + PlatformTag::Ios => "ios", + PlatformTag::Android => "android", + PlatformTag::Macos => "macos", + PlatformTag::Windows => "windows", + PlatformTag::Linux => "linux", + } + } +} + +/// The cohort hash: +/// `SHA-256( canonical-CBOR([ "capsule-device-cohort/v1", user_id, platform_tag, primary_id ]) )`. +/// +/// `primary_id` is the platform's single primary identifier per the owner doc's +/// table (Keychain seed / SSAID / IOPlatformUUID / MachineGuid / machine-id) — +/// exactly one input per platform, chosen for stability, never a concatenated +/// fingerprint. +pub fn cohort_hash(user_id: Uuid, platform: PlatformTag, primary_id: &str) -> Hash32 { + let array = Value::Array(vec![ + Value::Text(DEVICE_COHORT_V1.to_string()), + Value::Bytes(user_id.as_bytes().to_vec()), + Value::Text(platform.as_str().to_string()), + Value::Text(primary_id.to_string()), + ]); + hash_bytes(&cbor::value_to_canonical_vec(&array)) +} + +#[cfg(test)] +mod tests { + use super::*; + + const ID: &str = "9774d56d682e549c"; + + #[test] + fn deterministic_across_invocations() { + let u = Uuid::from_u128(0x1234); + assert_eq!( + cohort_hash(u, PlatformTag::Android, ID), + cohort_hash(u, PlatformTag::Android, ID), + ); + } + + #[test] + fn user_id_fold_makes_accounts_unlinkable() { + // Same physical device (same primary id), two accounts → distinct + // hashes: the cross-account correlation surface is removed at the source. + let a = cohort_hash(Uuid::from_u128(1), PlatformTag::Android, ID); + let b = cohort_hash(Uuid::from_u128(2), PlatformTag::Android, ID); + assert_ne!(a, b); + } + + #[test] + fn platform_and_label_are_domain_separating() { + let u = Uuid::from_u128(0x1234); + assert_ne!( + cohort_hash(u, PlatformTag::Android, ID), + cohort_hash(u, PlatformTag::Linux, ID), + ); + // Structured encoding, not concatenation: shifting bytes between + // components must change the hash. + assert_ne!( + cohort_hash(u, PlatformTag::Android, "ab"), + cohort_hash(u, PlatformTag::Android, "a"), + ); + } +} diff --git a/capsule-core/src/lib.rs b/capsule-core/src/lib.rs index eb910b6f..81091739 100644 --- a/capsule-core/src/lib.rs +++ b/capsule-core/src/lib.rs @@ -1,5 +1,6 @@ pub mod backup; pub mod cbor; +pub mod cohort; pub mod constants; pub mod crypto; pub mod db; diff --git a/capsule-docs/src/content/docs/design/authentication.md b/capsule-docs/src/content/docs/design/authentication.md index 6656846e..65dde08f 100644 --- a/capsule-docs/src/content/docs/design/authentication.md +++ b/capsule-docs/src/content/docs/design/authentication.md @@ -99,6 +99,47 @@ The asymmetric authentication on (3) addresses a damage scenario that pure sessi Note: the server can theoretically just kick off sessions because session tokens are stored server-side and the server holds the encrypted data. But this should not ever be implemented and an attempt to do so would be a bug — it bypasses the audit trail of a user-initiated revoke. +## Device Cohorts + +The session ledger has a legibility problem: reinstalling the app re-enrolls with a **new** `device_id` by design (device keys are hardware-bound and non-exportable — [Metadata, Add-id Binding](/design/metadata/#add-id-binding)), so one physical phone accumulates several ledger entries over its life and the user cannot tell them apart. The **device cohort hash** groups sessions from the same physical device. It is a grouping aid, nothing more. + +### Per-Platform Primary Identifier + +One primary identifier per platform — fewer, better-chosen inputs beat a concatenated fingerprint that splits whenever any component shifts: + +| Platform | Primary identifier | Survives app reinstall | OS reinstall | Factory reset | +| --- | --- | --- | --- | --- | +| iOS | Keychain-persisted random 128-bit cohort seed (`ThisDeviceOnly`, **non-synchronized** — iCloud sync would merge distinct physical devices) | yes in practice (not Apple-guaranteed) | no | no | +| Android | SSAID (app-signing-key-scoped `ANDROID_ID`) | yes | no | no | +| macOS | `IOPlatformUUID` | yes | yes | **yes** | +| Windows | `MachineGuid` | yes | no | no | +| Linux | `/etc/machine-id` (never used raw — hashed below, per systemd guidance) | yes | no | no | + +(IDFV was rejected for iOS: it resets on reinstall when no sibling app remains — precisely the case cohorts exist for.) + +**Honest scope.** "The same device even after a factory reset" is technically impossible on iOS and Android: a reset destroys every app-accessible identifier by OS design, and post-reset attestation (DeviceCheck, Play Integrity) yields verdicts, not identifiers. The promise is therefore: **reinstall-stable everywhere; reset-stable only where the OS allows (macOS)**. A factory-reset phone starts a new cohort, and the doc says so rather than pretending otherwise — the same honesty rule as [platform limitations](/design/import/download-sync/#platform-limitations). (DeviceCheck's per-device bits may later corroborate a boolean "this hardware enrolled before"; deliberately out of scope for v1.) + +### Encoding + +```text +cohort_hash = SHA-256( canonical-CBOR([ "capsule-device-cohort/v1", user_id, platform_tag, primary_id ]) ) +``` + +Domain-separated, [canonical CBOR](/design/metadata/#canonical-cbor-encoding) (never naive string concatenation), with a closed `platform_tag` enum. **`user_id` is folded in** so the same physical device under two accounts yields unlinkable hashes — the cross-account correlation surface is removed at the source. The pure function lives in `capsule-core` (`cohort` module) so every platform computes it identically. + +### Privacy and Scope + +- Sent **only** in the session-creation request body during the auth ceremony; **never** in signed artifacts (manifests, sidecars, the device directory), never to federated peers, never in `.well-known`. It is registered in the [`X-Capsule-*` header census](/design/threat-model/validation/#universal-headers) by pointer as a body field so no header variant drifts into existence. +- **Advisory-only, structurally:** the value is client-asserted and unverifiable, so **no authorization or capability decision may read it** — otherwise it becomes spoofable attack surface. It never substitutes for `device_id` (random UUIDv4, security-bearing) or the DSK. A server that receives an absent or garbage cohort value behaves identically to one that receives a valid one. + +### Server Storage and Surfacing + +The session record carries `cohort_hash`, and a small durable `device_cohorts(user_id, cohort_hash, first_seen, last_seen)` map persists it beyond session expiry — session-store-only would forget cohorts exactly when the "seen before" question matters. The session-listing surface returns the cohort per session plus the cohort map; clients group the ledger by cohort. + +### UX and Support Contract + +The client **asserts, it does not litigate**: "a device you've used before (last seen *date*)" — there is deliberately no "this isn't my device" toggle, because the user cannot adjudicate a hash and the value is advisory anyway. The dispute path is a **support report**: one tap bundles `{cohort_hash, [(device_id, session_id, first_seen, last_seen)]}` — the exact hash and device-id map — for a bug report. + ## Validation - **Token issuance round-trip (unit).** Generate a session token; issue an access JWT from it; verify the JWT under the server's Ed25519 key. Repeat with rotated keys; assert old JWTs verify under the old key for their grace window. @@ -106,6 +147,9 @@ Note: the server can theoretically just kick off sessions because session tokens - **Revoke-all master-key proof (unit).** Issue a revoke-all without master-key proof; assert rejection. With proof; assert success and invalidation of every other session. - **Login flow (smoke).** Full OIDC handshake against a testcontainer IdP; assert session token issued, persisted, and usable for an immediate access-token request. Re-run after a server restart; assert resilience. - **Account portability (smoke).** Issue a moved certificate from server A; assert server B can register the same IK; assert federated peers honor the move after fetching A's well-known. +- **Cohort hash vectors (unit).** Known-answer vectors for `cohort_hash` through the cross-language canonical-CBOR conformance gate; same `primary_id` under two `user_id`s → distinct hashes. +- **Cohort is advisory (unit).** Session creation with an absent, malformed, or colliding cohort value behaves identically to a valid one; a tripwire test asserts no signed-structure schema contains a cohort field. +- **Cohort grouping (smoke).** Two sessions with one cohort group together in the session listing; a reinstall (new `device_id`, same cohort) groups with "previously used"; the durable map outlives session expiry. The cross-module case — auth → query library schema — is one bounded E2E test listed in [Module Map](/design/module-map/#e2e-test-surface). diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index 0c9f14b9..20ccc991 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -113,7 +113,7 @@ The rules are stated once, in REST terms (headers + HTTP statuses). On the gRPC | `X-Capsule-Protocol-Max` | server on every response | the highest protocol version this server accepts | | `X-Capsule-Min-Client-Build` | server on responses | semver deprecation cutoff; advisory unless the path is hard-deprecated | -This table is also the **census of the `X-Capsule-*` header namespace**. Surface-specific headers register here by pointer: the upload protocol's `X-Capsule-Offset`, `X-Capsule-Content-Length`, `X-Capsule-Upload-Status` (server → client on `HEAD /upload/{id}`), `X-Capsule-Checksum` (**required** on `PATCH /upload/{id}`), and `X-Capsule-Suggested-Chunk-Size` (semantics owned by [Import — Upload Protocol](/design/import/upload-protocol/#endpoints)). A new `X-Capsule-*` header MUST be registered here when introduced — two homes for the namespace is how headers drift. +This table is also the **census of the `X-Capsule-*` header namespace**. Surface-specific headers register here by pointer: the upload protocol's `X-Capsule-Offset`, `X-Capsule-Content-Length`, `X-Capsule-Upload-Status` (server → client on `HEAD /upload/{id}`), `X-Capsule-Checksum` (**required** on `PATCH /upload/{id}`), and `X-Capsule-Suggested-Chunk-Size` (semantics owned by [Import — Upload Protocol](/design/import/upload-protocol/#endpoints)). A new `X-Capsule-*` header MUST be registered here when introduced — two homes for the namespace is how headers drift. (Registered by pointer as a **body field, deliberately not a header**: the advisory `cohort_hash` in the session-creation request — semantics owned by [Authentication — Device Cohorts](/design/authentication/#device-cohorts).) ### Fail-Closed Rules From d35d972b3e19af6a0bc53ff2a8e04e46dffa587a Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 20:04:25 -0400 Subject: [PATCH 28/58] feat(core): recovery-key verification cadence and single-root invariant MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two hardening moves on the recovery story, no re-architecture — the audit confirmed the master key is already the sole escrowed root: - The Single-Root Invariant is now normative: every recovery path terminates at the one account master key, and the recovery secret (or its Shamir quorum) is the only user-held secret class. The wraps-not-roots audit is written into the doc (escrow, backup artifact, Shamir, OGK/Drop-Key escrows, the sponsoree exception) so no future feature quietly introduces a second secret class and nobody re-derives the chain by hand. - A recovery secret written on a napkin a year ago is one the user only BELIEVES they have, so the app now periodically verifies it: local-only (Argon2id unwrap of the cached escrow + HKDF derived-tag compare — no server round-trip, hence no guessing oracle and no lock-out potential), 7d -> 90d -> 180d cadence with re-arm on device-add/rotation/restore, a stale-cache refresh-before-fail rule so rotations don't manufacture false failures, and a guided re-wrap after repeated failure — a fresh secret re-wrapping the SAME master key: O(1) escrow replacement, no data re-encryption. Clients are forbidden from persisting the passphrase to auto-pass the check (duty line in clients.md). capsule_core::backup gains the tested verify_recovery_secret seam + the capsule-recovery-verify/v1 label; cadence/UX is slice S-D12. --- capsule-core/src/backup/mod.rs | 74 ++++++++++++++++++- .../content/docs/design/backup-recovery.md | 34 +++++++++ .../src/content/docs/design/clients.md | 1 + 3 files changed, 108 insertions(+), 1 deletion(-) diff --git a/capsule-core/src/backup/mod.rs b/capsule-core/src/backup/mod.rs index be8a2e0d..39a976a8 100644 --- a/capsule-core/src/backup/mod.rs +++ b/capsule-core/src/backup/mod.rs @@ -21,7 +21,7 @@ use thiserror::Error; use crate::crypto::primitives::DeviceTier; use crate::crypto::pwkdf::{self, WrappedSecret}; -use crate::crypto::{CryptoError, rng}; +use crate::crypto::{CryptoError, kdf, rng}; /// The backup artifact format version. pub const ARTIFACT_FORMAT_VERSION: u16 = 1; @@ -69,6 +69,47 @@ pub fn recover_master_key( .map_err(|_| BackupError::Auth("escrowed master key wrong length")) } +// ── Recovery verification cadence (S-D12) ─────────────────────────────────── + +/// Domain-separation label for the recovery-verification derived-tag compare +/// (SSoT: [Backup — Recovery Verification Cadence]). +/// +/// [Backup — Recovery Verification Cadence]: https://docs/design/backup-recovery/#recovery-verification-cadence +pub const RECOVERY_VERIFY_V1: &[u8] = b"capsule-recovery-verify/v1"; + +/// Outcome of a local recovery-secret verification. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum VerifyOutcome { + /// The entered secret unwraps the (cached) escrow to the device-held master key. + Verified, + /// The entered secret does not open the blob — wrong secret, or the cached + /// blob is stale after a rotation. Per the stale-cache rule the caller + /// refreshes the blob and retries once before recording a failure. + NotVerified, +} + +/// Local-only verification that the user still holds the recovery secret: +/// unwrap the cached escrow blob, then compare an HKDF-derived tag against the +/// tag derived from the device-held master key (derived-tag compare — raw key +/// bytes are never compared or surfaced). Zero network I/O by construction; the +/// cadence, snooze, and refresh-before-fail policy live in the client (S-D12). +pub fn verify_recovery_secret( + cached_escrow: &WrappedSecret, + passphrase: &[u8], + device_master: &[u8; 32], +) -> VerifyOutcome { + let Ok(unwrapped) = recover_master_key(cached_escrow, passphrase) else { + return VerifyOutcome::NotVerified; + }; + let tag_a = kdf::hkdf_sha512(&unwrapped, b"", RECOVERY_VERIFY_V1, 32); + let tag_b = kdf::hkdf_sha512(device_master, b"", RECOVERY_VERIFY_V1, 32); + if tag_a == tag_b { + VerifyOutcome::Verified + } else { + VerifyOutcome::NotVerified + } +} + // ── Opt-in Shamir 2-of-3 social recovery ──────────────────────────────────── /// Split a 32-byte recovery seed into 3 Shamir shares; any 2 reconstruct it, 1 reveals @@ -103,6 +144,37 @@ pub fn new_recovery_seed() -> [u8; 32] { mod tests { use super::*; + #[test] + fn recovery_verification_tag_compare() { + let master = [0x42u8; 32]; + // Fast params for the test. + let blob = pwkdf::wrap_with( + &master, + b"correct horse battery staple", + crate::crypto::primitives::Argon2Params { + mem_kib: 64, + t_cost: 1, + p_cost: 1, + }, + ) + .unwrap(); + // Correct secret against the matching device master → Verified. + assert_eq!( + verify_recovery_secret(&blob, b"correct horse battery staple", &master), + VerifyOutcome::Verified + ); + // Wrong secret → NotVerified (caller applies the refresh-then-retry rule). + assert_eq!( + verify_recovery_secret(&blob, b"wrong", &master), + VerifyOutcome::NotVerified + ); + // Right secret, different device master (rotated elsewhere) → NotVerified. + assert_eq!( + verify_recovery_secret(&blob, b"correct horse battery staple", &[0x43u8; 32]), + VerifyOutcome::NotVerified + ); + } + #[test] fn master_key_escrow_round_trip() { let master = [0x42u8; 32]; diff --git a/capsule-docs/src/content/docs/design/backup-recovery.md b/capsule-docs/src/content/docs/design/backup-recovery.md index 28aeb837..2f7457cf 100644 --- a/capsule-docs/src/content/docs/design/backup-recovery.md +++ b/capsule-docs/src/content/docs/design/backup-recovery.md @@ -61,6 +61,14 @@ The account master key is the single backed-up root of the key hierarchy (see [C - Derive the wrapping key with the [password-based KDF](/design/cryptography/primitives/#password-based-kdf). Store the wrapped blob server-side. - If you can run enclaves (SGX/Nitro/SEV-SNP), Signal's SVR pattern is the *only* sanctioned way to soften that floor: rate-limit unwrap attempts inside the enclave so a shorter, human-friendly PIN is still safe. Without an attested enclave, the ≥128-bit floor is mandatory — never a 4-digit PIN, never a short numeric code. +## Single-Root Invariant + +**Every recovery path terminates at the one account master key, and the recovery secret (or a quorum of its Shamir shares) is the only user-held secret class. No feature may introduce a second escrowed secret class or a recovery path that bypasses the master key.** + +This is already true; the invariant fixes it against drift. The audit, stated so nobody has to re-derive it: the server **escrow** is the master key wrapped by the recovery secret (above); the **backup artifact's** wrap key derives from that same recovery passphrase; **Shamir** splits that same seed; the [OGK](/design/cryptography/keys/#owner-group-keys-ogks) and Drop-Key escrows are wraps *reachable from* the master key, not additional roots; [sponsored accounts](/design/authentication/#account-types) deliberately hold no root of their own — every path routes through the sponsor's, which is why the verification cadence below never prompts a sponsoree. These are **wraps, not roots** — holding the recovery secret reaches all of them; losing everything but the recovery secret loses nothing. + +One versioning seam to know about, not remove: an already-exported backup artifact stays bound to the passphrase in force at export. Rotating the recovery secret (below) therefore ends with "re-export or destroy old artifacts" guidance — a property of offline artifacts, not a second secret class. + ## Recovery Mechanisms Two recovery mechanisms ship by default; a third is available opt-in for users who want extra redundancy without compromising the default's simplicity. These complement the [seven independent recovery paths](/design/cryptography/failure-modes/#redundant-recovery-paths); this section names the mechanisms a user actually invokes. @@ -83,6 +91,28 @@ Users who want to spread recovery across trusted parties or storage locations ca This is the social-recovery escape hatch — useful for users who would otherwise lose access from a single forgotten passphrase plus a single dead device. +## Recovery Verification Cadence + +A recovery secret that was written on a napkin thirteen months ago is a recovery secret the user *believes* they have. The client therefore occasionally asks the user to **verify** it — and the check is designed so it can never itself become a lock-out or a guessing oracle. + +### Local Verification + +Verification is **local-only**: the client keeps a cached copy of the escrow blob (fetched at enrollment, refreshed opportunistically via sync); the user enters the passphrase; the client runs the [password KDF](/design/cryptography/primitives/#password-based-kdf), unwraps the cached blob (`capsule_core::backup::verify_recovery_secret`), and compares an HKDF-derived tag (`capsule-recovery-verify/v1`) against the same tag derived from the device-held master key — a derived-tag compare, never raw key bytes. No server round-trip: it works offline, creates no server-side guessing surface, and a failure can't lock anything. + +**Stale-cache rule:** before recording a failure, the client refreshes the escrow blob from the server (it may have been rotated from another device) and retries once — otherwise every rotation would manufacture false failures on the user's other devices. + +### Schedule and Triggers + +- **7 days** after setup (catches the lost-napkin case while re-setup is cheap), then **90 days**, backing off to a **180-day cap** after two consecutive successes. +- **Re-arm triggers** (reset to the 7-day step): a new device enrolls (the prompt lands on the *new* device — it has never seen the passphrase); the recovery secret rotates; a restore-from-escrow completes. +- Snooze: 24 h or 7 d, at most 3 consecutive snoozes, then a persistent non-blocking badge. The check **never blocks** sync, unlock, or any critical flow — it is advisory by design. + +### On Repeated Failure: Guided Re-Wrap + +After 3 failures across ≥ 2 app sessions — or an explicit "I lost it" — the client runs the guided rotation flow: mint a fresh ≥128-bit recovery secret, **re-wrap the same master key**, replace the server escrow (a single active escrow; the old blob is deleted, so the lost secret unwraps nothing), re-run the setup-style type-back gate, re-issue Shamir shares if enrolled (old shares explicitly invalidated and surfaced as such), and surface the old-backup-artifact guidance from the [single-root invariant](#single-root-invariant). + +**This is wrap rotation, not key rotation**: an O(1) escrow-blob replacement with no data re-encryption and no blob-hash changes. Rotating the master key itself remains the separate compromise procedure owned by [Cryptography — Keys](/design/cryptography/keys/). + ## Backup Verification A restore that overwrites live state silently is the worst foot-gun a backup system can ship. Capsule therefore makes **dry-run the default**: a `restore` invocation runs in dry-run mode unless the user passes an explicit `--commit` flag (or its UI equivalent: a confirm-with-typed-phrase dialog after the dry-run report is shown). The mode hierarchy: @@ -105,6 +135,10 @@ The MANIFEST.cbor carries the exporter's device id, the export timestamp, the so ## Validation +- **Local verify round-trip (unit).** Offline: the correct passphrase verifies with zero network I/O; a wrong one fails without side effects. Rotate the escrow out-of-band; assert the stale-cache rule refreshes before recording a failure and then passes with the new passphrase. +- **Cadence schedule (unit).** Under a mocked clock: 7 d → 90 d → 180 d backoff; re-arm on device-add, rotation, and restore; snooze caps; never blocks a critical flow. +- **Guided re-wrap (smoke).** After the failure threshold: a new escrow wraps the *same* master key (fixture assets still decrypt; blob hashes unchanged), the old escrow is rejected everywhere, Shamir re-issue is prompted where enrolled, and the old-artifact guidance is surfaced. +- **Single-root audit (unit).** A doc-driven test walks the seven recovery paths and asserts each terminates at the master key; a tripwire asserts no second escrowed secret class exists in the schema. - **Artifact round-trip (unit).** Export → import a small library; assert byte-equal blob set, sidecars, and provenance chains. Determinism check: re-export the same library twice; assert byte-identical archives. - **MANIFEST verification (unit).** Tamper individual entries; assert HMAC mismatch detected. Tamper MANIFEST itself and re-HMAC; assert exporter-signature mismatch detected. Strip the exporter from the device directory; assert restore refusal. - **AMK-completeness check (unit).** Build an artifact whose `keys/amk-ledger.cbor` is deliberately missing an `amk_version` that an included asset references; assert detection at dry-run, before any commit. Build a self-sufficient artifact; assert every included asset decrypts from the artifact's own ledger with no server contact. diff --git a/capsule-docs/src/content/docs/design/clients.md b/capsule-docs/src/content/docs/design/clients.md index 47447f49..b81aa379 100644 --- a/capsule-docs/src/content/docs/design/clients.md +++ b/capsule-docs/src/content/docs/design/clients.md @@ -33,6 +33,7 @@ Clients are not trusted to enforce their own correctness — but they **are** re - **Honor the [forbidden behaviors checklist](/design/threat-model/schema-rules/#forbidden-client-behaviors).** A client that backdates timestamps, strips unknown sidecar fields, overwrites provenance, signs for an epoch it does not hold, or invokes `revoke_all_sessions` without master-key proof is *buggy by definition*. Centralizing the validation logic in `capsule-core` ensures each native client gets the same enforcement; the wrapper layer that issues UI surfaces for quarantine and protocol-mismatch errors is the platform-specific portion. +- **Run the [recovery verification cadence](/design/backup-recovery/#recovery-verification-cadence)** — and never persist the passphrase (or any derivative able to satisfy the check) to auto-pass it: a client that does so is buggy by definition, because the check exists to verify the *user* still holds the secret. ## Reading State From a Newer Client From acafac4f3631c2887713a4a1e4d95f827c205fac Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 20:05:07 -0400 Subject: [PATCH 29/58] docs(design): formalize the local-source-to-album scope grammar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The (scope -> album) override map existed as a named hook with no grammar, which made 'how do folders on my phone map to remote albums' per-client improvisation. The scope is now a canonical identity — closed platform + source-kind enums plus a per-platform canonical locator, hashed through domain-separated canonical CBOR into a deterministic scope_id — so two devices looking at the same source agree without coordination. Locators are chosen for reinstall-stability per platform (Android uses relative paths, never BUCKET_ID, which is a display-name hash that differs across devices; removable media key on volume UUID so a re-mounted card stays the same scope). The mapping rows live in the E2E-encrypted library-settings document as LWW registers; resolution order is explicit and the planner records which rule fired; unmapped sources ask once instead of silently inventing destinations. This completes the concrete half of the S-Z1 library-settings schema. --- .../src/content/docs/design/organization.md | 28 ++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/capsule-docs/src/content/docs/design/organization.md b/capsule-docs/src/content/docs/design/organization.md index 2e729f1e..3aa52cce 100644 --- a/capsule-docs/src/content/docs/design/organization.md +++ b/capsule-docs/src/content/docs/design/organization.md @@ -33,9 +33,35 @@ A container album must be explicitly created, but a brand-new account has none - **De facto and nameless.** It is an ordinary container album in every cryptographic and lifecycle respect — its own MLS group, random per-epoch AMK, `history_policy`, `protocol_version` pin, retention — but carries no user-assigned name; a client typically surfaces it as the library's primary view. - **Specially identified.** Its album ID is **derived deterministically from the account master key** (the master key derives the *identifier*, not any key — see [Keys — Key Chain](/design/cryptography/keys/#key-chain)). The ID is therefore unique per user, unguessable before creation, and recomputable on any of the user's devices and after recovery — so a device can locate the default album from the master key alone, without waiting on a synced pointer. - **Designation is a server-side owner pointer.** Which container is *currently* the default is a non-secret `default_album_id` on the owner record ([Filesystem — Server](/design/filesystem/server/#ownership-partitioning-and-quota)), defaulting to the derived de facto album. The pointer is not security-bearing — a write still requires real album write capability ([server-side invariants](/design/threat-model/validation/#server-side-validation-invariants), invariant 6). -- **One or more defaults, context-driven.** A client may register **scope overrides** — `(scope → album)` mappings that re-point the default for a context (a per-source auto-import mapping; "while viewing album X, new photos default to X"). The resolution rule, `resolve_default_album(context)`, returns the active scope's override if set, else the owner pointer, else the derived de facto album. It **always** resolves to a container — a [view](#system--smart-albums-views) can never be an import destination. The [import planner](/design/import/pipeline/#plan--confirm) consumes this when the user picks no album. +- **One or more defaults, context-driven.** A client may register **scope overrides** — `(scope → album)` mappings that re-point the default for a context (a per-source auto-import mapping; "while viewing album X, new photos default to X"). The resolution rule, `resolve_default_album(context)`, returns the active scope's override if set, else the owner pointer, else the derived de facto album. It **always** resolves to a container — a [view](#system--smart-albums-views) can never be an import destination. The [import planner](/design/import/pipeline/#plan--confirm) consumes this when the user picks no album. The scope grammar is formalized below. - **Stable.** Re-designating the default just moves the pointer. The current default **cannot be deleted while designated** — the user must repoint first, or the client recreates the derived de facto album — so import always has a home. +#### Scope Grammar (Local Source → Album Mapping) + +How a local folder, camera roll, or watched directory on each platform maps to a remote album is a formal contract, not per-client improvisation. A **scope** is the canonical identity of an import source: + +```rust +Scope { + platform: PlatformTag, // closed enum (shared with device cohorts) + source_kind: SourceKind, // closed enum: camera_roll | screenshots | app_collection | folder | watched_dir | removable_volume + locator: String, // canonical, per the platform table below +} +// scope_id = SHA-256( canonical-CBOR([ "capsule-import-scope/v1", platform, source_kind, locator ]) ) +``` + +`scope_id` is deterministic (domain-separated canonical CBOR — the same discipline as every derived identifier), so two devices of the same platform looking at the same source compute the same scope, and the mapping table needs no coordination protocol. Per-platform canonical locators, chosen for stability across reinstall: + +| Platform | Source | Canonical locator | Notes | +| --- | --- | --- | --- | +| iOS | camera roll / screenshots / user collection | the smart-album subtype name, or the user collection's title-independent `localIdentifier` | `localIdentifier` survives reinstall on the same device; cross-device it maps only via the override table | +| Android | MediaStore bucket | **relative path** (`DCIM/Camera`, `Pictures/Screenshots`) | never `BUCKET_ID` — it is a hash of the display name that differs across devices and OS versions | +| Desktop | folder / watched dir | library-relative or absolute canonicalized path (symlinks resolved) | | +| Any | removable volume | volume UUID + relative path | the volume UUID makes a re-mounted card the same scope regardless of mount point | + +- **The mapping table lives in the [library-settings document](/design/metadata/#how-operations-travel)** (`scope_id → album_id` rows) — per-owner, E2E-encrypted, synced across devices with the same CRDT semantics as other collaborative metadata (each row an LWW register keyed by `scope_id`). The server never learns what a scope means. +- **Resolution order** (first match wins): explicit user pick at import time → `scope_id` override row → per-source-kind default row (e.g. "all screenshots → Screenshots") → the owner's `default_album_id` pointer → the derived de facto album. Deterministic by construction; the planner records which rule fired so a surprising destination is explainable after the fact. +- **Unmapped sources ask once.** The first import from a new scope surfaces a "where should photos from *X* go?" choice, whose answer is written as the scope's override row — automated imports never silently invent destinations. + ### System & Smart Albums (Views) View albums are organizational surfaces computed entirely client-side over the assets the user can already decrypt (the union of their container-album memberships), materialized by querying the [local index](/design/filesystem/client/#local-index-staleness). The [aggregated federated album](/design/federation/#federated-shared-albums-aggregated-albums) is a view in exactly this sense — its predicate is an album-group id spanning constituents on different home servers. A view is **not** an MLS group, holds **no** AMK, **owns no assets**, and is **not** a sharing or access-control boundary — sharing happens only at the container tier. Two kinds: From 435b42460abe84e2e462f8b30bd0f8070e93061c Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 20:07:55 -0400 Subject: [PATCH 30/58] docs: reconcile SLICES.md with the finalized contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The design-review pass changed several inter-slice contracts and created work that had no home; SLICES.md is the executable index, so it moves in lockstep: - Twelve new slices: S-B4 staged uploads; S-C13 cohort storage; S-D7 SDK auth/session foundation + auto token refresh (the SDK owns the complete user flow — stated as the Lane D preamble); S-D8 spargen REST client (blocked-external: our OpenAPI 3.1 generator, no schema downgrades); S-D9 capsule-sdk uniffi FFI for Swift/Kotlin/Linux; S-D10 adverse-network hardening; S-D11 cohort emission; S-D12 recovery cadence; S-D13 culling UX; S-D14 local-gallery gates; S-E4 aggregated federated albums; S-H4 group evaluations (post-v1, sequencing contract fixed now). - Contract rewrites on existing rows: S-C1 (strictness-table tests, discard floor, CAS finalize, manifest+metadata visibility gate, error codes), S-C2 (original_held feed fact), S-C12 (single-active-escrow replace), S-D1 (hand-written client — the stateful protocol is a poor codegen target; recovery matrix driven by error.* codes), S-D2 (networking.md taxonomy), S-D5 (+S-D7), S-Z1 (scope grammar now specified; remainder narrowed). - New 'In-House and External Library Gates' section so slices gated on rawshift (RAW decode in S-B1/B2), spargen (S-D8), geocoordinates-rs (display conversions), and openmls (Lane X) are visible as such instead of failing their Done-when mysteriously. - Dependency graph extended accordingly; the in-tree skeleton census now lists the seams this review added (upload contract types, sidecar registers, cohort + recovery-verify functions, policy/tier/class enums, original_held). --- SLICES.md | 279 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 255 insertions(+), 24 deletions(-) diff --git a/SLICES.md b/SLICES.md index 265035f0..48a145fa 100644 --- a/SLICES.md +++ b/SLICES.md @@ -48,7 +48,13 @@ everything else networked is skeleton or legacy (below). `streaming_recommended` and `release_is_safe` predicates implemented); `capsule-api-media::{verify, drops}` + `capsule-api-auth::devices` route stubs; `capsule-api-upload::envelope` gate stub; the `capsule.sync.v1` proto + `SyncFeedService` -stub. Each names its slice. +stub. The design-review pass added: the upload server's contract-complete session/ +request types + append-only store + `error.upload.*` taxonomy (enforcement = S-C1); +sidecar `stack_membership: Lww>`, `cull`, and `hidden` (implemented + +tested); `capsule_core::cohort` and `capsule_core::backup::verify_recovery_secret` +(implemented + tested); `UploadPolicy`/`UploadTier`, `capsule-sdk::net` +(`ConnectionClass`/`RetryClass`), and `original_held` on the sync proto. Each names +its slice. ## Slice index @@ -63,6 +69,7 @@ stub. Each names its slice. | S-B1 | Thumbnail/LQIP generation | media/import | — | L | ready | | S-B2 | Signed-path import-executor rewrite | media/import | S-B1 | L | ready | | S-B3 | Streaming import (probe, `total_size`, drive mode) | media/import | S-D1, S-D4 | L | ready | +| S-B4 | Staged uploads (low-data tier ladder) | media/import | S-C1, S-C2, S-D1 | M | ready | | S-C1 | Upload-server hardening (envelope gate + invariants) | server | — | L | ready | | S-C2 | Key-free sync feed | server | S-C1 | L | ready | | S-C3 | Storage-verification endpoint | server | — | M | ready | @@ -75,15 +82,25 @@ stub. Each names its slice. | S-C10 | Key-free media serving conformance | server | — | M | ready | | S-C11 | Refcount GC + retention purge worker | server | S-C1 | M | ready | | S-C12 | Backup escrow server surface | server | — | S | ready | -| S-D1 | SDK upload client (+ OpenAPI regen) | sdk/clients | S-C1 | M | ready | +| S-C13 | Session device-cohort storage + grouping | server | — | S | ready | +| S-D1 | SDK upload client (hand-written, stateful protocol) | sdk/clients | S-C1 | M | ready | | S-D2 | SDK sync/download client + connection-class budget | sdk/clients | S-C2, S-C9 | L | ready | | S-D3 | Web guest drop client (WASM) | sdk/clients | S-A6, S-C5 | L | ready | | S-D4 | Verify-before-destroy wiring | sdk/clients | S-C3 | M | ready | | S-D5 | CLI auth/sync/list | sdk/clients | S-D1, S-D2 | M | ready | | S-D6 | Web server gateway (key-free reads) | sdk/clients | S-D2 | L | ready | +| S-D7 | SDK auth/session foundation + auto token refresh | sdk/clients | — | M | ready | +| S-D8 | spargen REST client integration | blocked-external | in-house spargen | M | blocked | +| S-D9 | capsule-sdk uniffi FFI bindings | sdk/clients | S-F1, S-D7 | M | ready | +| S-D10 | Adverse-network hardening | sdk/clients | S-D1, S-D2 | M | ready | +| S-D11 | Client cohort emission + devices grouping UI | sdk/clients | S-C13, S-D7 | M | ready | +| S-D12 | Recovery verification cadence + guided re-wrap | sdk/clients | S-C12 | M | ready | +| S-D13 | Culling workflow client UX | sdk/clients | — | M | ready | +| S-D14 | Local-gallery security gates | sdk/clients | — | S | ready | | S-E1 | Share-link end-to-end serving | fed/sharing | S-C4 | M | ready | | S-E2 | Federation capabilities + pulls | fed/sharing | S-C2, S-A3 | L | ready | | S-E3 | LAN peering | fed/sharing | S-D2, S-C7 | L | ready | +| S-E4 | Aggregated federated albums (album-group view) | fed/sharing | S-E2, S-D2 | L | ready | | S-F1 | uniffi consolidation (0.29 catalog vs 0.31 core) | platform/FFI | — | M | ready | | S-F2 | Secure Enclave / StrongBox hybrid composition | platform/FFI | S-A4, S-F1 | L | ready | | S-F3 | Xcode/Gradle binding wiring + on-device CI | platform/FFI | S-F2 | L | ready | @@ -96,6 +113,7 @@ stub. Each names its slice. | S-H1 | Embeddings + sqlite-vec index | ML | — | L | ready | | S-H2 | Model registry + version regen | ML | S-H1 | M | ready | | S-H3 | Semantic/face features | ML | S-H1 | L | ready | +| S-H4 | Group-scoped evaluations (best shot/framing/exposure) | ML | S-H3 | M | post-v1 | | S-X1 | OpenMLS backend → `OpenMlsAuthority` | blocked-external | upstream | L | blocked | | S-X2 | MLS membership + Welcome/history delivery | blocked-external | S-X1 | L | blocked | | S-X3 | Album upgrade ceremony + MLS resilience | blocked-external | S-X2 | L | blocked | @@ -131,8 +149,34 @@ graph LR H1[S-H1 embeddings] --> H2[S-H2 registry] H1 --> H3[S-H3 semantic/face] --> G3 X1[S-X1 openmls] --> X2[S-X2 membership] --> X3[S-X3 upgrade ceremony] + C1 --> B4[S-B4 staged uploads] + C2 --> B4 + D1 --> B4 + D1 --> D10[S-D10 adverse-net] + D2 --> D10 + C13[S-C13 cohort store] --> D11[S-D11 cohort client] + D7[S-D7 sdk auth] --> D11 + D7 --> D9[S-D9 sdk ffi] + F1 --> D9 + C12[S-C12 escrow] --> D12[S-D12 recovery cadence] + E2 --> E4[S-E4 aggregated albums] + D2 --> E4 + H3 --> H4[S-H4 group evaluations] ``` +## In-House and External Library Gates + +Some slices depend on libraries that are ours but not yet stable, or on upstream +projects. A gated slice can start its non-gated parts; its "Done when" cannot fully +pass until the gate lifts. + +| Library | Status | Gates | +| --- | --- | --- | +| `rawshift` (in-house RAW decode; git submodule, alpha, consumed by nothing yet) | stabilizing | Full RAW support in S-B1/S-B2. v1 ships the zune-jpeg format set; the `media::image::formats::raw` stub is the integration point. | +| `spargen` (in-house OpenAPI **3.1** client generator) | in development | S-D8 (typed REST client + `AuthenticatedClient` revival). Progenitor is gone — we do not downgrade schemas to 3.0. | +| `geocoordinates-rs` (in-house WGS-84 ↔ GCJ-02/BD-09 conversions) | planned | The deterministic client-side coordinate conversion named in [Metadata — Geolocation](capsule-docs/src/content/docs/design/metadata.md); consumed by map display and S-H3 geo features. Until it lands, WGS-84 storage is unaffected (conversion is display-only). | +| `openmls` ciphersuite `0x004D` | blocked upstream | S-X1 → S-X2 → S-X3 (tracked in Lane X). | + ## Lane A — core crypto ### S-A1 — Wrapped file-key mode @@ -252,6 +296,23 @@ graph LR probe test flips. - **Tier:** Unit (auto-detect) + Smoke (release gating, halt-on-disconnect). +### S-B4 — Staged uploads (low-data tier ladder) + +- **Contract:** [Download & Sync — Upload Tiering (Staged Uploads)](capsule-docs/src/content/docs/design/import/download-sync.md); + seams: `UploadPolicy`/`UploadTier` in `capsule-core::import::upload`, `original_held` + on the sync proto. +- **Deliverable:** the client-side staged scheduler — sessions open T0 (manifest + + metadata w/ LQIP) → T1 (thumb + preview) → T2 (original) per asset, T2 gated on the + large-reconciliation criteria; the `awaiting-original` derived state end-to-end + (badge UX, `error.blob.pending_upload` handling, GC carve-out server-side); tier + queue re-derived from server truth on resume. Zero server mode branches by + construction — the policy is session ordering only. +- **Depends on:** S-C1 (visibility gate + `original_held` derivation), S-C2 (feed + field), S-D1 (upload client). **Done when:** the download-sync doc's staged + Validation bullets pass (ladder order, awaiting-original semantics, release gate, + resume-from-server-truth, staged×streaming exclusion). +- **Tier:** Unit + Smoke. + ## Lane C — server (key-free surfaces) ### S-C1 — Upload-server hardening @@ -260,13 +321,25 @@ graph LR [Validation invariants 1–15](capsule-docs/src/content/docs/design/threat-model/validation.md). - **Deliverable:** the `EnvelopeGate` (skeleton in `capsule-api-upload/src/envelope.rs`) wired ahead of every write with `capsule_core::validation::protocol_gate` + - `check_manifest_envelope` (already implemented and unit-tested in core), the - idempotency tuples, the finalization transaction ordering, the startup scrub, and - `error.*` codes on every rejection. -- **Done when:** invariants 1–15 each have a rejecting test against the real server - (testcontainer Postgres); the upload doc's session-lifecycle smoke passes; crash - injection between rename and commit recovers per the atomicity invariants. -- **Tier:** Unit + Smoke + E2E case 2/11. **Blocks:** S-C2, S-C5, S-C11, S-D1. + `check_manifest_envelope` (already implemented and unit-tested in core) plus the + top-level↔envelope consistency check (`error.upload.envelope_mismatch`); the + idempotency machinery (tuple replay store returning byte-identical duplicate + responses; create-dedup returning the active session / `409 duplicate_blob` per the + doc's Idempotency and Resumption section, in one `SELECT…FOR UPDATE` transaction); + the atomic status CAS on finalization + the finalization transaction ordering; the + visibility gate on **manifest + metadata** finalization with `original_held` + derivation (staged-uploads contract); the discard machinery (≥1 h progress floor, + pressure eviction least-recently-progressed-first, startup scrub of orphan + `incoming/*.bin` + length-diverged sessions); the uploader-scoped session index; and + the `error.upload.*` code on every rejection (constants already generated). +- **Done when:** invariants 1–15 (as amended) each have a rejecting test against the + real server (testcontainer Postgres + Valkey) asserting status **and** `error.*` + code; every row of the upload doc's Strictness Table has a test; the + session-lifecycle smoke passes; the discard-floor test passes (progress within 1 h + is never evicted under injected pressure); crash injection between append and + counter-increment, and between rename and commit, recovers per the atomicity + invariants. +- **Tier:** Unit + Smoke + E2E case 2/11. **Blocks:** S-C2, S-C5, S-C11, S-D1, S-B4. ### S-C2 — Key-free sync feed @@ -275,8 +348,10 @@ graph LR [API Surfaces](capsule-docs/src/content/docs/design/api-surfaces.md). - **Deliverable:** `SyncFeedService` implemented — per-album `sync_seq` minted in the finalization transaction, the HMAC'd opaque cursor (invariant 22), entries carrying - the manifest as opaque CBOR + metadata blob + blob refs; gRPC metadata negotiation - per the api-surfaces mapping; the salvo↔tonic bridge verified end-to-end. + the manifest as opaque CBOR + metadata blob + blob refs + the `original_held` + completeness fact (proto field in-tree; staged-uploads contract); gRPC metadata + negotiation per the api-surfaces mapping; the salvo↔tonic bridge verified + end-to-end. - **Depends on:** S-C1. **Blocks:** S-C8, S-D2, S-E2, S-G1, S-G2. - **Done when:** the download-sync doc's sync-feed Validation bullets (monotonicity, forward-version rejection, rewind rejection, cursor authenticity) pass server-side. @@ -380,23 +455,52 @@ graph LR ### S-C12 — Backup escrow server surface - **Contract:** [Backup — Master-Key Escrow](capsule-docs/src/content/docs/design/backup-recovery.md). -- **Deliverable:** store/fetch of the wrapped master-key escrow blob (opaque to the - server; the wrap format is implemented in core), with the ≥128-bit recovery-secret - rule surfaced client-side. +- **Deliverable:** store/fetch/**replace** of the wrapped master-key escrow blob + (opaque to the server; the wrap format is implemented in core) — replace is + single-active-escrow: the old blob is deleted in the same transaction, per the + guided re-wrap contract in the backup doc — with the ≥128-bit recovery-secret rule + surfaced client-side. - **Done when:** escrow round-trips through the server and unwraps with the passphrase - path already tested in core. **Tier:** Smoke. + path already tested in core; after a replace, the prior blob is gone and unwraps + nothing. **Tier:** Smoke. **Blocks:** S-D12. + +### S-C13 — Session device-cohort storage + grouping + +- **Contract:** [Authentication — Device Cohorts](capsule-docs/src/content/docs/design/authentication.md); + pure hash in `capsule_core::cohort` (implemented + tested). +- **Deliverable:** accept the advisory `cohort_hash` in the session-creation body, + store it on the session record + the durable `device_cohorts(user_id, cohort_hash, + first_seen, last_seen)` map, and surface both through the session listing. + Advisory-only invariant enforced structurally: no authorization path reads it. +- **Done when:** the authentication doc's cohort Validation bullets pass (advisory + behavior under absent/garbage values; grouping; durable map outlives sessions). +- **Tier:** Unit + Smoke. **Blocks:** S-D11. ## Lane D — SDK / clients +`capsule-sdk` is the **sanctioned network path**: it owns the session/token store and +auto refresh (S-D7), the complete user-flow primitives (login → upload → status → +sync), and their FFI exposure to Swift/Kotlin/Linux (S-D9). Native apps consume the +SDK; they never hand-roll network flows. + ### S-D1 — SDK upload client - **Contract:** [Import — Upload Protocol](capsule-docs/src/content/docs/design/import/upload-protocol.md); the `todo!()` stubs in `capsule-sdk/src/upload.rs`. -- **Deliverable:** regenerate `openapi.json` against the hardened server (S-C1) and - implement the chunked, resumable, adaptive upload client (the `X-Capsule-*` headers - are registered in the validation doc's header census). -- **Depends on:** S-C1. **Blocks:** S-B3, S-D5. **Done when:** the upload doc's - client-side Validation bullets pass against a real server; E2E case 2 lives. +- **Deliverable:** the hand-written chunked, resumable, adaptive upload client — the + protocol is too stateful for codegen; the spargen-generated REST client (S-D8) + covers the plain request/response surfaces instead. Implements create/PATCH/HEAD/ + DELETE/list with `application/octet-stream`, the **required** `X-Capsule-Checksum` + (lowercase-hex SHA-256), `X-Capsule-Offset`, and the handshake headers; the + adaptive algorithm per the doc (normative), clamped to the protocol bounds + `[PROTOCOL_MIN_CHUNK, PROTOCOL_MAX_CHUNK]` with alignment guaranteed by + construction; and the code-driven recovery matrix (`offset_mismatch` → HEAD + re-align; `session_not_found` → re-create; `duplicate_blob` → merge; `426` → + abort-with-upgrade; `checksum_mismatch` → re-send) — clients switch on `error.*` + codes, never bare statuses. +- **Depends on:** S-C1. **Blocks:** S-B3, S-B4, S-D5. **Done when:** the upload doc's + client-side Validation bullets pass against a real server; the recovery matrix has + a mocked-HTTP test per code; E2E case 2 lives. - **Tier:** Unit + Smoke. ### S-D2 — SDK sync/download client @@ -405,7 +509,10 @@ graph LR - **Deliverable:** the gRPC sync consumer (cursor high-water marks, per-album `sync_seq` anti-rewind, forward-version rejection), tiered on-demand fetch with the degrade ladder (403-as-authorization-change), resumable ranged blob fetch, and the - connection-class detection that feeds the cache-eviction byte budget. + connection-class detection (taxonomy owned by + [Networking](capsule-docs/src/content/docs/design/networking.md); `ConnectionClass` + seam in `capsule-sdk::net`) that feeds the cache-eviction byte budget and the + staged-upload tier gates. - **Depends on:** S-C2, S-C9. **Blocks:** S-D5, S-D6, S-E3, S-G1, S-G2. - **Done when:** the download-sync doc's client Validation bullets pass; E2E case 3 lives. **Tier:** Unit + Smoke. @@ -435,7 +542,7 @@ graph LR - **Contract:** [Clients](capsule-docs/src/content/docs/design/clients.md) (the CLI is a client). - **Deliverable:** the `todo!()` CLI commands (`auth login/logout`, `sync`, `list`) over the SDK clients. -- **Depends on:** S-D1, S-D2. **Done when:** `capsule auth login && capsule sync && +- **Depends on:** S-D1, S-D2, S-D7 (the token store — the CLI never hand-rolls auth). **Done when:** `capsule auth login && capsule sync && capsule list` round-trips against a dev server. **Tier:** Smoke. ### S-D6 — Web server gateway @@ -450,6 +557,99 @@ graph LR retirement precondition). **Done when:** the gateway methods run against a dev server with the mock gateway deleted. **Tier:** Smoke (`mise run check-web` + bun tests). +### S-D7 — SDK auth/session foundation + auto token refresh + +- **Contract:** [Authentication — Session and Access Tokens](capsule-docs/src/content/docs/design/authentication.md); + the parked `AuthenticatedClient` shape in `capsule-sdk/src/lib.rs`. +- **Deliverable:** the SDK-owned session/token store and refresh engine — a quick + asynchronous pre-flight check on token expiry before each request, single-flight + refresh, 401-retry-once — hand-rolled `reqwest` against the real `capsule-api-auth` + endpoints (no spargen dependency), exposing the login → authenticated-call → logout + primitives. This is the "SDK owns the complete user flow" foundation: native apps + never juggle raw tokens. +- **Done when:** login/refresh/expiry flows round-trip against a dev server; a mocked + clock exercises pre-flight refresh + single-flight; `capsule-sdk` stays in every + Rust gate. **Tier:** Unit + Smoke. **Blocks:** S-D9, S-D11; S-D5 consumes it. + +### S-D8 — spargen REST client integration + +- **Contract:** [API Surfaces — Why Two Transports](capsule-docs/src/content/docs/design/api-surfaces.md); + the parked wrapper in `capsule-sdk/src/lib.rs`. +- **Blocked on:** in-house `spargen` (OpenAPI 3.1 client generator) reaching usable + stability **and** the server's OpenAPI schema stabilizing post-S-C1. Unblock check: + spargen repo milestones; re-evaluate monthly. +- **Deliverable:** generate the typed REST client from the OpenAPI 3.1 schema (no 3.0 + downgrade, ever), revive `AuthenticatedClient` over it (composing S-D7's token + store), and delete the parked comment blocks. + +### S-D9 — capsule-sdk uniffi FFI bindings + +- **Contract:** [Clients](capsule-docs/src/content/docs/design/clients.md), + [Module Map](capsule-docs/src/content/docs/design/module-map.md) (`capsule-sdk` row). +- **Deliverable:** the uniffi surface over `capsule-sdk`'s user-flow primitives + (login, upload file, upload/sync status, sync) so iOS/macOS (Swift), Android + (Kotlin), and Linux consumers call one SDK instead of reimplementing flows — + async-capable bindings, sharing the single-uniffi-version strategy S-F1 lands; + binding generation joins `gen-bindings`/`verify-examples`. +- **Depends on:** S-F1, S-D7. **Done when:** Swift + Kotlin harnesses drive a + login→upload→status round-trip against a dev server through the bindings. +- **Tier:** Smoke per platform. + +### S-D10 — Adverse-network hardening + +- **Contract:** [Networking — Adverse-Network Posture](capsule-docs/src/content/docs/design/networking.md); + `ConnectionClass`/`RetryClass` seams in `capsule-sdk::net`. +- **Deliverable:** behavioral `adverse` promotion/demotion (reset/stall counters over + a sliding window), stall-detection cuts (no-bytes-for-T) with offset/Range resume, + bounded transfer windows under `adverse`, chunk-size floor coupling, Happy Eyeballs + at dial, and the three retry policy classes as a shared engine the sync/upload/fetch + paths instantiate. +- **Depends on:** S-D1, S-D2. **Done when:** the networking doc's four Validation + bullets pass (mocked-signal class matrix; promotion/demotion; stall-cut-resume with + zero duplicate bytes; backoff discipline). **Tier:** Unit + Smoke. + +### S-D11 — Client cohort emission + devices grouping UI + +- **Contract:** [Authentication — Device Cohorts](capsule-docs/src/content/docs/design/authentication.md). +- **Deliverable:** per-platform primary-identifier readers (Keychain seed / SSAID / + IOPlatformUUID / MachineGuid / hashed machine-id), `cohort_hash` emission at session + creation, the grouped devices view with assert-don't-litigate copy, and the one-tap + support bundle (`cohort_hash` + device-id/session map). +- **Depends on:** S-C13, S-D7. **Done when:** a reinstall groups with "previously + used" in the devices view; the support bundle round-trips. **Tier:** Unit + Smoke. + +### S-D12 — Recovery verification cadence + guided re-wrap + +- **Contract:** [Backup — Recovery Verification Cadence](capsule-docs/src/content/docs/design/backup-recovery.md); + `capsule_core::backup::verify_recovery_secret` (implemented + tested). +- **Deliverable:** the escrow-blob cache + refresh, the cadence scheduler + (7 d → 90 d → 180 d, re-arm triggers, snooze caps, never-blocking), the + verification prompt UX, and the guided re-wrap flow (new secret, same master key, + escrow replace via S-C12, Shamir re-issue, old-artifact guidance). +- **Depends on:** S-C12. **Done when:** the backup doc's cadence Validation bullets + pass (mocked clock; stale-cache rule; re-wrap smoke with unchanged blob hashes). +- **Tier:** Unit + Smoke. + +### S-D13 — Culling workflow client UX + +- **Contract:** [Organization — Culling](capsule-docs/src/content/docs/design/organization.md) + (schema landed: sidecar `cull` LWW register). +- **Deliverable:** the keyboard/swipe-driven review mode writing `cull` flags, + flag-filtered views, derived group cull state, and the reject-sweep (batch-move to + trash — the only destructive step, soft per retention). +- **Done when:** the flag → filter → sweep loop round-trips on a fixture library; + concurrent flags from two devices converge. **Tier:** Unit + Smoke. + +### S-D14 — Local-gallery security gates + +- **Contract:** [Local Gallery — Security Requirements](capsule-docs/src/content/docs/design/local-gallery.md). +- **Deliverable:** the fresh-local-auth gate (biometric → credential fallback, per-view + 5-minute grace) on the Recently Deleted and Hidden views, and the cache/temp + placement audit asserting no plaintext lands outside the library root. +- **Done when:** the local-gallery doc's unit Validation bullets pass; the NFR1 + no-network-on-read-paths smoke runs with a socket-refusing harness. +- **Tier:** Unit + Smoke; the airplane-mode E2E case rides the Module Map surface. + ## Lane E — federation / sharing ### S-E1 — Share-link end-to-end serving @@ -482,6 +682,23 @@ graph LR devices). **Done when:** the peering doc's six Validation bullets pass; E2E case 5 lives. **Tier:** Unit + Smoke per platform. +### S-E4 — Aggregated federated albums (album-group view) + +- **Contract:** [Federation — Federated Shared Albums](capsule-docs/src/content/docs/design/federation.md) + (assertion schema), [Organization — views](capsule-docs/src/content/docs/design/organization.md). +- **Deliverable:** the `AlbumGroupAssertion` write/merge on the collaborative-metadata + op path, group-aware invites (group_id + sibling hints riding the existing album + invite), the aggregate view renderer (member-of ∧ asserts-group inclusion rule, + capture-time ordering, per-origin partial-view indicator), leave = assertion + removal (+ optional unshare), per-origin moderation drop. Zero new server surface — + buildable against `ReferenceAuthority` fixtures while MLS membership (S-X2) is + blocked; user-facing multi-user invites ride S-X2 (same caveat as organization's + invitation surface). +- **Depends on:** S-E2 (cross-server read path), S-D2 (feed consumer). **Done when:** + the federation doc's aggregated-album Validation bullets pass (composition, + injection-refusal, partial view, leave propagation, LWW rename convergence). +- **Tier:** Unit + Smoke. + ## Lane F — platform / FFI ### S-F1 — uniffi consolidation @@ -588,6 +805,18 @@ its design here. InsightFace-AdaFace) on the deterministic execution path with the platform-partition fallback; `tags_ai` population. **Depends on:** S-H1. **Blocks:** S-G3. +### S-H4 — Group-scoped evaluations (post-v1) + +- **Contract:** [AI — Quality Assessment](capsule-docs/src/content/docs/design/ai.md) + (sequencing contract fixed now; category deferred post-v1). +- **Deliverable:** best-shot/framing/exposure scoring over stacks, bursts, and + similarity groups — strictly **after** grouping, keyed + `(group_id, membership_hash, model_id, model_version)` so any regroup invalidates + by key construction; deterministic recompute (ties broken by asset id); results as + AI-namespaced derived state feeding the `role = primary` suggestion. +- **Depends on:** S-H3 (and the stacking surfaces). **Status: post-v1** — indexed now + so the sequencing contract has an owner; not part of the v1 cut. + ## Lane X — blocked on upstream ### S-X1 — OpenMLS backend → `OpenMlsAuthority` @@ -620,5 +849,7 @@ its design here. - **Contract:** [Metadata — How Operations Travel](capsule-docs/src/content/docs/design/metadata.md), [Organization — views](capsule-docs/src/content/docs/design/organization.md). - **Deliverable:** the concrete schema for the per-owner E2E-encrypted library-settings - document (smart-album definitions, scope-override map) as a design-doc addition — - the docs declare the surface but not its fields. + document (smart-album definitions; per-viewer aggregated-album cover preferences) as + a design-doc addition. The scope-override map's rows and grammar are now specified + in [Organization — Scope Grammar](capsule-docs/src/content/docs/design/organization.md); + what remains is the smart-album predicate schema and the document's envelope/versioning. From 909010d364238fd01c7d3a23dfdf051182a3b2e7 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 20:08:53 -0400 Subject: [PATCH 31/58] chore: remove dead GraphQL artifacts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two leftovers from before the REST/gRPC decision that read as if GraphQL were still a plan rather than a retiring legacy: an empty capsule-api/graphql/ directory (only a stray .DS_Store — the real legacy crate is capsule-api-library, correctly frozen behind slice S-G1), and a capsule-web comment sketching the future server gateway as 'likely GraphQL for library queries'. The comment now states the committed design: rich queries have no server surface at all — the gateway is the browser's sync-fed client-side read path (S-D6) plus ranged REST blob fetches. (Local biome/rspack web tooling is broken on this machine independent of this change — verified identical failure on a clean tree; the edit is comment-only.) --- capsule-web/src/data/server/server-gateway.ts | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/capsule-web/src/data/server/server-gateway.ts b/capsule-web/src/data/server/server-gateway.ts index 1143d4e9..5a3e3150 100644 --- a/capsule-web/src/data/server/server-gateway.ts +++ b/capsule-web/src/data/server/server-gateway.ts @@ -5,10 +5,12 @@ import type { CapsuleGateway } from '../gateway'; * * The capsule-api server is mid-rewrite to the end-to-end-encrypted, key-free * model described in the design docs, so there is no stable schema to implement - * against yet. When there is, this adapter implements `CapsuleGateway` over the - * high-level capsule-sdk / server APIs (likely GraphQL for library queries plus - * REST for content-addressed blobs), and ../index.ts selects it (e.g. behind a - * PUBLIC_DATA_SOURCE env flag). + * against yet. When there is, this adapter implements `CapsuleGateway` as the + * browser's client-side read path (slice S-D6 in the repo-root SLICES.md): a + * sync-fed local store queried in the client — the web analogue of + * library.sqlite — plus ranged REST fetches for content-addressed blobs. There + * is no GraphQL: rich queries have no server surface at all (api-surfaces + * design doc). ../index.ts selects it (e.g. behind a PUBLIC_DATA_SOURCE flag). */ const notImplemented = (): never => { throw new Error( From e2a779e32595cc88420977d99288d528d876f6e2 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sat, 4 Jul 2026 20:32:04 -0400 Subject: [PATCH 32/58] docs(design): final consistency pass across the revised set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh full read of all 43 docs + SLICES.md after the revision series, fixing what it surfaced (the set was otherwise consistent: all 187 cross-doc anchors resolve, the six known overlap clusters are link-not-restate factored, every slice reference exists): - Terminology: 'backup' stays reserved for the encrypted export artifact — three drifted uses reworded (embeddings 'server-side backup', envelope 'key-free backup' fallback, staleness-nag phrasing). - Pre-append-only leftovers: 'assembled blob' comments in the blob-store layout and storage-verification reworded; the flat '24-hour TTL' in the deployment profiles now names itself as the cap and points at the floor/discard owner. - The wrapped-key passage in keys.md restated ~4 sentences of encryption.md's normative KDF discipline; trimmed to the hierarchy fact + authorization neutrality with pointers (the one substantial restatement the redundancy audit found). - The deprecated X-Capsule-Upload-Protocol alias is removed outright rather than kept 'accepted but deprecated': nothing shipped speaks it, it was absent from the header census, and an unregistered alias kept alive is exactly the hidden behavior the strict protocol forbids. - The server index-rebuild rule gains the staged-uploads carve-out (a missing original on an awaiting-original asset is expected state, not a dangling reference). --- .../src/content/docs/design/cryptography/encryption.md | 2 +- capsule-docs/src/content/docs/design/cryptography/keys.md | 2 +- capsule-docs/src/content/docs/design/filesystem/server.md | 8 ++++---- .../src/content/docs/design/import/download-sync.md | 2 +- .../content/docs/design/import/storage-verification.md | 2 +- .../src/content/docs/design/import/upload-protocol.md | 2 +- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/capsule-docs/src/content/docs/design/cryptography/encryption.md b/capsule-docs/src/content/docs/design/cryptography/encryption.md index 2c77f63e..3f70e87c 100644 --- a/capsule-docs/src/content/docs/design/cryptography/encryption.md +++ b/capsule-docs/src/content/docs/design/cryptography/encryption.md @@ -94,7 +94,7 @@ Not all metadata can be encrypted — some must stay server-readable for routing - **Encrypted** (AES-256-GCM under a key derived from the album's AMK, fresh random nonce per blob): the CBOR sidecar / metadata blobs — including the [chromahash LQIP](/design/thumbnails/#lqip) and `dominant_color`, so image-derived display hints never leak to a server that never decodes assets. Each blob is independently versioned and signed like an [asset manifest](/design/cryptography/provenance/#asset-manifest). - **Server-plaintext by necessity:** `owner_id`, the [ciphertext content hash](/design/cryptography/primitives/), and the ciphertext size — the routing and storage-accounting facts a key-less server needs. This is a deliberate, documented trade-off. -- **AI embeddings** (semantic-search vectors, face embeddings) are sensitive — a user can be re-identified from them. They are kept plaintext *locally* (vector search requires it) but encrypted at rest in the server-side backup. +- **AI embeddings** (semantic-search vectors, face embeddings) are sensitive — a user can be re-identified from them. They are kept plaintext *locally* (vector search requires it) but always encrypted at rest on the server. CBOR metadata blobs use **deterministic encoding** per the [canonical CBOR ruleset](/design/metadata/#canonical-cbor-encoding) owned by [Metadata](/design/metadata/) — the same byte-exact rules the plaintext sidecar follows, since the metadata blob's plaintext *is* that CBOR document. Because a blob's hash is what content-addresses it and what the [signed manifest](/design/cryptography/provenance/#asset-manifest) commits to, two implementations encoding the same logical metadata must produce byte-identical output — otherwise the hash diverges and the signature fails to verify across [federated](/design/federation/) peers. Conformance to the canonical ruleset is mandatory and is the load-bearing check behind cross-platform and cross-language interop. diff --git a/capsule-docs/src/content/docs/design/cryptography/keys.md b/capsule-docs/src/content/docs/design/cryptography/keys.md index f98607ee..129f7397 100644 --- a/capsule-docs/src/content/docs/design/cryptography/keys.md +++ b/capsule-docs/src/content/docs/design/cryptography/keys.md @@ -33,7 +33,7 @@ Construction rules (consistent with the [KDF choice](/design/cryptography/primit - The 512-bit KDF output is truncated to 32 bytes for the AES-256 file key. - Each **encryption** gets a fresh derived key: the per-encryption `nonce_prefix` is folded into the `salt` (`file_id || nonce_prefix`; see [Encryption — Asset Key Derivation](/design/cryptography/encryption/#asset-key-derivation)), so even a same-`file_id` [`replace`](/design/authorization/#the-closed-action-set) under the same epoch re-rolls the key, never merely the nonce. The STREAM nonce can therefore safely start at zero per encryption, and no `(file_key, nonce_prefix)` pair is ever reused. -**Two file-key modes.** The derivation above is the **derived** mode — the default for every asset a client imports itself, where no per-file key is stored because the file key is recomputed from the AMK on demand. A second, **wrapped** mode exists for assets whose bytes a client did *not* author and whose key it therefore cannot re-derive: a [web-upload drop](/design/web-upload/) the user adopts in place arrives encrypted under a random key `K` the guest chose, so the adopting client *carries* `K` wrapped under the AMK rather than deriving it. The wrap key follows the same KDF discipline as the file key (the `asset-keywrap/v1` derivation, owned by [Encryption — Asset Key Derivation](/design/cryptography/encryption/#asset-key-derivation)), with a fresh `wrap_nonce` folded into the salt so no `(wrap_key, wrap_nonce)` pair repeats. The mode is recorded in the signed [`key_mode`](/design/cryptography/provenance/#asset-manifest) manifest field; wrapped mode is set only at the adopting `create`, and any later `replace` re-encrypts under derived mode. Both modes feed the same [STREAM construction](/design/cryptography/encryption/#stream-construction) and the same [`verify_asset`](#write-authorization) authorization checks — the mode changes only how the decryption key is obtained, never who is authorized to write. +**Two file-key modes.** The derivation above is the **derived** mode — the default, where no per-file key is stored because the file key is recomputed from the AMK on demand. The **wrapped** mode is the one stored-key exception in the hierarchy, for assets whose bytes a client did not author (an adopted [web-upload drop](/design/web-upload/)): the asset key travels wrapped under the AMK instead of being derived from it. The wrap derivation, its nonce discipline, and the create/replace lifecycle are owned by [Encryption — Asset Key Derivation](/design/cryptography/encryption/#asset-key-derivation); the signed [`key_mode`](/design/cryptography/provenance/#asset-manifest) field records the mode. Either way the mode changes only how the decryption key is obtained — never who is authorized to write ([`verify_asset`](#write-authorization) is mode-blind). The master key also derives one **identifier** — the [default album](/design/organization/#the-default-album)'s `album_id`, via HKDF with a dedicated `info` label — so any device can recompute which album is the de facto default from the master key alone, even after recovery. This derives an *ID*, not a key: the default album is an ordinary album with a random per-epoch AMK like any other. diff --git a/capsule-docs/src/content/docs/design/filesystem/server.md b/capsule-docs/src/content/docs/design/filesystem/server.md index 56875a17..bb3a1185 100644 --- a/capsule-docs/src/content/docs/design/filesystem/server.md +++ b/capsule-docs/src/content/docs/design/filesystem/server.md @@ -14,7 +14,7 @@ The server's durable state is always split across **two required systems** plus - **Blob store** (filesystem) — the encrypted bytes of every asset. *Required.* - **PostgreSQL** — the authoritative index: ownership, album references, blob references, lifecycle state, and (in the default profile) upload-session state. *Required.* -- **Valkey** — volatile upload-session state (offsets, status) with a 24-hour TTL. *Optional.* Recommended only for deployments where upload-session hot-path contention on PostgreSQL becomes measurable. +- **Valkey** — volatile upload-session state (offsets, status); the store's 24-hour TTL is the lifetime **cap** (the ≥1-hour survival floor and pressure-discard semantics are owned by [Upload Protocol — Session Lifetime and Discard](/design/import/upload-protocol/#session-lifetime-and-discard)). *Optional.* Recommended only for deployments where upload-session hot-path contention on PostgreSQL becomes measurable. This gives two concrete deployment profiles: @@ -30,7 +30,7 @@ Switching profiles is operationally invisible to clients — the [upload protoco ```text {blob_root}/ ├── incoming/ -│ └── {upload_id}.bin # assembled blob, pre-verification +│ └── {upload_id}.bin # in-progress append-only upload, pre-verification ├── blobs/ │ └── {hash[0:2]}/{hash[2:4]}/ │ └── {hash} # finalized blob, content-addressed @@ -46,7 +46,7 @@ Switching profiles is operationally invisible to clients — the [upload protoco ## Uniform, Opaque Blobs -A single asset produces a **bundle** of blobs (see [Import — Upload Protocol: What Gets Uploaded](/design/import/upload-protocol/#what-gets-uploaded)): the encrypted original, encrypted derivatives (thumbnails, previews), and the encrypted CBOR metadata blob (which carries the LQIP) — every one of them fully opaque, content-addressed ciphertext the store does not distinguish. Beside them, each write persists its signed **manifest envelope object** (see [Provenance — Physical placement](/design/cryptography/provenance/#asset-manifest)): a small, deliberately server-visible signed CBOR object, stored content-addressed like any blob, whose append-only sequence is the asset's provenance chain. The hot-path mapping from an asset to its blobs and their roles lives in PostgreSQL, with the envelope objects as its durable, key-free backup. +A single asset produces a **bundle** of blobs (see [Import — Upload Protocol: What Gets Uploaded](/design/import/upload-protocol/#what-gets-uploaded)): the encrypted original, encrypted derivatives (thumbnails, previews), and the encrypted CBOR metadata blob (which carries the LQIP) — every one of them fully opaque, content-addressed ciphertext the store does not distinguish. Beside them, each write persists its signed **manifest envelope object** (see [Provenance — Physical placement](/design/cryptography/provenance/#asset-manifest)): a small, deliberately server-visible signed CBOR object, stored content-addressed like any blob, whose append-only sequence is the asset's provenance chain. The hot-path mapping from an asset to its blobs and their roles lives in PostgreSQL, with the envelope objects as its durable, key-free fallback. ## Recovering the Index from Blobs Alone @@ -59,7 +59,7 @@ The server-visible envelope includes: - `created_by_user`, `created_by_device`, `album_id`, `file_id`, `prior_provenance_hash`, `action` — owner, provenance chain link, and lifecycle action - the device's hybrid signature — provenance attribution; verifiable against the public device directory even without any key the server holds -A rebuild walks the **envelope objects** under `blobs/`, verifies each device signature against the cached device directory, and writes index rows for the blobs each envelope names (original and derivatives by `ciphertext_hash` + role, the metadata blob by `metadata_blob_hash`). A ciphertext blob referenced by no envelope surfaces as an orphan for [GC](#deletion-and-garbage-collection); an envelope naming a missing blob surfaces as a dangling reference. The rebuild is idempotent: re-running it against an existing index produces no changes. The full envelope check list a server runs at recovery is the same list it runs at write time — see [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). +A rebuild walks the **envelope objects** under `blobs/`, verifies each device signature against the cached device directory, and writes index rows for the blobs each envelope names (original and derivatives by `ciphertext_hash` + role, the metadata blob by `metadata_blob_hash`). A ciphertext blob referenced by no envelope surfaces as an orphan for [GC](#deletion-and-garbage-collection); an envelope naming a missing blob surfaces as a dangling reference — **except** a missing *original* on an asset whose feed state is [`awaiting-original`](/design/import/download-sync/#upload-tiering-staged-uploads), which is expected staged-upload state, not corruption. The rebuild is idempotent: re-running it against an existing index produces no changes. The full envelope check list a server runs at recovery is the same list it runs at write time — see [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). An envelope object that fails structural validation during rebuild is **quarantined**, not silently dropped — moved to `{blob_root}/quarantine/` with a sibling `.reason.json` recording the rejection code. This guarantees that an unrecoverable byte sequence is preserved for forensic inspection rather than vanishing on rebuild. diff --git a/capsule-docs/src/content/docs/design/import/download-sync.md b/capsule-docs/src/content/docs/design/import/download-sync.md index 74ae1884..bd6d2ba5 100644 --- a/capsule-docs/src/content/docs/design/import/download-sync.md +++ b/capsule-docs/src/content/docs/design/import/download-sync.md @@ -87,7 +87,7 @@ Mobile OSes do not let an app sync whenever it likes; the scheduler is written a ### Notifications -When the auto sync criteria have not been met for a prolonged period — **two weeks** specifically — the library falls silently out of date, which defeats the purpose of a backup. The client surfaces this rather than letting it pass unnoticed: +When the auto sync criteria have not been met for a prolonged period — **two weeks** specifically — the library falls silently out of date, which defeats the point of keeping every device's content safe elsewhere. The client surfaces this rather than letting it pass unnoticed: - After two weeks without a completed sync *while changes remain un-synced* — including originals still pending under a [staged upload policy](#upload-tiering-staged-uploads) — the user is notified that the library is behind and offered a one-tap **force sync now**, which proceeds regardless of the metered/Wi-Fi criteria with their explicit consent. (A device idle long enough to trigger this may also be approaching the session's [sliding inactivity expiry](/design/authentication/#sliding-inactivity-expiry); the force-sync flow routes through re-authentication when the session has lapsed rather than failing the sync.) - The notification can be **snoozed** until a later date (e.g. another two weeks) or **disabled** outright. Snoozing only suppresses the warning; disabling opts out of the warning entirely and does not affect auto sync itself. diff --git a/capsule-docs/src/content/docs/design/import/storage-verification.md b/capsule-docs/src/content/docs/design/import/storage-verification.md index ac1d7923..16cc1af2 100644 --- a/capsule-docs/src/content/docs/design/import/storage-verification.md +++ b/capsule-docs/src/content/docs/design/import/storage-verification.md @@ -4,7 +4,7 @@ description: The endpoint a client calls to confirm an asset is durably stored, status: draft --- -[Upload finalization](/design/import/upload-protocol/#finalization-and-integrity) confirms, **once**, that the bytes the server assembled match the declared hash. That `Completed` acknowledgement is a transfer receipt, not a standing durability guarantee a client can re-check later — and [`verify_asset`](/design/cryptography/keys/#write-authorization) proves *cryptographic* validity and authorization, never that the server still holds the bytes. A client that is about to discard local data therefore has no way, today, to ask the server: *do you actually still have this, indexed and retrievable?* +[Upload finalization](/design/import/upload-protocol/#finalization-and-integrity) confirms, **once**, that the bytes the server stored match the declared hash. That `Completed` acknowledgement is a transfer receipt, not a standing durability guarantee a client can re-check later — and [`verify_asset`](/design/cryptography/keys/#write-authorization) proves *cryptographic* validity and authorization, never that the server still holds the bytes. A client that is about to discard local data therefore has no way, today, to ask the server: *do you actually still have this, indexed and retrievable?* This doc defines that missing query and the rule that every client follows before any destructive local action. The endpoint lives in `capsule-api-media` (planned; it answers from the blob store plus the Postgres index); the client-side gate is a pure predicate in `capsule-core` invoked from `capsule-sdk` (planned — the offline core's `verify_asset` already covers the complementary crypto-validity half of the gate). diff --git a/capsule-docs/src/content/docs/design/import/upload-protocol.md b/capsule-docs/src/content/docs/design/import/upload-protocol.md index 3dcc6a6a..8a1ecb22 100644 --- a/capsule-docs/src/content/docs/design/import/upload-protocol.md +++ b/capsule-docs/src/content/docs/design/import/upload-protocol.md @@ -99,7 +99,7 @@ The upload session is gated by Capsule's universal [protocol handshake](/design/ Versioning is **date-based** (`YYYY-MM-DD` — the day a protocol revision is frozen), not integer or semver. An integer version conveys nothing about ordering granularity and invites a bump for every change; semver implies a minor/patch backward-compatibility contract finer than we are willing to maintain on a hot path. A date is unambiguously ordered, human-readable, and maps directly onto a release. -- Every client sends `X-Capsule-Protocol: ` on every request (the upload-specific alias `X-Capsule-Upload-Protocol` remains accepted but is deprecated). The server advertises the inclusive range it accepts via `X-Capsule-Protocol-Min` and `-Max` on every response, errors included. +- Every client sends `X-Capsule-Protocol: ` on every request. (The old upload-specific alias `X-Capsule-Upload-Protocol` is **gone**, not deprecated — nothing shipped speaks it, and an unregistered header kept alive 'just in case' is exactly the hidden behavior this protocol forbids.) The server advertises the inclusive range it accepts via `X-Capsule-Protocol-Min` and `-Max` on every response, errors included. - A `POST /upload` whose version falls outside the accepted range is rejected with `426 Upgrade Required` *before* any session or pending asset row is created. The response names the supported range so the client can show an actionable message ("update Capsule to keep uploading"). Per [Threat Model](/design/threat-model/validation/#protocol-and-capability-negotiation), the same rule applies to every other write surface. - This is a one-shot **compatibility gate**, not negotiation: there is no back-and-forth to settle on a shared version, and the protocol carries no capability flags. A client either speaks a version the server accepts, or it does not upload. - The server supports a *window* of past protocol versions, not only the newest, so a staggered client rollout keeps working. A version leaves the window only after the deprecation period defined in [Threat Model — Min-Supported-Client Deprecation Policy](/design/threat-model/schema-rules/#min-supported-client-deprecation-policy); dropping one is a breaking change announced ahead of time. From 64c9e8cb8cb6e1727a6bfa2d5ebc1fbb9f85712f Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:52:27 -0400 Subject: [PATCH 33/58] feat(docs): render design-doc status as a title badge The status frontmatter (draft/stable) was schema-validated but invisible on the rendered site. A Starlight PageTitle override now shows it as a badge beside the page title (draft = caution, stable = success). Biome only parses .astro frontmatter, so template-only usage of imports and consts false-positives noUnusedImports/noUnusedVariables; scoped override added per Biome's Astro guidance. --- capsule-docs/astro.config.mjs | 4 +++ capsule-docs/biome.jsonc | 17 +++++++++- capsule-docs/src/components/PageTitle.astro | 35 +++++++++++++++++++++ 3 files changed, 55 insertions(+), 1 deletion(-) create mode 100644 capsule-docs/src/components/PageTitle.astro diff --git a/capsule-docs/astro.config.mjs b/capsule-docs/astro.config.mjs index a8f4404c..61310694 100644 --- a/capsule-docs/astro.config.mjs +++ b/capsule-docs/astro.config.mjs @@ -27,6 +27,10 @@ export default defineConfig({ baseUrl: 'https://github.com/justin13888/Capsule/tree/master/capsule-docs', }, + components: { + // Renders the `status` frontmatter (draft/stable) as a title badge. + PageTitle: './src/components/PageTitle.astro', + }, sidebar: [ { label: 'Guides', diff --git a/capsule-docs/biome.jsonc b/capsule-docs/biome.jsonc index 261e89c3..f48d695b 100644 --- a/capsule-docs/biome.jsonc +++ b/capsule-docs/biome.jsonc @@ -10,5 +10,20 @@ "enabled": true, "clientKind": "git", "useIgnoreFile": true - } + }, + "overrides": [ + { + // Biome parses only the frontmatter of .astro files, so values used + // exclusively in the template read as unused. Per Biome's Astro guidance. + "includes": ["**/*.astro"], + "linter": { + "rules": { + "correctness": { + "noUnusedImports": "off", + "noUnusedVariables": "off" + } + } + } + } + ] } diff --git a/capsule-docs/src/components/PageTitle.astro b/capsule-docs/src/components/PageTitle.astro new file mode 100644 index 00000000..17d3820f --- /dev/null +++ b/capsule-docs/src/components/PageTitle.astro @@ -0,0 +1,35 @@ +--- +// Starlight PageTitle override: renders the `status` frontmatter field +// (see src/content.config.ts) as a badge beside the page title. +import { Badge } from '@astrojs/starlight/components'; +import Default from '@astrojs/starlight/components/PageTitle.astro'; + +const { status } = Astro.locals.starlightRoute.entry.data; + +const badge = + status === 'stable' + ? { text: 'Stable', variant: 'success' as const } + : status === 'draft' + ? { text: 'Draft', variant: 'caution' as const } + : undefined; +--- + +{ + badge ? ( +
+ + +
+ ) : ( + + ) +} + + From 5273b211c92ea6e99e1d1d934d8e2725ff171b83 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:55:37 -0400 Subject: [PATCH 34/58] docs(design): re-pin the MLS suite and correct the blocker to its verified state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified upstream (2026-07): OpenMLS ships the SHA-256 X-Wing suite (0x004D) today via its libcrux provider — a formally verified Rust backend (the old note wrongly called it C), pre-1.0 and not fully audited — while the RustCrypto effort (openmls#1940) and the IETF pq-ciphersuites draft have converged on SHA-512 variants, making 0x004D a likely dead-end codepoint. Re-pin the planned suite to MLS_256_XWING_CHACHA20POLY1305_SHA512_Ed25519 (codepoint pending) and rewrite the status note as a deliberate hold: non-final draft, no shipping backend for the target suite, unaudited pre-1.0 backend for the superseded one. Architecture unchanged — the AlbumAuthority seam makes any final re-pin an inventory edit, not a redesign. Nothing implements MLS yet, so the 0x0001 bundle edit is pre-deployment-safe. --- SLICES.md | 11 +++++++---- capsule-core/src/crypto/primitives.rs | 2 +- .../src/content/docs/design/cryptography/mls.md | 2 +- .../content/docs/design/cryptography/primitives.md | 6 +++--- capsule-docs/src/content/docs/design/module-map.md | 2 +- 5 files changed, 13 insertions(+), 10 deletions(-) diff --git a/SLICES.md b/SLICES.md index 48a145fa..4e8b014c 100644 --- a/SLICES.md +++ b/SLICES.md @@ -175,7 +175,7 @@ pass until the gate lifts. | `rawshift` (in-house RAW decode; git submodule, alpha, consumed by nothing yet) | stabilizing | Full RAW support in S-B1/S-B2. v1 ships the zune-jpeg format set; the `media::image::formats::raw` stub is the integration point. | | `spargen` (in-house OpenAPI **3.1** client generator) | in development | S-D8 (typed REST client + `AuthenticatedClient` revival). Progenitor is gone — we do not downgrade schemas to 3.0. | | `geocoordinates-rs` (in-house WGS-84 ↔ GCJ-02/BD-09 conversions) | planned | The deterministic client-side coordinate conversion named in [Metadata — Geolocation](capsule-docs/src/content/docs/design/metadata.md); consumed by map display and S-H3 geo features. Until it lands, WGS-84 storage is unaffected (conversion is display-only). | -| `openmls` ciphersuite `0x004D` | blocked upstream | S-X1 → S-X2 → S-X3 (tracked in Lane X). | +| `openmls` X-Wing/SHA-512 ciphersuite (codepoint pending) | blocked upstream | S-X1 → S-X2 → S-X3 (tracked in Lane X). | ## Lane A — core crypto @@ -823,9 +823,12 @@ its design here. - **Contract:** [Cryptography — MLS](capsule-docs/src/content/docs/design/cryptography/mls.md) (status note), [Keys — Write Authority Interface](capsule-docs/src/content/docs/design/cryptography/keys.md). -- **Blocked on:** an OpenMLS RustCrypto backend for ciphersuite `0x004D` - (openmls#1940) or IETF finalization of the PQ ciphersuites draft. **Unblock check:** - openmls release notes; re-evaluate quarterly. +- **Blocked on:** the deliberate hold in the MLS status note — the target suite + `MLS_256_XWING_CHACHA20POLY1305_SHA512_Ed25519` has no IANA codepoint (non-final + `draft-ietf-mls-pq-ciphersuites`) and no shipping OpenMLS backend (openmls#1940 is + open with no PR; today's libcrux provider ships only the superseded SHA-256 + variant). **Unblock check:** openmls release notes + draft status; re-evaluate the + suite pin itself, not just backend availability; quarterly. - **Deliverable:** `OpenMlsAuthority` behind `&dyn AlbumAuthority` — drops in without touching `verify_asset`; the `ReferenceAuthority` epoch ledger stays as the offline and test authority. diff --git a/capsule-core/src/crypto/primitives.rs b/capsule-core/src/crypto/primitives.rs index 19c744cd..b5b96099 100644 --- a/capsule-core/src/crypto/primitives.rs +++ b/capsule-core/src/crypto/primitives.rs @@ -21,7 +21,7 @@ pub const PROTOCOL_VERSION: &str = "2026-05-31"; #[non_exhaustive] pub enum SuiteId { /// `0x0001`: SHA-256 · HKDF-SHA512 · Argon2id · AES-256-GCM(+STREAM) · - /// Ed25519+ML-DSA-65 · X-Wing · MLS `0x004D`. + /// Ed25519+ML-DSA-65 · X-Wing · MLS X-Wing/SHA-512 suite (codepoint pending). V1, } diff --git a/capsule-docs/src/content/docs/design/cryptography/mls.md b/capsule-docs/src/content/docs/design/cryptography/mls.md index 30d7f20c..425a4a97 100644 --- a/capsule-docs/src/content/docs/design/cryptography/mls.md +++ b/capsule-docs/src/content/docs/design/cryptography/mls.md @@ -6,7 +6,7 @@ status: draft Capsule's group layer is the [MLS ciphersuite](/design/cryptography/primitives/#mls-ciphersuite) from the inventory. It will be implemented in `capsule-core::crypto::mls` (planned) as a thin wrapper over OpenMLS — the wrapper is what binds MLS to Capsule's identity layer ([Keys](/design/cryptography/keys/)) and to the in-band AMK distribution. -**Status: blocked upstream.** The chosen ciphersuite (`MLS_256_XWING_CHACHA20POLY1305_SHA256_Ed25519`, `0x004D`) rides a non-final IETF draft with no IANA codepoint, and OpenMLS supports it only through a C (`libcrux`) backend — there is no RustCrypto PQ backend yet (openmls#1940). Until that lands, the epoch-authority role this doc assigns to the MLS commit chain is filled offline by the admin-signed epoch ledger behind the [`AlbumAuthority` interface](/design/cryptography/keys/#write-authority-interface); the membership ceremonies, `Welcome`/history delivery, and the [album upgrade ceremony](/design/versioning/#album-upgrade-ceremony) wait for live MLS. Docs that depend on live MLS link to this note rather than restating it. +**Status: blocked upstream — a deliberate hold, not an absence of code.** Capsule targets `MLS_256_XWING_CHACHA20POLY1305_SHA512_Ed25519`, the X-Wing suite the upstream ecosystem is converging on; it has no IANA codepoint yet. Three facts gate adoption (verified 2026-07): (1) the PQ suites ride `draft-ietf-mls-pq-ciphersuites`, a non-final informational draft with no IANA codepoints — groups created on a pre-final suite must migrate when codepoints land; (2) what OpenMLS ships today is the *earlier* SHA-256 variant (`0x004D`) via its `libcrux` provider — a formally verified **Rust** backend, but pre-1.0 and not fully audited, and that suite is a likely dead-end codepoint; (3) the RustCrypto-backed implementation of the SHA-512 suites is requested upstream ([openmls#1940](https://github.com/openmls/openmls/issues/1940)) with no PR yet. Until the draft finalizes or #1940 lands, the epoch-authority role this doc assigns to the MLS commit chain is filled offline by the admin-signed epoch ledger behind the [`AlbumAuthority` interface](/design/cryptography/keys/#write-authority-interface); the membership ceremonies, `Welcome`/history delivery, and the [album upgrade ceremony](/design/versioning/#album-upgrade-ceremony) wait for live MLS. Because nothing implements MLS yet, a final suite re-pin at adoption time is a one-line inventory edit behind the same seam, not a redesign. Docs that depend on live MLS link to this note rather than restating it. The ciphersuite's choice of [ChaCha20-Poly1305](/design/cryptography/primitives/#mls-control-aead) (rather than the [AES-GCM](/design/cryptography/primitives/#bulk-aead) used for user data) is acceptable because: diff --git a/capsule-docs/src/content/docs/design/cryptography/primitives.md b/capsule-docs/src/content/docs/design/cryptography/primitives.md index 92b13171..b7138a42 100644 --- a/capsule-docs/src/content/docs/design/cryptography/primitives.md +++ b/capsule-docs/src/content/docs/design/cryptography/primitives.md @@ -19,7 +19,7 @@ The primitive identities themselves live in `capsule-core::crypto::primitives` a | [MLS control AEAD](#mls-control-aead) | ChaCha20-Poly1305 | Inherited from the [MLS ciphersuite](#mls-ciphersuite) | | [Signature scheme](#signature-scheme) | Hybrid Ed25519 + ML-DSA-65 | Identity, device, asset manifest, write tier | | [KEM](#kem) | X-Wing (X25519 + ML-KEM-768) | MLS HPKE | -| [MLS ciphersuite](#mls-ciphersuite) | `MLS_256_XWING_CHACHA20POLY1305_SHA256_Ed25519` (0x004D) | Group key management | +| [MLS ciphersuite](#mls-ciphersuite) | `MLS_256_XWING_CHACHA20POLY1305_SHA512_Ed25519` (codepoint pending) | Group key management | | [Randomness](#randomness) | OS CSPRNG (`getrandom`) | All keys, salts, nonces | | [Transport](/design/cryptography/failure-modes/#transport-security) | TLS 1.3 with hybrid X25519+ML-KEM | Client-server, server-server | @@ -37,7 +37,7 @@ A faulty, malicious, or version-mismatched client could damage data by writing u `crypto_suite_id = 0x0001` denotes exactly the [Primitives Inventory](#primitives-inventory) above. Retiring any primitive (a broken SHA-256, a deprecated AEAD) **does not edit the row** — it adds a new row and a new suite id. An old AssetManifest carrying `0x0001` keeps verifying against the original row forever; new writes use the new suite id. This is the single-doc edit the inventory promises, generalized to the bundle. -`crypto_suite_id` values are a **Capsule-owned namespace**, unrelated to the MLS ciphersuite codepoint namespace: `0x0001` (the Capsule bundle) and `0x004D` (the OpenMLS id of the [MLS ciphersuite](#mls-ciphersuite) *inside* that bundle) are identifiers in two different registries, and neither implies the other. +`crypto_suite_id` values are a **Capsule-owned namespace**, unrelated to the MLS ciphersuite codepoint namespace: `0x0001` (the Capsule bundle) and the eventual IANA codepoint of the [MLS ciphersuite](#mls-ciphersuite) *inside* that bundle are identifiers in two different registries, and neither implies the other. The signatures on every manifest cover `crypto_suite_id` and `protocol_version`, so a downgrade-attempt (re-signing an existing manifest under a weaker suite) cannot be silently produced. @@ -97,7 +97,7 @@ MLS LeafNode signatures stay Ed25519-only (pinned by the ciphersuite); the ML-DS ### MLS Ciphersuite -**`MLS_256_XWING_CHACHA20POLY1305_SHA256_Ed25519`** (OpenMLS ciphersuite 0x004D) — MLS (RFC 9420) with the PQ ciphersuites from `draft-ietf-mls-pq-ciphersuites`. See [MLS](/design/cryptography/mls/) for how the ciphersuite's choices (X-Wing KEM, ChaCha20-Poly1305 control AEAD, SHA-256 hash, Ed25519 leaf sigs) interact with the identity layer. +**`MLS_256_XWING_CHACHA20POLY1305_SHA512_Ed25519`** (codepoint pending — the X-Wing suite the upstream ecosystem is converging on; adoption gating owned by the [status note in MLS](/design/cryptography/mls/)) — MLS (RFC 9420) with the PQ ciphersuites from `draft-ietf-mls-pq-ciphersuites`. The suite's SHA-512 is MLS-internal (transcript and tree hashing); Capsule's [content-address hash](#cryptographic-hash) is a separate primitive and unaffected. See [MLS](/design/cryptography/mls/) for how the ciphersuite's choices (X-Wing KEM, ChaCha20-Poly1305 control AEAD, Ed25519 leaf sigs) interact with the identity layer. ### Randomness diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 6193a718..4b08e4cd 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -117,7 +117,7 @@ Navigation from a design doc back to where the code lives. | [Cryptography — Primitives](/design/cryptography/primitives/) | `capsule-core::crypto::primitives` | | [Cryptography — Keys](/design/cryptography/keys/) | `capsule-core::crypto::keys` (`Signer`/`HardwareSigner` seams; software + TPM reference adapters; Secure Enclave / StrongBox adapters in `capsule-core-swift`/`-kotlin`; P-256 hybrid-DSK variant planned) | | [Cryptography — Keys: Write Authority Interface](/design/cryptography/keys/#write-authority-interface) | `capsule-core::crypto::authority` (`AlbumAuthority` trait + `ReferenceAuthority` epoch ledger; `OpenMlsAuthority` blocked with MLS) | -| [Cryptography — MLS](/design/cryptography/mls/) | `capsule-core::crypto::mls` (blocked upstream — no usable OpenMLS backend for the `0x004D` ciphersuite; see the [MLS status note](/design/cryptography/mls/)) | +| [Cryptography — MLS](/design/cryptography/mls/) | `capsule-core::crypto::mls` (blocked upstream; see the [MLS status note](/design/cryptography/mls/)) | | [Cryptography — Encryption](/design/cryptography/encryption/) | `capsule-core::crypto::encryption` | | [Cryptography — Provenance](/design/cryptography/provenance/) | `capsule-core::crypto::provenance` + `verify_asset` chokepoint | | [Cryptography — Failure Modes](/design/cryptography/failure-modes/) | Cross-cutting: `capsule-core::backup`, `capsule-core::library`, `capsule-core::crypto::*` | From 840aadd0442b8e366ca852745d74cd4273f353b9 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:56:24 -0400 Subject: [PATCH 35/58] docs(design): record the verified hardware-key floor behind the P-256 variant Fact-checked against vendor/TCG documentation: Secure Enclave is P-256-only; StrongBox's mandatory set is RSA-2048 + P-256 (Curve25519 is TEE-only, explicitly not expected of StrongBox); TPM 2.0's PC Client profile mandates RSA-2048 + P-256. P-256 is therefore the unique curve in the intersection, which is the technical justification for the P-256 hybrid-DSK variant. TPM 1.2 (no ECC, SHA-1-only signing) is explicitly excluded from hardware binding rather than accommodated via RSA-2048. --- capsule-docs/src/content/docs/design/cryptography/keys.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/capsule-docs/src/content/docs/design/cryptography/keys.md b/capsule-docs/src/content/docs/design/cryptography/keys.md index 129f7397..526048ed 100644 --- a/capsule-docs/src/content/docs/design/cryptography/keys.md +++ b/capsule-docs/src/content/docs/design/cryptography/keys.md @@ -60,6 +60,13 @@ Each device's keys are cross-signed into the [device directory](#device-director Both are signed by the IK (hybrid signature). The **classical half** of each device key is **generated inside and never leaves hardware** — Secure Enclave (iOS), StrongBox/Keystore (Android), TPM (desktop) — and is non-exportable; the **PQ half** (ML-DSA-65 / ML-KEM-768) is software-sealed under hardware-protected material, because no shipping secure element implements the PQ algorithms. Shipping elements also expose **P-256**, not Ed25519/X25519, so the hardware-backed composition is the planned **P-256 hybrid variant**; the software composition (both halves in software) is what integrates end-to-end today. Because hardware halves cannot be backed up, devices are treated as disposable either way: a lost device is removed and a new one re-bootstrapped from the master key. +**Verified hardware floor (2026-07).** P-256 is not a convenience choice — it is the *only* signing curve in the intersection of the secure elements Capsule targets: + +- **Apple Secure Enclave** (present since the iPhone 5s, so every supported iPhone): NIST P-256 EC keys **only** — no Ed25519, no RSA, no P-384. +- **Android StrongBox** (2018+ devices that ship it): the mandatory algorithm set is RSA-2048 + ECDSA/ECDH **P-256**. Curve25519 support (KeyMint v2, Android 13+) applies only to the TEE-backed Keystore — StrongBox implementations are explicitly not expected to provide it. The TEE-backed Keystore, the fallback on devices without StrongBox, also guarantees P-256. +- **TPM 2.0** (PC Client platform profile): RSA-2048 and ECC **P-256** are mandatory-to-implement; P-384 is optional. +- **TPM 1.2 / no TPM** (desktops at the old end of the support window, ≈ pre-2016): no ECC at all and a SHA-1-only signing path — **excluded from hardware binding by design**. These devices run the software keystore, which the disposable-device model already covers. The strict lowest common denominator *including* TPM 1.2 would be RSA-2048, and it is deliberately rejected: it would inherit TPM 1.2's broken SHA-1 signing and add an entire algorithm lineage for a shrinking device tail. + A device key can be revoked without affecting the user's identity or other devices. Revocation is done by signing a revocation statement with the IK and publishing it to a well-known location. The server then refuses to deliver new key wraps to that device, and remaining devices rotate any group keys the revoked device had access to. The revoked device's directory entry is **retained** — marked with `revoked_at` (RFC 3339), never deleted — so the manifests it signed *before* revocation stay verifiable forever (provenance is append-only; see [Provenance](/design/cryptography/provenance/#what-an-attacker-with-all-current-keys-still-cannot-do)). ### Owner Group Keys (OGKs) From 3ef2ff06f37efec4fecd5b0687bf8a26b06a9bec Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:57:03 -0400 Subject: [PATCH 36/58] docs(design): link filesystem-index claims to their owner anchor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 'key-free facts' statements in the filesystem index restated what Server Filesystem — 'PostgreSQL: What the Server Knows' owns normatively. Link the claim to its owner anchor instead so the exact field list has a single home and the summary cannot drift. --- capsule-docs/src/content/docs/design/filesystem/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/capsule-docs/src/content/docs/design/filesystem/index.md b/capsule-docs/src/content/docs/design/filesystem/index.md index 09608101..60c0a5bf 100644 --- a/capsule-docs/src/content/docs/design/filesystem/index.md +++ b/capsule-docs/src/content/docs/design/filesystem/index.md @@ -22,7 +22,7 @@ This index covers the principles both sides share. The import pipeline, the uplo These follow directly from [Core Principles](/design/principles/): -- **Recovery-first.** No database is required to interpret canonical data. On the client, sidecar files are the source of truth and the index is a rebuildable cache. On the server, PostgreSQL is the authoritative index, but it holds only key-free facts. +- **Recovery-first.** No database is required to interpret canonical data. On the client, sidecar files are the source of truth and the index is a rebuildable cache. On the server, PostgreSQL is the authoritative index, holding only the [key-free facts enumerated in Server Filesystem](/design/filesystem/server/#postgresql-what-the-server-knows). - **Atomic writes.** Every write that must not tear uses temp-file + atomic rename on the same filesystem. Direct overwrites risk corruption on power loss. The full per-granularity rules live in [Maintenance — Atomic Writes](/design/filesystem/maintenance/#atomic-writes-and-crash-recovery). - **Ephemeral derived data.** Only originals and their canonical metadata are irreplaceable. Thumbnails, transcodes, parsed-metadata caches, and the query index can all be regenerated and are treated as such. - **4 KiB alignment.** Data is processed and written block-aligned to 4 KiB, which matches memory and disks and keeps the [append-only upload path](/design/import/upload-protocol/#append-only-storage) page-aligned. @@ -35,6 +35,6 @@ These follow directly from [Core Principles](/design/principles/): | Holds keys | No | Yes | | Stored form | Opaque ciphertext blobs | Plaintext media + CBOR sidecars | | Naming | Content-addressed by ciphertext hash | UUIDv7 stems, date-bucketed | -| Index | PostgreSQL (key-free facts only) | SQLite (rebuildable, full plaintext metadata) | +| Index | PostgreSQL ([key-free facts only](/design/filesystem/server/#postgresql-what-the-server-knows)) | SQLite (rebuildable, full plaintext metadata) | | Derived data | Stored as client-generated encrypted blobs | Generated locally, cached, rebuildable | | Originals | Always retained while referenced | Present only if synced locally | From 6daad74644f29c32ac437d3efa64d67965e3f1ca Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 12:59:09 -0400 Subject: [PATCH 37/58] docs(design): make Valkey required and drop the Postgres-only profile The implementation has exactly one session-store code path (Valkey via bb8_redis, for upload and auth sessions alike); the optional-Valkey / Postgres-only deployment-profile story existed only in the docs. Replace Deployment Profiles with Required Services (blob store + PostgreSQL + Valkey, all required), delete the profile table, the store-agnostic claim, and the profile-parity validation bullet, and update every inbound reference (upload protocol, versioning, principles, filesystem index). We do not offer multiple code paths for this. --- .../content/docs/design/filesystem/index.md | 2 +- .../content/docs/design/filesystem/server.md | 24 +++++++------------ .../docs/design/import/upload-protocol.md | 6 ++--- .../src/content/docs/design/principles.md | 2 +- .../src/content/docs/design/versioning.md | 2 +- 5 files changed, 14 insertions(+), 22 deletions(-) diff --git a/capsule-docs/src/content/docs/design/filesystem/index.md b/capsule-docs/src/content/docs/design/filesystem/index.md index 60c0a5bf..f32061fb 100644 --- a/capsule-docs/src/content/docs/design/filesystem/index.md +++ b/capsule-docs/src/content/docs/design/filesystem/index.md @@ -12,7 +12,7 @@ The on-disk layout is itself part of the contract — the filenames, directory s | Sub-doc | Concern | Primary crate(s) | | ----------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------- | -| [Server Filesystem](/design/filesystem/server/) | Blob store layout, Postgres index, deployment profiles, ownership, deletion | `capsule-api` + storage glue | +| [Server Filesystem](/design/filesystem/server/) | Blob store layout, Postgres index, required services, ownership, deletion | `capsule-api` + storage glue | | [Client Filesystem](/design/filesystem/client/) | Desktop / mobile library layout, local SQLite index, space recovery | `capsule-core::{library,db}` + per-platform glue | | [Maintenance](/design/filesystem/maintenance/) | Self-validation, scrubbing, repair, intra-library dedup, atomic-write granularity | `capsule-core::library` (client) + `capsule-api` (server-side scrub) | diff --git a/capsule-docs/src/content/docs/design/filesystem/server.md b/capsule-docs/src/content/docs/design/filesystem/server.md index bb3a1185..878bd182 100644 --- a/capsule-docs/src/content/docs/design/filesystem/server.md +++ b/capsule-docs/src/content/docs/design/filesystem/server.md @@ -1,29 +1,22 @@ --- title: Server Filesystem -description: The server's blob store layout, Postgres index, and deployment profiles +description: The server's blob store layout, Postgres index, and required services status: draft --- The server's job is to hold ciphertext blobs and a key-free index that maps assets to blobs. It performs no decoding, no metadata extraction, and no thumbnail generation — it cannot, since it never holds a decryption key. The blob layout below **is** the contract: a server-side rebuild (re-deriving the Postgres index from blob bytes) depends on the file naming and the manifest envelope being exactly as specified here. -Implemented in `capsule-api` (blob storage, Postgres index, manifest envelope validation). The session-state store is a [deployment choice](#deployment-profiles), not a versioned API surface. +Implemented in `capsule-api` (blob storage, Postgres index, manifest envelope validation). Volatile session state lives in Valkey (see [Required Services](#required-services)); it is not a versioned API surface. -## Deployment Profiles +## Required Services -The server's durable state is always split across **two required systems** plus an **optional third** for high-concurrency deployments: +The server's state is split across **three required systems** — one code path, no optional profiles: -- **Blob store** (filesystem) — the encrypted bytes of every asset. *Required.* -- **PostgreSQL** — the authoritative index: ownership, album references, blob references, lifecycle state, and (in the default profile) upload-session state. *Required.* -- **Valkey** — volatile upload-session state (offsets, status); the store's 24-hour TTL is the lifetime **cap** (the ≥1-hour survival floor and pressure-discard semantics are owned by [Upload Protocol — Session Lifetime and Discard](/design/import/upload-protocol/#session-lifetime-and-discard)). *Optional.* Recommended only for deployments where upload-session hot-path contention on PostgreSQL becomes measurable. +- **Blob store** (filesystem) — the encrypted bytes of every asset. +- **PostgreSQL** — the authoritative durable index: ownership, album references, blob references, lifecycle state, and the pending-asset rows uploads create. +- **Valkey** — volatile session state: upload-session records (offsets, status) keyed `upload:session:{id}`, with the store's native 24-hour TTL as the lifetime **cap** (the ≥1-hour survival floor and pressure-discard semantics are owned by [Upload Protocol — Session Lifetime and Discard](/design/import/upload-protocol/#session-lifetime-and-discard)); [auth session records](/design/authentication/) ride the same store. -This gives two concrete deployment profiles: - -| Profile | Session state lives in | When to choose it | -| --------------------------- | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- | -| **Default (Postgres-only)** | `upload_sessions` table with `expires_at` TTL column and a periodic sweep | Self-hosted, small-to-medium servers, single-node deployments. Reduces ops surface. | -| **High-concurrency** | Valkey (keyed `upload:session:{id}`) with native 24-hour TTL; PostgreSQL still holds the durable pending-asset row | Large multi-tenant deployments where session-table contention is a measured bottleneck | - -Switching profiles is operationally invisible to clients — the [upload protocol](/design/import/upload-protocol/) does not change, only where the server stores volatile session counters. The protocol is written to be store-agnostic. +The durable/volatile split is design, not a tuning knob: the hot upload path — offset increments and status transitions — never touches the durable Postgres asset row, which is written exactly twice per upload (pending row at session creation, `uploaded` flip at finalization). A Postgres-resident session table would be a second implementation of the same contract; Capsule ships exactly one. ## Blob Store Layout @@ -115,7 +108,6 @@ Clients need to confirm an asset is *safely stored* before they discard their on - **Layout round-trip (unit).** Upload, finalize, rename, and assert the blob lives at exactly `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` on disk. Recompute the hash from disk; assert match. - **Index rebuild idempotency (smoke).** Take a real testcontainer Postgres + a populated `blobs/` tree, drop the index tables, run the rebuild routine, assert every row matches a hand-derived expected set. Re-run; assert zero changes. - **Quarantine on malformed envelope (unit).** Inject a blob with a corrupted manifest envelope into `blobs/`; run rebuild; assert the blob moves to `quarantine/` with a `.reason.json` that names the structural check that failed. -- **Deployment-profile parity (smoke).** Run the upload-server smoke suite against the Postgres-only profile and the Postgres+Valkey profile; assert byte-identical client-observable behavior. - **Reference-count GC safety (unit).** Decrement a blob's last reference; assert eligibility for GC; assert GC only proceeds after a configurable grace period; concurrent re-reference during the grace period cancels GC. - **Dangling-reference safety (unit).** Point a committed row at a blob hash absent from `blobs/`; run the integrity check; assert the row is surfaced/quarantined and **never** auto-deleted, and that the missing blob is not treated as collectable. - **Storage-verification verdict (unit).** Compose the no-key verdict for a finalized asset (stored + indexed + retrievable → `durable`); then mark a referenced blob `collectable_since` and assert it reports non-retrievable, and remove a blob from `blobs/` and assert non-stored. Owner: [Import — Storage Verification](/design/import/storage-verification/). diff --git a/capsule-docs/src/content/docs/design/import/upload-protocol.md b/capsule-docs/src/content/docs/design/import/upload-protocol.md index 8a1ecb22..2e34cd73 100644 --- a/capsule-docs/src/content/docs/design/import/upload-protocol.md +++ b/capsule-docs/src/content/docs/design/import/upload-protocol.md @@ -50,7 +50,7 @@ All endpoints are authenticated with a bearer JWT. | `DELETE` | `/upload/{id}` | Cancel the session — removes the upload file, the session record, and the pending asset row. Rejected with `409` while finalization is running. | | `GET` | `/upload/sessions` | List the **uploader's** active sessions (keyed by `upload_user_id` — the resuming party — not `owner_id`), so a client can resume across app restarts. | -Creating a session writes a *pending* asset row to Postgres (`uploaded = false`) and a session record to the configured **session-state store** (see [Filesystem — Server: Deployment Profiles](/design/filesystem/server/#deployment-profiles): Postgres by default, Valkey in the high-concurrency profile). The pending row reserves the asset ID that derivative and metadata blobs reference. The session record carries everything finalization needs — declared size and hash, `content_type`, `crypto_suite_id`, `protocol_version`, `blob_role`, the manifest envelope, `album_id`/`owner_id`/`intent_id`, and the uploader — so a session is finalizable from its own record with no further client input. +Creating a session writes a *pending* asset row to Postgres (`uploaded = false`) and a session record to **Valkey** (see [Filesystem — Server: Required Services](/design/filesystem/server/#required-services)). The pending row reserves the asset ID that derivative and metadata blobs reference. The session record carries everything finalization needs — declared size and hash, `content_type`, `crypto_suite_id`, `protocol_version`, `blob_role`, the manifest envelope, `album_id`/`owner_id`/`intent_id`, and the uploader — so a session is finalizable from its own record with no further client input. Per-endpoint header census (the `X-Capsule-*` namespace itself is owned by the [universal header census](/design/threat-model/validation/#universal-headers)): @@ -128,7 +128,7 @@ Pending ─▶ Uploading ─▶ WaitingForProcessing ─▶ Completed - **Completed** — hash verified, asset marked uploaded. Terminal. - **FailedProcessing** — terminal failure. The upload file and the pending asset row are removed at the transition; the session *status record* is retained as a receipt. Terminal. -Session records live in the [session-state store](/design/filesystem/server/#deployment-profiles) with an uploader-keyed index for listing. This split is intentional: the session store holds only volatile transfer state, so the hot path — offset increments and status transitions — never touches the durable Postgres asset row. (In the default Postgres-only profile, sessions live in an `upload_sessions` table with an `expires_at` column and a periodic sweep; in the high-concurrency profile, they live in Valkey under keys `upload:session:{id}` with atomic `HINCRBY`/`HSET` and native TTL.) Postgres's durable asset record is written exactly twice per upload regardless of profile: once at session creation (the pending row) and once at finalization (mark uploaded). +Session records live in [Valkey](/design/filesystem/server/#required-services) — keys `upload:session:{id}` with atomic `HINCRBY`/`HSET` and native TTL — with an uploader-keyed index for listing. This split is intentional: the session store holds only volatile transfer state, so the hot path — offset increments and status transitions — never touches the durable Postgres asset row. Postgres's durable asset record is written exactly twice per upload: once at session creation (the pending row) and once at finalization (mark uploaded). ## Session Lifetime and Discard @@ -267,7 +267,7 @@ These checks deduplicate at upload time. Byte-identical assets that still slip i The wire protocol is the boundary across two modules, so both sides have rich isolated test surfaces: -- **Server protocol conformance (smoke).** Exercise the full state machine against the real server against a testcontainer Postgres (and Valkey for the high-concurrency profile): create session → PATCH chunks → finalize → verify Completed. Mock the client at the HTTP layer using a generated request fixture set. +- **Server protocol conformance (smoke).** Exercise the full state machine against the real server against testcontainer Postgres + Valkey: create session → PATCH chunks → finalize → verify Completed. Mock the client at the HTTP layer using a generated request fixture set. - **Server strictness rejection (unit).** Every row of the [strictness table](#chunk-rules-and-strictness) and every row of the [error taxonomy](#error-taxonomy) has a unit test asserting the exact HTTP status **and** `error.*` code. - **Server idempotency (unit).** Replay each idempotent endpoint with identical input; assert byte-identical response. Duplicate create returns the active session; finalized-hash create returns `duplicate_blob`. - **Server lifetime (unit).** A session with an accepted chunk in the last hour is never evicted under injected pressure; a stalled session past the floor is; terminal receipts survive pressure and die at the cap. diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 77608633..33278242 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -41,7 +41,7 @@ The owner docs are: | Device enrollment + cross-device add ceremony | [Device Enrollment](/design/device-enrollment/) | | ML model identities + embedding provenance | [AI/ML Integrations](/design/ai/) | | LQIP scheme + thumbnail/preview formats | [Thumbnails and Previews](/design/thumbnails/) | -| Server filesystem (blob store, Postgres index, deployment profiles) | [Filesystem — Server](/design/filesystem/server/) | +| Server filesystem (blob store, Postgres index, required services) | [Filesystem — Server](/design/filesystem/server/) | | Client filesystem (library layout, local index, space recovery) | [Filesystem — Client](/design/filesystem/client/) | | Library self-maintenance + atomic-write granularity | [Filesystem — Maintenance](/design/filesystem/maintenance/) | | Session/access tokens + identity binding + auth flow | [Authentication](/design/authentication/) | diff --git a/capsule-docs/src/content/docs/design/versioning.md b/capsule-docs/src/content/docs/design/versioning.md index 9d3c3bb9..eb77d5dd 100644 --- a/capsule-docs/src/content/docs/design/versioning.md +++ b/capsule-docs/src/content/docs/design/versioning.md @@ -16,7 +16,7 @@ Versioning happens on multiple layers, each owned by the doc that defines it: - **Cryptographic primitive bundle** — `crypto_suite_id` on every manifest and metadata blob (see [Cryptography — Versioning Identifiers](/design/cryptography/primitives/#versioning-identifiers)). - **Wire protocol** — `protocol_version` (date-based, `YYYY-MM-DD`) on every API request and album pin. See [Threat Model — Protocol Negotiation](/design/threat-model/validation/#protocol-and-capability-negotiation) for the universal handshake. - **Client cache** — internal and rebuildable; cache schema changes drop and rebuild rather than migrate. -- **Server data structures** — PostgreSQL schema migrations forward-only. The session-state store is a deployment choice, not a versioned API surface (see [Filesystem — Server: Deployment Profiles](/design/filesystem/server/#deployment-profiles)). +- **Server data structures** — PostgreSQL schema migrations forward-only. Volatile session state in Valkey is not a versioned API surface (see [Filesystem — Server: Required Services](/design/filesystem/server/#required-services)). ## Negotiation Headers From c192d9fbc721ebddd5bbdfd55773db136ba95ccf Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 13:00:45 -0400 Subject: [PATCH 38/58] docs(design): make client_version identify the exact producing build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit client_version and generated_by_client were free-form strings (the one real producer writes capsule-core/{crate-version}), which cannot scope a defective-build incident to the assets it touched. Define the normative producer grammar {client_id}/{semver}+{commit}[.dirty] in Provenance: client_id names the client product (and thereby the platform), commit is the client's own source hash. Nothing non-identifying rides along — OS version and device model don't identify the code, and created_by_device already identifies the device. Audit-only as before: the server bounds the size but never validates the grammar. Surface the field in Server Filesystem's envelope and Postgres lists so a defective build's writes are enumerable, and add slice S-D15 for the build-time commit embedding + per-client client_id injection. --- SLICES.md | 16 ++++++++++++++ .../docs/design/cryptography/provenance.md | 22 +++++++++++++++++-- .../content/docs/design/filesystem/server.md | 2 ++ 3 files changed, 38 insertions(+), 2 deletions(-) diff --git a/SLICES.md b/SLICES.md index 4e8b014c..f929c5da 100644 --- a/SLICES.md +++ b/SLICES.md @@ -97,6 +97,7 @@ its slice. | S-D12 | Recovery verification cadence + guided re-wrap | sdk/clients | S-C12 | M | ready | | S-D13 | Culling workflow client UX | sdk/clients | — | M | ready | | S-D14 | Local-gallery security gates | sdk/clients | — | S | ready | +| S-D15 | Exact client build identification | sdk/clients | — | S | ready | | S-E1 | Share-link end-to-end serving | fed/sharing | S-C4 | M | ready | | S-E2 | Federation capabilities + pulls | fed/sharing | S-C2, S-A3 | L | ready | | S-E3 | LAN peering | fed/sharing | S-D2, S-C7 | L | ready | @@ -650,6 +651,21 @@ SDK; they never hand-roll network flows. no-network-on-read-paths smoke runs with a socket-refusing harness. - **Tier:** Unit + Smoke; the airplane-mode E2E case rides the Module Map surface. +### S-D15 — Exact client build identification + +- **Contract:** [Provenance — Client Build Identification](capsule-docs/src/content/docs/design/cryptography/provenance.md). +- **Deliverable:** build-time git-commit embedding (`build.rs` `git rev-parse` + dirty + detection — no vergen-class dependency needed) feeding the manifest producer in + `capsule-core::lifecycle`, which today writes `capsule-core/{CARGO_PKG_VERSION}`; + a `client_id` injection point through the SDK/FFI surface so each app reports + itself (`capsule-ios`, `capsule-cli`, …) rather than `capsule-core`; the same value + on `generated_by_client`. +- **Done when:** the provenance doc's client-build-identification Validation bullet + passes; a locally built CLI writes `capsule-cli/{semver}+{commit}` (`.dirty` on a + modified tree). Test fixtures may keep arbitrary strings — the grammar is producer + discipline, not a verify gate. +- **Tier:** Unit. + ## Lane E — federation / sharing ### S-E1 — Share-link end-to-end serving diff --git a/capsule-docs/src/content/docs/design/cryptography/provenance.md b/capsule-docs/src/content/docs/design/cryptography/provenance.md index 4125c1d4..b4f4f9c8 100644 --- a/capsule-docs/src/content/docs/design/cryptography/provenance.md +++ b/capsule-docs/src/content/docs/design/cryptography/provenance.md @@ -37,7 +37,7 @@ AssetManifest { // AMK (see Encryption — Asset Key Derivation). Opaque to the server. created_by_user: UUID, created_by_device: UUID, - client_version: String, + client_version: String, // exact producing build; grammar owned by Client Build Identification below timestamp: RFC3339, // self-asserted capture/write time; audit-only (see Keys — Write Authorization) action: enum, // create | replace | delete | metadata-update // | derivative-add | derivative-replace | trash-restore @@ -75,6 +75,23 @@ The closed action enum is owned by [Authorization — The Closed Action Set](/de - The two options in v1's original schema — `prior_provenance_hash`, `retention_until` — encode as **present with `null`** when logically absent (the encoding existing signatures were produced over; kept for signature stability). - Fields added within v1 after that — `metadata_blob_hash`, `key_mode`, `wrapped_file_key`, and the `DerivativeManifest`'s `amk_version`/`protocol_version` — encode as **absent keys** at their default/absent state, so manifests signed before the fields existed re-verify byte-identically. In particular an absent `key_mode` **means `derived`**; an implementation that writes `key_mode: "derived"` explicitly, or `metadata_blob_hash: null`, emits different signed bytes and breaks verification. +## Client Build Identification + +`client_version` (and the `DerivativeManifest`'s `generated_by_client`) pins a write to the **exact client build** that produced it, so a defect in one shipped build — of any client, in-repo or not — is traceable across every asset it touched. The grammar is normative for every real producer: + +```text +client_version = client_id "/" semver "+" commit [".dirty"] + +client_id the client product, which also names the platform: + capsule-ios | capsule-android | capsule-desktop | capsule-cli + | capsule-web | an out-of-repo client's own stable id +commit git commit hash of the client's own source tree + (full, or a prefix of ≥ 12 hex chars) +".dirty" appended when built from a modified tree +``` + +Example: `capsule-ios/1.4.2+9f3a1c7d2b4e`. The field deliberately carries **nothing else** — OS version, device model, and locale do not identify the code that encrypted the bytes, and `created_by_device` already identifies the device. Like `timestamp`, the value is audit-only and never load-bearing for authorization; the server enforces only the [bounded-size rule](/design/threat-model/schema-rules/), not the grammar, so a nonconforming value is a producer bug surfaced in audit, not a rejected write. The grammar constrains the *content* of an existing required string field — no wire or signature impact. + ## Provenance of Library Modifications Every modification of data or metadata produces a **provenance record** — timestamp, device, client version, action — anchored by the [signed manifest](#asset-manifest) above. The records form an **append-only, hash-chained log per asset**, which is the only structure that lets a key-holding attacker be detected after the fact. @@ -132,7 +149,7 @@ DerivativeManifest { format: String, // e.g. "image/avif", "embedding/mobileclip-b" ciphertext_hash: bytes, generated_by_device: UUID, - generated_by_client: String, + generated_by_client: String, // exact producing build; see Client Build Identification model_id: Option, // for embeddings; see AI/ML Integrations model_version: Option, // for embeddings generated_at: RFC3339, @@ -154,6 +171,7 @@ This is the cryptography sub-doc most directly responsible for the `verify_asset - **Chain advance enforcement** — unit test that appending a record whose `prior_provenance_hash` does not match the current head is rejected. Both client-side (`verify_asset`) and server-side (no-key envelope check) reject the same way. - **Append-only enforcement (cryptographic, not just storage).** The guarantee is the signature chain, not the file mode. A unit test drops or rewrites a record in a serialized chain and asserts the forward walk from `create` detects the break (a non-matching prior hash, or a signature that no longer verifies). A companion test confirms the server rejects any overwrite or delete of an existing provenance entry at its structural validation layer (invariant 17), and that a client whose local `.provenance.cbor` has been tampered re-derives the authoritative chain from the server rather than trusting the local bytes. - **Derivative poisoning rejection** — unit test that a `derivative-replace` whose `prior_provenance_hash` does not chain to the current head for `(asset_id, role)` is rejected; the existing derivative is preserved. +- **Client build identification (unit).** The in-repo producers emit `client_id/semver+commit` with a real embedded commit hash (`.dirty` when the tree is modified); a grammar round-trip parses the emitted value. Manifests carrying arbitrary strings still verify — the grammar is producer discipline, not a `verify_asset` gate. - **What-an-attacker-with-all-current-keys-still-cannot-do** — scenario test that holds every *current* key, attempts to rewrite a past record, and confirms the chain walker detects the break. The cross-module case (a manifest moving through upload → server envelope validation → finalization → client `verify_asset` on download) is bounded E2E surface, listed in [Module Map](/design/module-map/#e2e-test-surface). diff --git a/capsule-docs/src/content/docs/design/filesystem/server.md b/capsule-docs/src/content/docs/design/filesystem/server.md index 878bd182..cb87b4d0 100644 --- a/capsule-docs/src/content/docs/design/filesystem/server.md +++ b/capsule-docs/src/content/docs/design/filesystem/server.md @@ -50,6 +50,7 @@ The server-visible envelope includes: - `crypto_suite_id`, `protocol_version`, `amk_version` — what bundle of primitives encrypted this asset and which album epoch - the ciphertext hash and declared size — content address and storage attribution - `created_by_user`, `created_by_device`, `album_id`, `file_id`, `prior_provenance_hash`, `action` — owner, provenance chain link, and lifecycle action +- `client_version` — the [exact client build](/design/cryptography/provenance/#client-build-identification) that produced the write, down to the commit hash — what scopes a defective-build incident to the assets it touched - the device's hybrid signature — provenance attribution; verifiable against the public device directory even without any key the server holds A rebuild walks the **envelope objects** under `blobs/`, verifies each device signature against the cached device directory, and writes index rows for the blobs each envelope names (original and derivatives by `ciphertext_hash` + role, the metadata blob by `metadata_blob_hash`). A ciphertext blob referenced by no envelope surfaces as an orphan for [GC](#deletion-and-garbage-collection); an envelope naming a missing blob surfaces as a dangling reference — **except** a missing *original* on an asset whose feed state is [`awaiting-original`](/design/import/download-sync/#upload-tiering-staged-uploads), which is expected staged-upload state, not corruption. The rebuild is idempotent: re-running it against an existing index produces no changes. The full envelope check list a server runs at recovery is the same list it runs at write time — see [Threat Model — Server-Side Validation Invariants](/design/threat-model/validation/#server-side-validation-invariants). @@ -78,6 +79,7 @@ The server index records only what can be known without a key: - declared ciphertext size and `content_type` - the `uploaded` flag and server-visible lifecycle state - the server's own trusted `received_at` per write — the authoritative clock for time-based policy (retention, rate limits) — alongside the client's self-asserted, audit-only `timestamp` +- `client_version` — the [exact producing client build](/design/cryptography/provenance/#client-build-identification), audit-only, kept queryable so one defective build's writes can be enumerated - provenance records (see [Cryptography — Provenance](/design/cryptography/provenance/#provenance-of-library-modifications)) No plaintext capture date, dimensions, EXIF, tags, or filename ever reaches the server. Those live inside the encrypted metadata blob (see [Metadata Encryption](/design/cryptography/encryption/#metadata-encryption)) and are readable only by authorized clients. From 2c617507f3c5df39d1e635e496d778c912732829 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 13:02:31 -0400 Subject: [PATCH 39/58] docs(design): specify the server-side Postgres-to-blob-store integrity scrub MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Postgres index is authoritative, but nothing proactively verified that a frozen index + blob-store pair is mutually consistent — GC and rebuild are reactive paths, and an implementation bug on either side would surface only when it bit. Maintenance now owns a read-only, operator-invoked integrity scrub with an explicit checklist: row-to-blob presence (awaiting-original carve-out), blob-to-row orphans, deep re-hash, envelope-chain-to-index agreement, mirrored-fact agreement, and debris/quarantine inventory. Findings are classified, never adjudicated (fault is not assumed to one side) and the scrub never repairs, so it cannot become the deletion bug it exists to catch. Slice S-C14 tracks the implementation. --- SLICES.md | 14 +++++++++++ .../docs/design/filesystem/maintenance.md | 24 +++++++++++++++++-- .../content/docs/design/filesystem/server.md | 2 +- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/SLICES.md b/SLICES.md index f929c5da..983fd962 100644 --- a/SLICES.md +++ b/SLICES.md @@ -83,6 +83,7 @@ its slice. | S-C11 | Refcount GC + retention purge worker | server | S-C1 | M | ready | | S-C12 | Backup escrow server surface | server | — | S | ready | | S-C13 | Session device-cohort storage + grouping | server | — | S | ready | +| S-C14 | Server integrity scrub (Postgres⇄blob-store) | server | S-C1 | M | ready | | S-D1 | SDK upload client (hand-written, stateful protocol) | sdk/clients | S-C1 | M | ready | | S-D2 | SDK sync/download client + connection-class budget | sdk/clients | S-C2, S-C9 | L | ready | | S-D3 | Web guest drop client (WASM) | sdk/clients | S-A6, S-C5 | L | ready | @@ -477,6 +478,19 @@ pass until the gate lifts. behavior under absent/garbage values; grouping; durable map outlives sessions). - **Tier:** Unit + Smoke. **Blocks:** S-D11. +### S-C14 — Server integrity scrub + +- **Contract:** [Maintenance — Server-Side Integrity Scrub](capsule-docs/src/content/docs/design/filesystem/maintenance.md). +- **Deliverable:** the operator-invoked, read-only scrub command in `capsule-api` — + row→blob presence (with the `awaiting-original` carve-out), blob→row orphan + detection, deep re-hash, envelope⇄index chain agreement, mirrored-fact agreement, + debris/quarantine inventory — classified structured findings, per-class counts, + non-zero exit on any finding, and **no mutation of any kind**. +- **Depends on:** S-C1 (the envelope persistence and finalization semantics it + audits). **Done when:** the maintenance doc's seeded-corruption matrix passes + against testcontainer Postgres + a real blob tree; clean-store idempotency holds. +- **Tier:** Unit + Smoke. + ## Lane D — SDK / clients `capsule-sdk` is the **sanctioned network path**: it owns the session/token store and diff --git a/capsule-docs/src/content/docs/design/filesystem/maintenance.md b/capsule-docs/src/content/docs/design/filesystem/maintenance.md index bd241793..c6005eff 100644 --- a/capsule-docs/src/content/docs/design/filesystem/maintenance.md +++ b/capsule-docs/src/content/docs/design/filesystem/maintenance.md @@ -1,12 +1,12 @@ --- title: Library Maintenance and Atomic Writes -description: How Capsule keeps client storage consistent, repairs what it can, and writes atomically +description: How Capsule keeps client and server storage consistent, repairs what it can, and writes atomically status: draft --- The data-integrity principle treats client storage as *potentially lost* (see [Core Principles](/design/principles/)): unlike the server, a client library sits on consumer hardware, syncs only partially, and is edited by a long-lived process that can be killed mid-write. A client therefore never assumes its library is consistent — it periodically *proves* it is, repairs what it can repair safely, and surfaces what it cannot. -The maintenance routines live in `capsule-core::library`: [`scrub`](#scrubbing), [self-validation](#self-validation), [repair](#repair), and [`dedup`](#deduplication). The server runs an equivalent scrub of stale upload files under `incoming/`. All routines are **conservative** — consistent with "we can NEVER delete data unexpectedly," irreplaceable data is never removed without explicit user confirmation. +The maintenance routines live in `capsule-core::library`: [`scrub`](#scrubbing), [self-validation](#self-validation), [repair](#repair), and [`dedup`](#deduplication). The server runs an equivalent scrub of stale upload files under `incoming/` and the [integrity scrub](#server-side-integrity-scrub) over its index and blob store. All routines are **conservative** — consistent with "we can NEVER delete data unexpectedly," irreplaceable data is never removed without explicit user confirmation. This doc also owns the granularity rules for [atomic writes](#atomic-writes-and-crash-recovery), which other docs reference but should not restate. @@ -34,6 +34,23 @@ Recomputes the [content hash](/design/cryptography/primitives/) of each locally Because hashing every original is heavy I/O, content validation is **not** run at startup: it is scheduled opportunistically (device idle, on power, unmetered) and throttled, can be triggered on demand, and re-verifies each original on a slow rolling cadence rather than all at once. +## Server-Side Integrity Scrub + +The client proves its library consistent with the tiers above; the server has the same obligation over its two stores. PostgreSQL is the [authoritative index](/design/filesystem/server/#postgresql-what-the-server-knows) and the blob store holds the bytes plus the [manifest envelopes](/design/cryptography/provenance/#asset-manifest) — but *authoritative* is a statement about which copy wins, not a guarantee that an implementation bug cannot let the two drift. The **integrity scrub** is the external code path that verifies a frozen (or live-quiesced) Postgres + blob-store pair against corruption — deliberately separate from the write path, so a hot-path bug cannot also be a bug in the check that would catch it. + +Implemented in `capsule-api` as an operator-invoked command, schedulable as a job. It is **read-only by design**: it classifies and reports, and never repairs. Repair stays with the paths that own it — the [reference-count GC](/design/filesystem/server/#deletion-and-garbage-collection) for orphans, the [index rebuild](/design/filesystem/server/#recovering-the-index-from-blobs-alone) for a lost index, operator action for quarantines — so the scrub can never itself become the deletion bug it exists to catch. + +What it validates, exactly: + +1. **Row → blob presence.** Every committed blob-referencing row (original, derivative, metadata blob, envelope object) resolves to a file at its content-addressed path under `blobs/`. A miss is a **dangling reference** — the loud integrity error [Server Filesystem owns](/design/filesystem/server/#deletion-and-garbage-collection), never auto-resolved — except a missing *original* on an [`awaiting-original`](/design/import/download-sync/#upload-tiering-staged-uploads) asset, which is expected staged-upload state. +2. **Blob → row presence.** Every file under `blobs/` is referenced by at least one committed row. A miss is an **orphan**, reported for the GC path — never removed by the scrub. +3. **Byte integrity (deep mode).** Blob bytes re-hash to their content-addressed name — the server-side bit-rot check, sharing semantics with [storage verification's `deep` flag](/design/import/storage-verification/). Heavy I/O, so rolling and throttled, like client content validation. +4. **Envelope chain ⇄ index agreement.** Per asset, the append-only envelope sequence in the blob store and the Postgres provenance rows carry the same records with the same chain head, and the chain walks forward from `create` with every `prior_provenance_hash` matching its predecessor. +5. **Mirrored-fact agreement.** The server-visible facts Postgres mirrors out of envelopes — declared sizes, `amk_version`, `action`, `retention_until`, `client_version` — match the envelope copies. +6. **Debris and quarantine inventory.** `incoming/*.bin` files with no live session, and everything under `quarantine/`, are enumerated in the report, so debris and unresolved forensics cannot silently accumulate. + +A discrepancy is **classified, never adjudicated**: the report names the failed check and both sides' evidence, and deliberately does not assume whether the index or the blob store is at fault — misassigned fault is how a "repair" deletes the last good copy. Every finding is logged structured per the [traceability principle](/design/principles/); the run emits per-class counts (zero on a clean store), and a non-zero finding count is the exit signal operators alert on. + ## Repair Repair follows directly from the data-integrity principle — *ephemeral data is rebuilt silently; irreplaceable data is never destroyed to resolve an inconsistency.* @@ -91,5 +108,8 @@ A backup is an export artifact — encrypted, self-describing, and kept outside - **Repair safety (unit).** Each row of the repair table is a unit test: trigger the finding, run repair, assert the *exact* action (delete vs quarantine vs re-fetch) was taken. - **Intra-library dedup correctness (unit).** Two assets with identical plaintext hash; assert dedup proposes the right survivor (union albums, max rating, earliest timestamps), records a soft-delete provenance for the loser, and is reversible. - **Atomic-write crash simulation (smoke).** Programmatically interrupt a bundle write between each pair of staged steps; assert no on-disk state reflects a partial bundle on next startup. +- **Server scrub seeded-corruption matrix (smoke).** Against a real testcontainer Postgres + populated `blobs/` tree, seed each corruption class — delete a referenced blob, flip one byte of a blob, orphan a blob, truncate an asset's envelope sequence, alter a mirrored fact — and assert the scrub reports exactly that class, exits non-zero, and mutates nothing (store and index byte-identical after the run). +- **Server scrub clean-store idempotency (unit).** A consistent store yields zero findings; re-running yields an identical report. +- **Server scrub staged-upload carve-out (unit).** An `awaiting-original` asset with no original blob produces no finding; the same gap on a fully-uploaded asset does. Cross-module case (server crash mid-finalization → recovery on restart) is bounded E2E surface in [Module Map](/design/module-map/#e2e-test-surface). diff --git a/capsule-docs/src/content/docs/design/filesystem/server.md b/capsule-docs/src/content/docs/design/filesystem/server.md index cb87b4d0..add643f3 100644 --- a/capsule-docs/src/content/docs/design/filesystem/server.md +++ b/capsule-docs/src/content/docs/design/filesystem/server.md @@ -57,7 +57,7 @@ A rebuild walks the **envelope objects** under `blobs/`, verifies each device si An envelope object that fails structural validation during rebuild is **quarantined**, not silently dropped — moved to `{blob_root}/quarantine/` with a sibling `.reason.json` recording the rejection code. This guarantees that an unrecoverable byte sequence is preserved for forensic inspection rather than vanishing on rebuild. -Operationally the rebuild is invoked when a PostgreSQL restore is incomplete or a logical-corruption event is detected; it is **never** the hot path. The hot path runs through the authoritative PG index. The recovery path's job is to make the index reconstructible if PG is lost, not to substitute for it. +Operationally the rebuild is invoked when a PostgreSQL restore is incomplete or a logical-corruption event is detected; it is **never** the hot path. The hot path runs through the authoritative PG index. The recovery path's job is to make the index reconstructible if PG is lost, not to substitute for it. Recovery is also not *verification*: the proactive, read-only check that a frozen Postgres + blob-store pair is mutually consistent — the path that detects the drift before anyone needs a rebuild — is the [server-side integrity scrub](/design/filesystem/maintenance/#server-side-integrity-scrub). ## Manifest Envelope Validation From 6a5be56d0e986a08b054be5655c996f8f01fc7b3 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 13:09:48 -0400 Subject: [PATCH 40/58] docs(design): provable server-side custody and loss MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing was server-signed about custody: finalization Completed and the /storage/verify verdict were both explicitly unsigned, so 'the server lost my photo' and 'the client never uploaded it' were symmetric, unfalsifiable claims. Storage Verification now owns two server-signed objects and their composition: - CustodyReceipt — signed inside the finalization transaction (receipt and uploaded flip commit together) over the server-recomputed ciphertext hash, chain-positioned via envelope_hash, and hash-chained into a per-server append-only receipt log (receipt_seq + prior_receipt_hash). Fetched via GET /upload/{id}/receipt and GET /assets/{asset_id}/receipts; persisted client-side as evidentiary {uuid}.receipts.cbor, included in the backup artifact. - StorageAttestation — the verify verdict signed on request with a client nonce echoed, so a stale durable=true cannot answer a fresh challenge. Priced like deep. - Proof of Loss — receipt (acceptance) + signed non-stored attestation or failed content-addressed retrieval (non-holding) + no authorized purge (the delete-manifest rebuttal). The negative direction is a burden shift: a custody claim without a receipt is unfalsifiable noise, so signature unforgeability is the server's negative proof. Split-view of the receipt log is the documented v1 residual, with gossiped log heads as the designated v2 extension. Signing uses a new long-lived hybrid Ed25519+ML-DSA-65 attestation key, distinct from the classical operational key — the primitives carve-out is clarified to be bounded by signature lifetime, and federation's well-known document gains an append-only key history so old receipts verify after rotation. The verify-before-destroy gate now also requires a verified receipt, so a receipt-withholding server never becomes the sole holder of an only-copy. Threat model: invariants 33-34, scenario rows 43-45, client-side receipt bullets. New slice S-C15; S-D4 gains the receipt half of the release gate. error.upload.receipt_not_available added to locales with platform files regenerated. --- SLICES.md | 46 ++++++++-- .../src/androidMain/res/values/strings.xml | 1 + .../docs/design/cryptography/primitives.md | 4 +- .../docs/design/cryptography/provenance.md | 2 +- .../src/content/docs/design/federation.md | 3 +- .../design/import/storage-verification.md | 87 ++++++++++++++++++- .../docs/design/import/upload-protocol.md | 6 +- .../docs/design/threat-model/scenarios.md | 3 + .../docs/design/threat-model/validation.md | 8 +- capsule-i18n/src/bundles/en.json | 1 + capsule-i18n/src/generated.rs | 3 + capsule-swift/Generated/Localizable.xcstrings | 10 +++ capsule-web/src/i18n/messages/en.json | 1 + locales/en.json | 4 + 14 files changed, 164 insertions(+), 15 deletions(-) diff --git a/SLICES.md b/SLICES.md index 983fd962..87717932 100644 --- a/SLICES.md +++ b/SLICES.md @@ -84,10 +84,11 @@ its slice. | S-C12 | Backup escrow server surface | server | — | S | ready | | S-C13 | Session device-cohort storage + grouping | server | — | S | ready | | S-C14 | Server integrity scrub (Postgres⇄blob-store) | server | S-C1 | M | ready | +| S-C15 | Custody receipts + signed storage attestation | server | S-C1, S-C3 | M | ready | | S-D1 | SDK upload client (hand-written, stateful protocol) | sdk/clients | S-C1 | M | ready | | S-D2 | SDK sync/download client + connection-class budget | sdk/clients | S-C2, S-C9 | L | ready | | S-D3 | Web guest drop client (WASM) | sdk/clients | S-A6, S-C5 | L | ready | -| S-D4 | Verify-before-destroy wiring | sdk/clients | S-C3 | M | ready | +| S-D4 | Verify-before-destroy wiring | sdk/clients | S-C3, S-C15 | M | ready | | S-D5 | CLI auth/sync/list | sdk/clients | S-D1, S-D2 | M | ready | | S-D6 | Web server gateway (key-free reads) | sdk/clients | S-D2 | L | ready | | S-D7 | SDK auth/session foundation + auto token refresh | sdk/clients | — | M | ready | @@ -144,6 +145,10 @@ graph LR C9[S-C9 directory] --> C7[S-C7 enrollment] --> E3[S-E3 peering] C9 --> D2 --> D6[S-D6 web gateway] --> G1[S-G1 graphql retire] --> G3[S-G3 entity retire] C3[S-C3 storage verify] --> D4[S-D4 verify-destroy] --> B3[S-B3 streaming import] + C1 --> C15[S-C15 custody receipts] + C3 --> C15 + C15 --> D4 + C1 --> C14[S-C14 integrity scrub] D1 --> B3 D1 --> D5 D2 --> E3 @@ -342,6 +347,8 @@ pass until the gate lifts. counter-increment, and between rename and commit, recovers per the atomicity invariants. - **Tier:** Unit + Smoke + E2E case 2/11. **Blocks:** S-C2, S-C5, S-C11, S-D1, S-B4. + (The custody-receipt insert that joins this finalization transaction is owned by + S-C15 — no scope change here.) ### S-C2 — Key-free sync feed @@ -366,8 +373,10 @@ pass until the gate lifts. - **Deliverable:** `POST /storage/verify` computing stored/indexed/retrievable from the blob store + Postgres, the `deep` re-hash (rate-limited, coalesced), and the GC-grace interaction that keeps a just-verified blob out of byte deletion. -- **Done when:** the storage-verification doc's six Validation bullets pass; the stub's - `todo!()` is gone. **Tier:** Unit + Smoke. **Blocks:** S-D4. +- **Done when:** the storage-verification doc's six unsigned-verdict Validation + bullets pass; the stub's `todo!()` is gone. (The signed `StorageAttestation` + extension of this endpoint is owned by S-C15.) **Tier:** Unit + Smoke. + **Blocks:** S-D4, S-C15. ### S-C4 — Share-link serving @@ -547,9 +556,15 @@ SDK; they never hand-roll network flows. the implemented pure predicate `capsule_core::library::release_is_safe`. - **Deliverable:** the SDK call to `POST /storage/verify` + the 60-second re-verify window, wired into the three destructive paths (device-owned-original release, - Move-import source deletion, streaming release) via `release_is_safe`. -- **Depends on:** S-C3. **Blocks:** S-B3. **Done when:** `storage_verify.rs`'s - `#[ignore]`d wiring test flips; the client.md verify-before-release smoke passes. + Move-import source deletion, streaming release) via `release_is_safe`; plus the + **receipt half of the gate** — fetch, verify (pinned attestation key, field match), + and persist the `CustodyReceipt` (`{uuid}.receipts.cbor`, included in the backup + artifact) for every finalized upload, with release refused when the receipt is + missing or unverified. +- **Depends on:** S-C3; S-C15 (receipt endpoints + attestation key). **Blocks:** S-B3. + **Done when:** `storage_verify.rs`'s `#[ignore]`d wiring test flips; the client.md + verify-before-release smoke passes; the receipt-gated-release smoke + (storage-verification doc) passes. - **Tier:** Unit + Smoke. ### S-D5 — CLI auth/sync/list @@ -665,6 +680,25 @@ SDK; they never hand-roll network flows. no-network-on-read-paths smoke runs with a socket-refusing harness. - **Tier:** Unit + Smoke; the airplane-mode E2E case rides the Module Map surface. +### S-C15 — Custody receipts + signed storage attestation + +- **Contract:** [Storage Verification — Custody Receipts / Signed Storage Attestation / + Proof of Loss](capsule-docs/src/content/docs/design/import/storage-verification.md), + [Validation invariants 33–34](capsule-docs/src/content/docs/design/threat-model/validation.md). +- **Deliverable:** the server **attestation keypair** (hybrid Ed25519+ML-DSA-65, + distinct from the operational key) with well-known publication + append-only key + history (federation doc); `CustodyReceipt` signing hooked into S-C1's finalization + transaction (receipt + `uploaded` flip commit together) with the per-server + `receipt_seq` chain; `GET /upload/{id}/receipt` + `GET /assets/{asset_id}/receipts` + (`error.upload.receipt_not_available` before Completed — key already in `locales/`); + `signed`/`nonce` on `POST /storage/verify` returning `StorageAttestation`, + rate-limited like `deep`. +- **Depends on:** S-C1 (finalization transaction), S-C3 (verify endpoint). +- **Done when:** the storage-verification doc's receipt/attestation/proof-of-loss + Validation bullets pass (issuance atomicity, log monotonicity, nonce echo, + loss-proof composition, delete rebuttal, cross-server replay, rotation continuity). +- **Tier:** Unit + Smoke. **Blocks:** the receipt half of S-D4's release gate. + ### S-D15 — Exact client build identification - **Contract:** [Provenance — Client Build Identification](capsule-docs/src/content/docs/design/cryptography/provenance.md). diff --git a/capsule-android/src/androidMain/res/values/strings.xml b/capsule-android/src/androidMain/res/values/strings.xml index 03aebd00..2df2fef4 100644 --- a/capsule-android/src/androidMain/res/values/strings.xml +++ b/capsule-android/src/androidMain/res/values/strings.xml @@ -29,6 +29,7 @@ The upload request was missing its position marker. The upload got out of sync and is realigning. You can\'t upload on behalf of this user. + The upload hasn\'t finished, so its storage receipt isn\'t available yet. This upload has already finished. The upload session expired. Starting over… The upload sent more data than it declared. diff --git a/capsule-docs/src/content/docs/design/cryptography/primitives.md b/capsule-docs/src/content/docs/design/cryptography/primitives.md index b7138a42..bd28b3d1 100644 --- a/capsule-docs/src/content/docs/design/cryptography/primitives.md +++ b/capsule-docs/src/content/docs/design/cryptography/primitives.md @@ -85,9 +85,9 @@ The salt is always a 32-byte CSPRNG draw. The tier chosen at *wrap* time is reco ### Signature Scheme -**Hybrid Ed25519 + ML-DSA-65** for all long-lived **identity** signatures: the user IK, device keys, asset manifests, and write-tier keys. Both halves must verify before a peer is accepted, so neither algorithm being broken alone compromises authentication. +**Hybrid Ed25519 + ML-DSA-65** for all long-lived **identity** signatures: the user IK, device keys, asset manifests, write-tier keys — and the server's [custody receipts and storage attestations](/design/import/storage-verification/#custody-receipts). Both halves must verify before a peer is accepted, so neither algorithm being broken alone compromises authentication. -**Short-lived operational signatures are classical Ed25519 only** — server-to-server federation, [federation capability tokens](/design/federation/#federation-capabilities), and [access-token JWTs](/design/authentication/#access-token). These live minutes to hours and rotate cheaply, so PQ hybridization buys no meaningful margin (a harvest-now-decrypt-later adversary gains nothing from a long-expired signature) and is not worth the wire-size and verification cost. This carve-out is owned here; consumers link to it rather than restating the choice. +**Short-lived operational signatures are classical Ed25519 only** — server-to-server federation, [federation capability tokens](/design/federation/#federation-capabilities), and [access-token JWTs](/design/authentication/#access-token). These live minutes to hours and rotate cheaply, so PQ hybridization buys no meaningful margin (a harvest-now-decrypt-later adversary gains nothing from a long-expired signature) and is not worth the wire-size and verification cost. This carve-out is owned here; consumers link to it rather than restating the choice. The boundary is signature **lifetime**, not who signs: a custody receipt is server-signed but is evidence for the life of the asset, so it is hybrid, outside the carve-out. MLS LeafNode signatures stay Ed25519-only (pinned by the ciphersuite); the ML-DSA half of a device's identity lives at the identity layer — see [MLS](/design/cryptography/mls/). diff --git a/capsule-docs/src/content/docs/design/cryptography/provenance.md b/capsule-docs/src/content/docs/design/cryptography/provenance.md index b4f4f9c8..497822b1 100644 --- a/capsule-docs/src/content/docs/design/cryptography/provenance.md +++ b/capsule-docs/src/content/docs/design/cryptography/provenance.md @@ -6,7 +6,7 @@ status: draft Every asset Capsule stores has a verifiable trace of *who* produced it. The trace is anchored in a small **signed manifest** — bound to the ciphertext, cheap to verify, streaming-compatible — and extended by an **append-only, hash-chained provenance log per asset**. Together these are what let an operator distinguish a legitimate delete from a malicious or bug-induced one after the fact, and what defeats the [stale-revival attack](/design/threat-model/scenarios/#damage-scenario--invariant-map). -The schemas live here and are the **single source of truth** for `AssetManifest`, `ProvenanceRecord`, and `DerivativeManifest`. They are implemented in `capsule-core::crypto::provenance`; verification flows through the single `verify_asset` chokepoint in `capsule-core::crypto` ([Write Authorization](/design/cryptography/keys/#write-authorization)). +The schemas live here and are the **single source of truth** for `AssetManifest`, `ProvenanceRecord`, and `DerivativeManifest`. They are implemented in `capsule-core::crypto::provenance`; verification flows through the single `verify_asset` chokepoint in `capsule-core::crypto` ([Write Authorization](/design/cryptography/keys/#write-authorization)). Everything here is **client-signed** — the server-signed complement, the custody receipt that makes server-side loss provable in both directions, is owned by [Storage Verification — Custody Receipts](/design/import/storage-verification/#custody-receipts). ## Asset Manifest diff --git a/capsule-docs/src/content/docs/design/federation.md b/capsule-docs/src/content/docs/design/federation.md index beeb45e5..233c0756 100644 --- a/capsule-docs/src/content/docs/design/federation.md +++ b/capsule-docs/src/content/docs/design/federation.md @@ -185,7 +185,8 @@ Federation introduces moderation hooks for handling abuse across servers; the fu ## Server Identity and Key Rotation -- Server-to-server requests are signed under the server's signing key (classical Ed25519 only, per the [operational-signature carve-out](/design/cryptography/primitives/#signature-scheme)), published at a well-known path. Matrix, ActivityPub (HTTP Signatures), and AT Protocol all converge on this pattern. +- A server holds **two** signing identities, split by signature lifetime. The **operational key** signs server-to-server requests (classical Ed25519 only, per the [operational-signature carve-out](/design/cryptography/primitives/#signature-scheme)). The **attestation key** — hybrid Ed25519 + ML-DSA-65 — signs long-lived evidence: [custody receipts and storage attestations](/design/import/storage-verification/#custody-receipts). Both are published at the same well-known path; Matrix, ActivityPub (HTTP Signatures), and AT Protocol all converge on the well-known-key pattern. +- The well-known document keeps an **append-only key history** for the attestation key (`key_id` → public key, active-from/active-to), never dropping a retired entry, so a receipt signed years ago still verifies; a receipt's `server_key_id` selects its verification key. The operational key needs no history — nothing it signs outlives it. - Servers cache each other's public keys (TOFU-pinned on first contact). A rotation is confirmed by a **perspective check**: before accepting a rotated key, a peer corroborates it against **at least two** independent vantage points — federated servers it already trusts, or a deployment-configured notary — and accepts only on **unanimous** agreement, so a single compromised network path (or a single colluding vantage) cannot substitute a forged key. A rotation that fails corroboration is surfaced, not silently accepted. This is the mechanism behind [Threat Model — scenario #26](/design/threat-model/scenarios/#damage-scenario--invariant-map). - Album protocol versions are pinned per album — see [Album Protocol Version Pinning](/design/versioning/#album-protocol-version-pinning). diff --git a/capsule-docs/src/content/docs/design/import/storage-verification.md b/capsule-docs/src/content/docs/design/import/storage-verification.md index 16cc1af2..e1eadd8f 100644 --- a/capsule-docs/src/content/docs/design/import/storage-verification.md +++ b/capsule-docs/src/content/docs/design/import/storage-verification.md @@ -6,7 +6,7 @@ status: draft [Upload finalization](/design/import/upload-protocol/#finalization-and-integrity) confirms, **once**, that the bytes the server stored match the declared hash. That `Completed` acknowledgement is a transfer receipt, not a standing durability guarantee a client can re-check later — and [`verify_asset`](/design/cryptography/keys/#write-authorization) proves *cryptographic* validity and authorization, never that the server still holds the bytes. A client that is about to discard local data therefore has no way, today, to ask the server: *do you actually still have this, indexed and retrievable?* -This doc defines that missing query and the rule that every client follows before any destructive local action. The endpoint lives in `capsule-api-media` (planned; it answers from the blob store plus the Postgres index); the client-side gate is a pure predicate in `capsule-core` invoked from `capsule-sdk` (planned — the offline core's `verify_asset` already covers the complementary crypto-validity half of the gate). +This doc defines that missing query and the rule that every client follows before any destructive local action. The endpoint lives in `capsule-api-media` (planned; it answers from the blob store plus the Postgres index); the client-side gate is a pure predicate in `capsule-core` invoked from `capsule-sdk` (planned — the offline core's `verify_asset` already covers the complementary crypto-validity half of the gate). It is also the **single source of truth** for the two server-signed objects that make custody and loss *provable* rather than merely checkable: the [`CustodyReceipt`](#custody-receipts) and the [signed storage attestation](#signed-storage-attestation), composed in [Proof of Loss](#proof-of-loss). ## What "Safely Stored" Means @@ -62,9 +62,82 @@ StorageVerification { The endpoint is deliberately cheap and idempotent so clients can call it freely on the destructive path below. +## Custody Receipts + +Everything above answers *is it still there?* — a cooperative, unsigned, point-in-time fact. It cannot answer the accountability question: *can either side later prove what the server was entrusted with?* Without that, "the server lost my photo" and "the client never uploaded it" are symmetric, unfalsifiable claims. The custody receipt closes both directions. + +```rust +CustodyReceipt { + version: "custody-receipt/v1", + crypto_suite_id: u16, + protocol_version: String, // YYYY-MM-DD + server_id: String, // this server's canonical origin — binds the receipt to one server + server_key_id: bytes, // fingerprint of the attestation key that signed; survives rotation + receipt_seq: u64, // strictly monotonic per server + prior_receipt_hash: Option<[u8;32]>, // SHA-256 over the previous receipt in the server's log; null only + // for the first receipt — the provenance chain's append-only + // discipline, applied to the server's own log + upload_id: UUID, // the session that produced custody + asset_id: UUID, + blob_role: enum, // original | derivative | metadata | provenance + ciphertext_hash: bytes, // RECOMPUTED by the server at finalization — never echoed from the client + size: u64, + envelope_hash: Option<[u8;32]>, // SHA-256 of the manifest envelope CBOR — binds the receipt to the + // asset's provenance-chain position + uploaded_by_user: UUID, + uploaded_by_device: Option, + received_at: RFC3339, // server's trusted clock at the finalization commit + + server_sig: Hybrid(Ed25519, ML-DSA-65), // under the server attestation key, over all fields above +} +``` + +Canonical CBOR with the same [wire-presence discipline](/design/cryptography/provenance/#asset-manifest) as manifests (absent optionals encode as absent keys), and deliberately server-visible — it carries ciphertext hashes only, no plaintext. Receipts are the server-signed complement of the client-signed [provenance chain](/design/cryptography/provenance/#provenance-of-library-modifications): the envelope proves what a client *claimed and signed*; the receipt proves what the server *accepted*, over a hash it recomputed itself. + +- **Issued inside the finalization transaction** ([Upload Protocol — Finalization](/design/import/upload-protocol/#finalization-and-integrity)): the receipt insert commits atomically with the `uploaded` flip. No receipt without durable custody; no custody-marking without a receipt. +- **Signed under the server attestation key** — a long-lived **hybrid Ed25519 + ML-DSA-65** keypair, distinct from the classical [operational key](/design/federation/#server-identity-and-key-rotation). A receipt is evidence for the life of the asset, so it sits outside the [operational-signature carve-out](/design/cryptography/primitives/#signature-scheme). The key is published, pinned, and rotated with the server identity, with an **append-only key history** so old receipts verify forever; `server_key_id` selects the verification key. +- **Hash-chained and enumerable.** `receipt_seq` + `prior_receipt_hash` form the server's append-only receipt log, persisted like envelopes — Postgres rows plus content-addressed objects in the blob store — with **no API path that overwrites or deletes** an entry. The chain lets the server affirmatively enumerate everything it has ever accepted, and bounds post-key-compromise forgery: a "receipt" that does not chain into the log is provably out-of-band. +- **Fetched** via `GET /upload/{id}/receipt` in the session window (pairing with the [lost-ACK recovery flow](/design/import/upload-protocol/#idempotency-and-resumption)) and `GET /assets/{asset_id}/receipts` durably (uploader or owner). Receipts are permanent objects, exempt from session GC. +- **Client persistence and verification.** The client appends receipts beside the provenance chain (`media/{YYYY}/{YYYY-MM}/{uuid}.receipts.cbor`) and includes them in the [backup artifact](/design/backup-recovery/). Unlike the provenance cache, this copy is **evidentiary, not merely a cache** — a server destroying the record of its own liability is exactly the adversary — so the client copy is first-class. On fetch the client verifies the signature under the pinned attestation key, that `ciphertext_hash`/`size`/`envelope_hash` match what it sent, and that `received_at` passes a gross-drift sanity bound (evidentiary only; ordering rides the chain, mirroring the manifest [`timestamp` rule](/design/cryptography/keys/#write-authorization)). + +## Signed Storage Attestation + +The `/storage/verify` verdict gains an on-request signed form: the request accepts `signed: true` and an optional client `nonce`. + +```rust +StorageAttestation { + version: "storage-attestation/v1", + crypto_suite_id: u16, + server_id: String, + server_key_id: bytes, + nonce: Option, // client-supplied freshness challenge, echoed verbatim — + // a stale durable=true cannot be replayed as current + verdict: StorageVerification, // the unchanged verdict above, incl. checked_at + server_sig: Hybrid(Ed25519, ML-DSA-65), // attestation key, over all fields above +} +``` + +Signing is server-priced exactly like `deep` — rate-limited per user and coalesced — so the hot verify-before-destroy loop stays on the unsigned verdict. The signed form exists for *evidence*: an honest server self-reporting loss (`stored = false` under its own signature) and periodic signed audits a client retains. + +## Proof of Loss + +The two objects compose into transferable proof, with the burden of proof placed where it can actually be discharged. + +**A client proves the server lost entrusted bytes** with three elements: + +1. **Acceptance** — a `CustodyReceipt` for hash `H`, verifying under the server's published attestation key: the server took custody at `received_at`. +2. **Later non-holding** — either a signed `StorageAttestation` reporting `H` non-`stored`/non-`durable` (the cooperative path), or a **failed retrieval challenge**: anyone can fetch `H` by content address, and content addressing makes the response self-verifying, so a 404 or wrong bytes is independently checkable evidence even against an uncooperative server. +3. **No authorized purge** — the server's legitimate rebuttal is the asset's own provenance chain it already holds: a device-signed `delete` manifest with an elapsed [`retention_until`](/design/organization/#retention-window) proves it was *dis*entrusted, not negligent. A verifier checks any presented delete against the receipt's `envelope_hash` chain position. + +**The server proves what it was — and was not — entrusted with** by a burden shift: a custody claim is valid only when backed by a receipt under the server's signature, so *"no valid receipt exists"* is the negative proof, resting on signature unforgeability rather than log completeness. The chained receipt log additionally lets the server enumerate its accepted set for audit; what clients *claimed* is already the device-signed envelope chain it persists. + +Residual limitation, accepted for v1: a dishonest server can show different receipt-log views to different auditors (split view). Any client holding a receipt at `receipt_seq = N` proves the log has at least `N` entries, which bounds silent truncation; gossiped signed log heads over the sync feed are the designated v2 extension — the receipt-log counterpart of the provenance chain's [v2 checkpoint note](/design/cryptography/provenance/#chained-append-only-structure). + +A server that simply **refuses to issue receipts** gains nothing: the release gate below refuses to ever make such a server the sole holder of an only-copy. + ## Verify Before Destroy -The standing rule, enforced across every client: **after any write the server is expected to durably persist, a client confirms `durable = true` from `/storage/verify` (and that [`verify_asset`](/design/cryptography/keys/#write-authorization) accepts the asset) before any post-write local cleanup of data tied to that write.** The two checks are complementary and both required — `verify_asset` for cryptographic validity, `/storage/verify` for server durability — before irreplaceable local bytes are dropped. +The standing rule, enforced across every client: **after any write the server is expected to durably persist, a client (1) holds and has verified the write's [`CustodyReceipt`](#custody-receipts), (2) confirms `durable = true` from `/storage/verify`, and (3) confirms [`verify_asset`](/design/cryptography/keys/#write-authorization) accepts the asset — before any post-write local cleanup of data tied to that write.** The three checks are complementary and all required before irreplaceable local bytes are dropped: the receipt for accountability (a server that withholds receipts never becomes the sole holder of an only-copy), `/storage/verify` for durability, `verify_asset` for cryptographic validity. Concretely, the gate applies to: @@ -88,6 +161,7 @@ A non-`durable` verdict never triggers a destructive action: the client retains - **vs. upload finalization.** Finalization's `Completed` is a one-time receipt for one transfer; `/storage/verify` is the re-checkable, after-the-fact confirmation that the receipt still holds — across app restarts, across devices, and after server-side GC or migration could have changed state. - **vs. `verify_asset`.** `verify_asset` is local and key-aware (signatures, provenance chain, write authorization); `/storage/verify` is remote and key-free (does the server physically have and serve these bytes). Neither subsumes the other. - **vs. the sync feed.** The sync feed tells a client *what exists*; `/storage/verify` tells it *whether a specific blob is durably serveable right now* — the question that gates destruction. +- **vs. custody receipts.** The verdict says whether the bytes are serveable *now*; the [receipt](#custody-receipts) proves the server accepted them *then*. Detection needs the verdict; accountability needs both (see [Proof of Loss](#proof-of-loss)). ## Validation @@ -97,5 +171,14 @@ A non-`durable` verdict never triggers a destructive action: the client retains - **Wrong-hash declaration (unit).** Declare a blob hash the server does not hold for the asset; assert `stored = false, indexed = false` rather than a silent omission. - **Verify-before-destroy gate (smoke).** Drive a device-owned-original release with the endpoint mocked to return non-`durable`; assert the client refuses to evict and surfaces the unconfirmed state; flip to `durable`; assert the release proceeds. - **Deep scan (unit).** Corrupt a stored blob's bytes on disk; assert the structural check still reports `stored=true` but `deep = true` reports a hash mismatch. +- **Receipt issuance (smoke).** Upload to `Completed`; fetch the receipt; verify the hybrid signature against the published attestation key; assert `ciphertext_hash`, `size`, `blob_role`, `envelope_hash`, and `received_at` match the finalized state. +- **No receipt without custody (unit).** Inject a failure between hash verification and the finalization commit; assert no receipt exists and `receipt_seq` did not advance — receipt and `uploaded` flip are atomic. +- **Receipt log append-only + monotonic (unit).** Finalize two blobs; assert strictly increasing `receipt_seq` and correct `prior_receipt_hash` chaining; assert any overwrite or delete attempt on a receipt is rejected at the structural layer (invariant 33). +- **Signed attestation (unit).** Request `signed: true` with a nonce; verify the signature and the nonce echo; mutate any verdict field or the nonce; assert verification fails (invariant 34). +- **Proof-of-loss composition (smoke).** Finalize and fetch the receipt; delete the blob bytes server-side; a signed verify reports `stored = false`; assert the (receipt, attestation) pair verifies as a loss proof under the same `server_key_id`, and that a content-addressed fetch of the hash fails. +- **Delete rebuttal (unit).** Finalize, then run a signed `delete` plus retention purge; assert the server can present receipt + delete envelope such that a verifier classifies the non-holding as authorized purge, not loss. +- **Cross-server replay rejection (unit).** Present server A's receipt as evidence against server B; the verifier rejects on the `server_id`/`server_key_id` binding. +- **Receipt-gated release (smoke).** Drive a device-owned-original release with the receipt fetch failing; assert the client refuses to release even on `durable = true`; restore the endpoint; the release proceeds. +- **Rotation continuity (unit).** Rotate the attestation key; assert a pre-rotation receipt still verifies via `server_key_id` against the published key history. The cross-module case — upload → finalize → `/storage/verify` durable → safe local release — is exercised inside the [full lifecycle](/design/module-map/#e2e-test-surface) E2E surface rather than as a standalone case. diff --git a/capsule-docs/src/content/docs/design/import/upload-protocol.md b/capsule-docs/src/content/docs/design/import/upload-protocol.md index 2e34cd73..161b08dc 100644 --- a/capsule-docs/src/content/docs/design/import/upload-protocol.md +++ b/capsule-docs/src/content/docs/design/import/upload-protocol.md @@ -49,6 +49,7 @@ All endpoints are authenticated with a bearer JWT. | `PATCH` | `/upload/{id}` | Append a chunk at `X-Capsule-Offset`, with `Content-Type: application/octet-stream` and the **required** per-chunk `X-Capsule-Checksum`. Returns `204` and the new offset. | | `DELETE` | `/upload/{id}` | Cancel the session — removes the upload file, the session record, and the pending asset row. Rejected with `409` while finalization is running. | | `GET` | `/upload/sessions` | List the **uploader's** active sessions (keyed by `upload_user_id` — the resuming party — not `owner_id`), so a client can resume across app restarts. | +| `GET` | `/upload/{id}/receipt` | Fetch the session's [`CustodyReceipt`](/design/import/storage-verification/#custody-receipts) once `Completed`. `409 error.upload.receipt_not_available` before then; receipts are permanent, so this also works after the session record itself expires — durably via `GET /assets/{asset_id}/receipts`. | Creating a session writes a *pending* asset row to Postgres (`uploaded = false`) and a session record to **Valkey** (see [Filesystem — Server: Required Services](/design/filesystem/server/#required-services)). The pending row reserves the asset ID that derivative and metadata blobs reference. The session record carries everything finalization needs — declared size and hash, `content_type`, `crypto_suite_id`, `protocol_version`, `blob_role`, the manifest envelope, `album_id`/`owner_id`/`intent_id`, and the uploader — so a session is finalizable from its own record with no further client input. @@ -161,12 +162,12 @@ When received bytes reach the declared size, the server finalizes: 1. Session transitions to **WaitingForProcessing** via an **atomic compare-and-set** on the status — two racing finalizers cannot both proceed; the loser observes the transition and returns a conflict. 2. The server recomputes the [content hash](/design/cryptography/primitives/) over the upload file on the blocking pool and compares it to the declared `hash`. 3. The manifest envelope is re-validated (invariant 15) — the same checks as at session creation, inside the finalization transaction, so nothing that changed since creation (a revoked device, a closed album) slips through. -4. **On match** — the file is renamed to its content address, the pending asset is marked uploaded inside a Postgres transaction, and the session transitions to **Completed**. +4. **On match** — the file is renamed to its content address; then, inside one Postgres transaction, the pending asset is marked uploaded and the write's [`CustodyReceipt`](/design/import/storage-verification/#custody-receipts) is signed and persisted — the receipt and the `uploaded` flip commit or fail together. The session transitions to **Completed**. 5. **On mismatch** — the upload file and the pending asset row are deleted, the session transitions to **FailedProcessing**, and `error.upload.content_hash_mismatch` is returned. A mismatch is always treated as corruption or tampering and is never silently retried server-side. The server verifies only the *ciphertext* hash — it has no other option. The client independently verifies the *plaintext* on download via the [STREAM construction](/design/cryptography/encryption/#stream-construction)'s per-chunk authentication tags, which detect truncation, reordering, and chunk deletion. The two checks are complementary: the server guarantees "the bytes I stored are the bytes you declared," and the AEAD guarantees "the plaintext I decrypted is authentic." -`Completed` is a one-time transfer receipt, not a standing durability guarantee a client can re-query later. After finalization, a client confirms an asset remains durably stored, indexed, and retrievable — the precondition for releasing its local copy — through the separate [storage-verification endpoint](/design/import/storage-verification/), which re-checks state that server-side GC, migration, or corruption could change after `Completed`. +`Completed` is a one-time transfer acknowledgement, not a standing durability guarantee a client can re-query later — but it is now accompanied by a durable, server-signed [`CustodyReceipt`](/design/import/storage-verification/#custody-receipts): the transferable evidence that this server accepted custody of exactly these bytes, fetched via `GET /upload/{id}/receipt` and persisted client-side. After finalization, a client confirms an asset *remains* durably stored, indexed, and retrievable — the precondition for releasing its local copy — through the separate [storage-verification endpoint](/design/import/storage-verification/), which re-checks state that server-side GC, migration, or corruption could change after `Completed`. ## Idempotency and Resumption @@ -222,6 +223,7 @@ Every rejection carries a machine-readable `error.*` code (the [i18n error-code | Missing/invalid `X-Capsule-Offset` header | `400` | `error.upload.missing_offset` | 9 | | Missing/malformed `X-Capsule-Checksum` | `400` | `error.upload.missing_checksum` | 12 | | Checksum ≠ received bytes | `400` | `error.upload.checksum_mismatch` | 12 | +| Receipt requested before `Completed` | `409` | `error.upload.receipt_not_available` | 33 | | Offset ≠ current received count | `409` | `error.upload.offset_mismatch` | 9 | | Same offset, different chunk hash | `409` | `error.upload.chunk_conflict` | 12 | | Cumulative bytes exceed declared `size` | `400` | `error.upload.size_exceeded` | 11 | diff --git a/capsule-docs/src/content/docs/design/threat-model/scenarios.md b/capsule-docs/src/content/docs/design/threat-model/scenarios.md index 82e98099..1ee36596 100644 --- a/capsule-docs/src/content/docs/design/threat-model/scenarios.md +++ b/capsule-docs/src/content/docs/design/threat-model/scenarios.md @@ -52,6 +52,9 @@ The lookup table for "what damage X is prevented by which invariant Y in which d | 40 | A guest brute-forces an upload link's optional passphrase to spend the owner's quota | The stored Argon2id verifier makes every guess pay the KDF cost; per-IP + per-`{opaque-id}` rate limits throttle attempts; per-link caps and quota-at-creation bound the damage even on success | [Web Upload — Security Contract](/design/web-upload/#security-contract) | | 41 | A peer doctors a capability's `min_protocol_version` to coax a downgraded parse of album events | The claim is inside the Ed25519-signed token — tampering breaks the signature — and every pulled event still passes the receiver's own protocol handshake and the album's pin regardless of what the token claims | [Federation — Federation Capabilities](/design/federation/#federation-capabilities) | | 42 | A share link's passphrase is brute-forced offline from the fetched wrapped secret | Argon2id cost per guess on the client-side unwrap; the wrapped-secret endpoint is rate-limited per IP + per link; the fragment-delivered link secret itself is never server-held, so the server cannot shortcut the search | [Share Links — Security Contract](/design/share-links/#security-contract) | +| 43 | A server loses entrusted bytes and denies ever having accepted them | The hybrid-signed `CustodyReceipt` issued inside the finalization transaction proves acceptance; a failed content-addressed retrieval (self-verifying) or a signed non-`stored` attestation proves later non-holding; a device-signed `delete` + elapsed retention is the server's only rebuttal | [Storage Verification — Proof of Loss](/design/import/storage-verification/#proof-of-loss) | +| 44 | A client fabricates a claim that it entrusted the server with bytes it never sent | Custody claims are valid only when backed by a receipt under the server's attestation signature — no receipt, no claim; the chained receipt log lets the server enumerate everything it did accept | [Storage Verification — Proof of Loss](/design/import/storage-verification/#proof-of-loss) | +| 45 | A receipt is replayed against a different server, or a stale `durable` attestation replayed as current | `server_id` + `server_key_id` are inside the signed receipt body, binding it to one server; signed attestations echo a client-chosen `nonce`, so a stale verdict cannot answer a fresh challenge | [Storage Verification — Custody Receipts](/design/import/storage-verification/#custody-receipts) | When a scenario surfaces during implementation that does not match any of the above, the rule is: add a row here, then declare the defense in exactly one owner doc. Never restate a defense in multiple docs. diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index 20ccc991..03741564 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -78,6 +78,11 @@ A [web-upload](/design/web-upload/) drop carries **no `AssetManifest`** — no s Drop **chunks** reuse the `PATCH` chunk rules (9–12) and **finalization** reuses the integrity checks (13–14) unchanged; only drop-session creation (26–31) and adoption (32) differ from the album upload path. +### On custody receipts and signed attestations + +- **33.** A [`CustodyReceipt`](/design/import/storage-verification/#custody-receipts) is signed and persisted **only** inside the finalization transaction that durably commits the blob and flips `uploaded` — both or neither. `receipt_seq` is strictly monotonic per server; `prior_receipt_hash` matches the preceding receipt's content hash; no API path overwrites or deletes an existing receipt. A receipt request before `Completed` is `409 error.upload.receipt_not_available`. Owner: [Storage Verification — Custody Receipts](/design/import/storage-verification/#custody-receipts). +- **34.** A signed [`StorageAttestation`](/design/import/storage-verification/#signed-storage-attestation) echoes the client-supplied `nonce` verbatim and is signed over the same state read that produced its verdict — never over state older than the unsigned path would have returned. Owner: [Storage Verification — Signed Storage Attestation](/design/import/storage-verification/#signed-storage-attestation). + Every rejection carries a machine-readable [`error.*` code](/design/i18n/#server-error-codes) alongside its transport status — the code, never the bare status, is what clients switch on and localize (see [API Surfaces — Rejection Mapping](/design/api-surfaces/#rejection-mapping)) — and is logged with it; the rejected hash is remembered (bounded, see [Federation — Soft-Fail Semantics](/design/federation/#soft-fail-semantics)) so divergence between Capsule's view and a permissive peer's view is detectable. ## Client-Side Validation Invariants @@ -90,7 +95,8 @@ Mirror checklist that every client implements before applying any received data - Reject an unknown enum value for any field whose enum is closed at the current schema (notably `action`, `content_type`, `gps.source`, `DerivativeManifest.role`). Unknown CBOR map keys are preserved per [Postel's Law](/design/principles/#postels-law-asymmetric) and never executed. - Maintain a local `latest_provenance_hash` per `asset_id`. Refuse to apply any manifest whose `prior_provenance_hash` is behind the local value. Surface it. - Round-trip the metadata blob on decode: the plaintext sidecar a client persists MUST be byte-identical to the canonical CBOR obtained by decrypting the asset's metadata blob, and the blob's content hash MUST equal the manifest's `metadata_blob_hash`. A divergence is quarantined, never persisted. See [Metadata — Local and Server Metadata Equivalence](/design/metadata/#local-and-server-metadata-equivalence). -- Before any post-write local cleanup that would discard the only copy of irreplaceable bytes — releasing a device-owned original, deleting a move-import source, streaming-mode release — confirm a `durable` verdict from the [storage-verification endpoint](/design/import/storage-verification/#verify-before-destroy) *in addition to* a `verify_asset` accept. A non-`durable` verdict means the local copy is retained, not dropped. This does not gate intentional deletes (trash/hard-purge) or reclaiming rebuildable/re-fetchable data. +- Fetch, verify, and persist the [`CustodyReceipt`](/design/import/storage-verification/#custody-receipts) for every finalized upload: signature under the pinned attestation key, `ciphertext_hash`/`size`/`envelope_hash` matching what was sent. A receipt that fails verification is surfaced, and the write is treated as unconfirmed. +- Before any post-write local cleanup that would discard the only copy of irreplaceable bytes — releasing a device-owned original, deleting a move-import source, streaming-mode release — confirm a verified [`CustodyReceipt`](/design/import/storage-verification/#custody-receipts) **and** a `durable` verdict from the [storage-verification endpoint](/design/import/storage-verification/#verify-before-destroy) *in addition to* a `verify_asset` accept. A missing receipt or non-`durable` verdict means the local copy is retained, not dropped. This does not gate intentional deletes (trash/hard-purge) or reclaiming rebuildable/re-fetchable data. - Maintain a per-user `directory_version` high-water mark. Refuse a `DeviceDirectory` whose `directory_version` is below it (a server attempting to roll back a revocation or hide a device); pin and surface the regression. - Reject an OR-set remove whose `add_id` was never observed locally as an add. - Refuse to follow a `revoke_all_sessions` confirmation that did not include a master-key proof. diff --git a/capsule-i18n/src/bundles/en.json b/capsule-i18n/src/bundles/en.json index 1fcd7200..7e7954db 100644 --- a/capsule-i18n/src/bundles/en.json +++ b/capsule-i18n/src/bundles/en.json @@ -28,6 +28,7 @@ "error.upload.missing_offset": "The upload request was missing its position marker.", "error.upload.offset_mismatch": "The upload got out of sync and is realigning.", "error.upload.owner_not_permitted": "You can't upload on behalf of this user.", + "error.upload.receipt_not_available": "The upload hasn't finished, so its storage receipt isn't available yet.", "error.upload.session_not_active": "This upload has already finished.", "error.upload.session_not_found": "The upload session expired. Starting over…", "error.upload.size_exceeded": "The upload sent more data than it declared.", diff --git a/capsule-i18n/src/generated.rs b/capsule-i18n/src/generated.rs index 8bd634dc..0ae1c242 100644 --- a/capsule-i18n/src/generated.rs +++ b/capsule-i18n/src/generated.rs @@ -97,6 +97,9 @@ pub mod error_codes { /// `error.upload.owner_not_permitted` pub const UPLOAD_OWNER_NOT_PERMITTED: &str = "error.upload.owner_not_permitted"; + /// `error.upload.receipt_not_available` + pub const UPLOAD_RECEIPT_NOT_AVAILABLE: &str = "error.upload.receipt_not_available"; + /// `error.upload.session_not_active` pub const UPLOAD_SESSION_NOT_ACTIVE: &str = "error.upload.session_not_active"; diff --git a/capsule-swift/Generated/Localizable.xcstrings b/capsule-swift/Generated/Localizable.xcstrings index bfa149e0..f125f917 100644 --- a/capsule-swift/Generated/Localizable.xcstrings +++ b/capsule-swift/Generated/Localizable.xcstrings @@ -291,6 +291,16 @@ } } }, + "error.upload.receipt_not_available": { + "localizations": { + "en": { + "stringUnit": { + "state": "translated", + "value": "The upload hasn't finished, so its storage receipt isn't available yet." + } + } + } + }, "error.upload.session_not_active": { "localizations": { "en": { diff --git a/capsule-web/src/i18n/messages/en.json b/capsule-web/src/i18n/messages/en.json index 1fcd7200..7e7954db 100644 --- a/capsule-web/src/i18n/messages/en.json +++ b/capsule-web/src/i18n/messages/en.json @@ -28,6 +28,7 @@ "error.upload.missing_offset": "The upload request was missing its position marker.", "error.upload.offset_mismatch": "The upload got out of sync and is realigning.", "error.upload.owner_not_permitted": "You can't upload on behalf of this user.", + "error.upload.receipt_not_available": "The upload hasn't finished, so its storage receipt isn't available yet.", "error.upload.session_not_active": "This upload has already finished.", "error.upload.session_not_found": "The upload session expired. Starting over…", "error.upload.size_exceeded": "The upload sent more data than it declared.", diff --git a/locales/en.json b/locales/en.json index e9d8c8cd..2d055f85 100644 --- a/locales/en.json +++ b/locales/en.json @@ -167,6 +167,10 @@ "message": "This upload is already being finalized.", "context": "HTTP 409: a concurrent finalization attempt lost the atomic status compare-and-set." }, + "error.upload.receipt_not_available": { + "message": "The upload hasn't finished, so its storage receipt isn't available yet.", + "context": "HTTP 409, invariant 33: GET /upload/{id}/receipt before the session reached Completed. The receipt is signed inside the finalization transaction; the client retries after Completed." + }, "error.upload.content_hash_mismatch": { "message": "The upload failed integrity verification and was discarded.", "context": "HTTP 400 at finalization, invariant 14: recomputed ciphertext hash differs from the declared hash. Treated as corruption/tampering; blob and pending row are deleted." From f1a4ee91d92c34ab7f19ac5f8e0c1c27b95d4489 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Sun, 5 Jul 2026 15:56:53 -0400 Subject: [PATCH 41/58] fix: align title and badge properly by adjusting container margins The h1's default top margin causes asymmetric spacing when using flex alignment. Move the margin to the container and switch to center alignment for consistent visual centering. --- capsule-docs/src/components/PageTitle.astro | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/capsule-docs/src/components/PageTitle.astro b/capsule-docs/src/components/PageTitle.astro index 17d3820f..5cd3e175 100644 --- a/capsule-docs/src/components/PageTitle.astro +++ b/capsule-docs/src/components/PageTitle.astro @@ -28,8 +28,17 @@ const badge = From 2e5f129d1a2283bfa8222804268d804f85f7f638 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 01:19:35 -0400 Subject: [PATCH 42/58] chore: add docs symlink to capsule-docs --- docs | 1 + 1 file changed, 1 insertion(+) create mode 120000 docs diff --git a/docs b/docs new file mode 120000 index 00000000..2e19d7d4 --- /dev/null +++ b/docs @@ -0,0 +1 @@ +capsule-docs/src/content/docs \ No newline at end of file From 2097b9a0d4cb589ed5b98ef80acee7c6b427f5ed Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 03:40:32 -0400 Subject: [PATCH 43/58] docs(design): mark the design overview stable --- capsule-docs/src/content/docs/design/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/capsule-docs/src/content/docs/design/index.md b/capsule-docs/src/content/docs/design/index.md index f2109d47..31e305a3 100644 --- a/capsule-docs/src/content/docs/design/index.md +++ b/capsule-docs/src/content/docs/design/index.md @@ -1,7 +1,7 @@ --- title: Design Overview description: How Capsule's design docs are organized and where to start -status: draft +status: stable --- Capsule is an end-to-end-encrypted personal photo and media store with optional federation. These design docs are its normative specification: every primitive, schema, and protocol is declared in exactly one **owner doc** and referenced by anchor everywhere else (the [Single Source of Truth rule](/design/principles/#single-source-of-truth)). From c91efad7441ab4312a1421d20817d0ea27b87df3 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 03:43:48 -0400 Subject: [PATCH 44/58] docs(design): pin canonical dependencies in a new owner doc design/dependencies.md is the single owner of non-crypto library pins per platform (jiff over chrono, thiserror/eyre, tracing, rustls, UUIDv7-default, and the standing server/web/FFI choices), registered in the principles owner table and summarized in AGENTS.md. The log->tracing migration the logging pin requires is indexed as slice S-F6. --- AGENTS.md | 9 +++ SLICES.md | 14 +++++ capsule-docs/astro.config.mjs | 1 + .../src/content/docs/design/dependencies.md | 63 +++++++++++++++++++ .../src/content/docs/design/principles.md | 1 + 5 files changed, 88 insertions(+) create mode 100644 capsule-docs/src/content/docs/design/dependencies.md diff --git a/AGENTS.md b/AGENTS.md index 04b67713..82930f0b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,6 +10,15 @@ - Mocking: Use mocks for all external dependencies and critical internal processes. This allows us to have deterministic tests and easily simulate edge cases and failure scenarios that are hard to reproduce with real dependencies. Do not try to wire up two incomplete complex systems to mock each other. - Linting and formatting is setup to be strict. Pre-commit and pre-push hooks are configured so you won't be able to push code that doesn't meet the standards. +## Dependencies + +- Datetime: `jiff`, never `chrono` — chrono exists only as the sea-orm column type inside `capsule-api/entity` and `capsule-cli/entity` (convert at the entity boundary) and in the frozen `capsule-api-library`. +- Errors: `thiserror` in libraries, `eyre`/`color-eyre` in binaries; no `anyhow`. +- Logging: `tracing`, never the `log` facade in new code. +- TLS: `rustls` only; never native-tls/openssl. +- Identifiers: UUIDv7 for new ids; UUIDv4 only where creation time must not leak. +- The canonical table (all platforms, exceptions, rationale) is the [Dependencies design doc](capsule-docs/src/content/docs/design/dependencies.md) — add a row there before introducing a dependency for a new domain. + ## Internationalization - No hardcoded user-facing strings. Every translatable string is a key in the canonical catalogs under `locales/` (ICU MessageFormat). Add the key there, not inline in app code. diff --git a/SLICES.md b/SLICES.md index 87717932..2c3c9da5 100644 --- a/SLICES.md +++ b/SLICES.md @@ -109,6 +109,7 @@ its slice. | S-F3 | Xcode/Gradle binding wiring + on-device CI | platform/FFI | S-F2 | L | ready | | S-F4 | Windows TPM (TBS) backend | platform/FFI | S-A4 | M | ready | | S-F5 | Hardware DEK binding | platform/FFI | S-F2 | M | ready | +| S-F6 | `log` → `tracing` migration (core + core-ffi) | platform/FFI | — | S | ready | | S-G1 | GraphQL retirement | legacy-retire | S-C2, S-D6 | M | blocked | | S-G2 | Legacy plaintext proto/service removal | legacy-retire | S-C2, S-D2 | S | blocked | | S-G3 | Plaintext entity retirement (face/person/smart_tag) | legacy-retire | S-G1, S-H3 | M | blocked | @@ -809,6 +810,19 @@ SDK; they never hand-roll network flows. - **Deliverable:** the device **encryption** key's classical half hardware-bound (P-256 ECDH), mirroring the DSK composition. **Depends on:** S-F2. **Tier:** Smoke. +### S-F6 — `log` → `tracing` migration + +- **Contract:** [Dependencies — Rust](capsule-docs/src/content/docs/design/dependencies.md) + (logging row: `tracing` is the sole facade). +- **Deliverable:** every `log::` call site in `capsule-core` and `capsule-core-ffi` + replaced with `tracing` (structured fields; `#[instrument]` spans on the hot paths + the traceability rule names), the platform bridges (oslog) re-wired through a + tracing subscriber, and the `log` workspace dependency dropped once nothing consumes + it. +- **Done when:** `rg 'log::'` finds no non-frozen hits in the two crates; the existing + unit suites pass unchanged; `mise run check-rust` green. +- **Tier:** Unit (existing suites). + ## Lane G — legacy retirement (frozen until preconditions) Every `LEGACY-PLAINTEXT (frozen)` marker in the tree names its slice here. Frozen code diff --git a/capsule-docs/astro.config.mjs b/capsule-docs/astro.config.mjs index 61310694..2fb1f45c 100644 --- a/capsule-docs/astro.config.mjs +++ b/capsule-docs/astro.config.mjs @@ -58,6 +58,7 @@ export default defineConfig({ { slug: 'design/module-map' }, { slug: 'design/api-surfaces' }, { slug: 'design/networking' }, + { slug: 'design/dependencies' }, ], }, { diff --git a/capsule-docs/src/content/docs/design/dependencies.md b/capsule-docs/src/content/docs/design/dependencies.md new file mode 100644 index 00000000..6316511a --- /dev/null +++ b/capsule-docs/src/content/docs/design/dependencies.md @@ -0,0 +1,63 @@ +--- +title: Dependencies +description: Canonical library and tooling pins for every Capsule platform +status: draft +--- + +This doc is the single owner of Capsule's **canonical implementation-dependency pins**: for each domain concern, the one library that implements it, where the pin applies, and the bounded exceptions. Introducing a dependency for a domain not listed here requires adding a row here first; changing a pin is a one-doc edit plus a migration slice in the repo-root `SLICES.md` — never an in-place drift. + +What this doc deliberately does **not** own, per the [SSoT rule](/design/principles/#single-source-of-truth): + +- **Cryptographic primitives and their crates** — [Cryptography — Primitives](/design/cryptography/primitives/). +- **TLS version policy** — [Failure Modes — Transport Security](/design/cryptography/failure-modes/#transport-security). This doc pins the *implementation* (rustls); the policy lives there. +- **Client test and performance tooling** — [Clients — Test and Performance Tooling](/design/clients/#test-and-performance-tooling). +- **Which existing identifier uses which UUID version** — [Metadata — Identifiers](/design/metadata/#identifiers). This doc owns only the default-for-new rule. +- **Media formats and codecs** — [Thumbnails and Previews](/design/thumbnails/) owns derivative formats; the sidecar's `content_type` set is owned by [Metadata](/design/metadata/#closed-enum-value-sets). + +Mechanically, every Rust version is pinned once in the root `Cargo.toml` `[workspace.dependencies]`; member crates consume it with `workspace = true` and never declare their own version. Toolchain versions (Rust nightly, bun, tuist, …) are pinned by mise per `CONTRIBUTING.md`. + +## Rust + +| Domain | Canonical choice | Scope | Exceptions | +| --- | --- | --- | --- | +| Datetime | `jiff` | All domain logic — parsing, formatting, arithmetic. Signed and wire formats carry RFC 3339 strings or integer epochs, never a datetime library type, so the pin never touches serialized bytes. | `chrono` remains **only** as the sea-orm column type in the entity crates (`capsule-api/entity`, `capsule-cli/entity`), converted to jiff at the entity boundary, and in the frozen `capsule-api-library` (async-graphql `chrono` scalars), which retires with slice S-G1. | +| Error handling | `thiserror` in libraries; `eyre` + `color-eyre` in binaries | Libraries define typed error enums; binaries (CLI, server `main`, xtask) wrap them in reports. | `anyhow` is not used. | +| Logging | `tracing` (facade) + `tracing-subscriber` (binaries) | All crates; structured fields and hot-path spans per the traceability rule in `AGENTS.md`. The `log` facade is forbidden in new code. | Remaining `log::` call sites in `capsule-core` / `capsule-core-ffi` migrate in slice S-F6. | +| TLS implementation | `rustls` | Wherever Capsule code holds a TLS stack: the SDK's HTTP client, [LAN-peering](/design/peering/) mutual TLS, server egress, sea-orm's `runtime-tokio-rustls`. Never native-tls/openssl. | `openssl` appears only as a transitive dependency of `webauthn-rs` attestation-certificate verification — it is never a TLS stack. | +| Identifiers | `uuid` — **UUIDv7 for every newly introduced identifier** | Time-ordered v7 is the default (index locality); the assignment of existing ids is owned by [Metadata — Identifiers](/design/metadata/#identifiers). | UUIDv4 where an id must not leak creation time (e.g. `device_id`). Capability-bearing opaque ids (share links, drops) are not UUIDs at all — they carry their own ≥128-bit entropy per their owner docs. | +| Async runtime | `tokio` | All async code. | — | +| HTTP server | `salvo` | `capsule-api` REST surfaces. | — | +| gRPC | `tonic` + `prost` | The `capsule.sync.v1` feed per [API Surfaces](/design/api-surfaces/). | — | +| HTTP client | `reqwest` (`default-features = false`, `rustls-tls`) | `capsule-sdk` — the sanctioned network path. | — | +| ORM | `sea-orm` (`sqlx-postgres` on the server, `sqlx-sqlite` in the CLI) | The rebuildable index databases only — sidecars stay canonical per [Principles](/design/principles/). | — | +| Embedded SQLite | `rusqlite` (`bundled`) | `capsule-core`'s `library.sqlite`. | — | +| CBOR | `ciborium` (+ `serde_bytes`, `half`) | All CBOR; the canonical-encoding rules are owned by [Metadata — Canonical CBOR Encoding](/design/metadata/#canonical-cbor-encoding). | — | +| Serialization | `serde` + `serde_json` | Derives and JSON surfaces. | — | +| CLI | `clap` (derive) | `capsule-cli`, xtask argument parsing where non-trivial. | — | +| Test runner | `cargo-nextest`; `testcontainers` (+ podman) for real backing services | The Unit/Smoke tiers per [Principles — Validation Tiers](/design/principles/#validation-tiers). | — | +| FFI bindings | `uniffi` | One workspace version — consolidation is slice S-F1. | — | + +## Web + +| Domain | Canonical choice | Notes | +| --- | --- | --- | +| Framework | React 19 + rsbuild | — | +| Routing / data | TanStack Router / TanStack Query | — | +| Styling | Tailwind v4, class-based dark mode (`@custom-variant dark` + `ThemeProvider`) | — | +| Validation | zod | — | +| Datetime | **none** — native `Intl` / `Date` | A date library is added only by adding a row here. | +| i18n runtime | FormatJS over the generated catalogs | Contract owned by [Internationalization](/design/i18n/). | +| Lint/format | Biome | — | +| Runtime / package manager | bun | — | + +## Swift + +Test frameworks and performance tooling are owned by [Clients — Test and Performance Tooling](/design/clients/#test-and-performance-tooling). Bindings come from the single-version uniffi strategy (slice S-F1). Project generation and formatting (Tuist, SwiftLint, SwiftFormat) are mise-pinned toolchain, not library pins. + +## Kotlin + +Bindings likewise ride the uniffi strategy (S-F1); JNA loads the produced library. JUnit 5 is the current test harness; the canonical pin is recorded in [Clients — Test and Performance Tooling](/design/clients/#test-and-performance-tooling) when the Android harness stabilizes. + +## Validation + +The pins are enforced structurally, not by convention: `cargo tree -i chrono -e no-dev` must resolve to the entity crates, the frozen library crate, and sea-orm internals only; `rg 'log::'` outside the S-F6 scope, and any `openssl`/`native-tls` edge outside webauthn-rs, are review-blocking. The per-platform `mise run check-*` gates run the pinned toolchains. diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 33278242..8d489d00 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -69,6 +69,7 @@ The owner docs are: | Code module → design doc mapping + bounded E2E test surface | [Module Map](/design/module-map/) | | API surface ↔ transport map + cross-transport negotiation/rejection mapping | [API Surfaces](/design/api-surfaces/) | | Storage durability verdict + verify-before-destroy rule | [Import — Storage Verification](/design/import/storage-verification/) | +| Canonical dependency + tooling pins (non-crypto libraries per platform) | [Dependencies](/design/dependencies/) | **Permitted secondary mentions.** Mechanism-explanatory phrasing inside a non-owner doc is fine — for example, "STREAM tags catch chunk reordering" inside [Peering](/design/peering/) is explaining a *behavior*, not declaring a *choice*. What the rule forbids is restating the choice itself ("we use SHA-256") outside the owner doc. When in doubt, link. From e4b6891f25f934d5d18d101a151d25cfca40a70f Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 03:44:21 -0400 Subject: [PATCH 45/58] docs(design): set the TLS 1.3-preferred, 1.2-floor transport policy Failure Modes stays the transport-security owner: 1.3 required wherever both endpoints support it, 1.2 admitted only for legacy ingress compatibility, nothing below ever negotiated. The rustls implementation pin for the hops Capsule code terminates itself lives in the Dependencies owner doc. Resolves the design half of issue #5. --- .../src/content/docs/design/cryptography/failure-modes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/capsule-docs/src/content/docs/design/cryptography/failure-modes.md b/capsule-docs/src/content/docs/design/cryptography/failure-modes.md index 927ea6f2..114bc581 100644 --- a/capsule-docs/src/content/docs/design/cryptography/failure-modes.md +++ b/capsule-docs/src/content/docs/design/cryptography/failure-modes.md @@ -67,7 +67,9 @@ These cross-cutting properties make recovery robust specifically against *catast All client-server communication is over HTTPS. While Capsule's stack aims to stay PQ-safe (within due course), the transport layer (TLS) must be configured by the server administrator to be PQ-resistant as well. As of 2026, the standard is **TLS 1.3 with hybrid X25519+ML-KEM key exchange** enabled. Since application servers do not terminate TLS, ensure the ingress/reverse proxy is properly configured. -This is the one piece of cryptographic configuration that lives outside the application layer — the application code cannot enforce it, only document the requirement. +The version policy everywhere: **TLS 1.3 is required wherever both endpoints support it; TLS 1.2 is the absolute floor**, admitted only for compatibility with legacy ingress/reverse-proxy deployments — nothing below 1.2 is ever negotiated, and the PQ-hybrid key-exchange recommendation applies to 1.3. Every Capsule client attempts 1.3 first; deployment documentation shows administrators how to verify their edge actually negotiates 1.3. + +The ingress hop is the one piece of cryptographic configuration that lives outside the application layer — the application code cannot enforce it, only document the requirement. Where Capsule code itself terminates or originates TLS — the SDK's HTTP client, [LAN peering](/design/peering/)'s mutual TLS, server-to-server egress — the implementation is the rustls pin in [Dependencies](/design/dependencies/#rust), and the same version policy applies in code. ## Validation From d18c343fc107dc66ab63c42a4c60f9ee0fa7e791 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 03:52:12 -0400 Subject: [PATCH 46/58] refactor(core): migrate domain datetime handling from chrono to jiff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All capsule-core datetime work now goes through jiff: RFC 3339 production/parsing (jiff::Timestamp), EXIF local wall-clock parsing (jiff::civil::DateTime, the offset-less analogue of NaiveDateTime), fixed UTC offsets (jiff::tz::Offset), and date bucketing via Zoned UTC dates. Serialized bytes are untouched: every signed format carries RFC 3339 strings or i64 epochs, never a datetime-library type. The soft-delete retention window now adds retain_days * 24 h of absolute time — identical semantics in UTC, stated explicitly at the call site. Pin: design/dependencies.md (datetime row). --- Cargo.lock | 128 ++++++++++++++++--- Cargo.toml | 1 + capsule-core/Cargo.toml | 2 +- capsule-core/src/crypto/verify_asset.rs | 4 +- capsule-core/src/exif/extract.rs | 6 +- capsule-core/src/exif/timezone.rs | 29 +++-- capsule-core/src/library/paths.rs | 11 +- capsule-core/src/lifecycle.rs | 27 ++-- capsule-core/src/media/image/formats/avif.rs | 2 +- capsule-core/src/media/image/formats/bmp.rs | 2 +- capsule-core/src/media/image/formats/dng.rs | 2 +- capsule-core/src/media/image/formats/gif.rs | 2 +- capsule-core/src/media/image/formats/heif.rs | 2 +- capsule-core/src/media/image/formats/jpeg.rs | 2 +- capsule-core/src/media/image/formats/jxl.rs | 2 +- capsule-core/src/media/image/formats/png.rs | 2 +- capsule-core/src/media/image/formats/raw.rs | 2 +- capsule-core/src/media/image/formats/tiff.rs | 2 +- capsule-core/src/media/image/formats/webp.rs | 2 +- capsule-core/src/media/image/metadata/mod.rs | 6 +- capsule-core/src/metadata/crdt/lww.rs | 4 +- capsule-core/src/metadata/file.rs | 18 ++- capsule-core/src/validation/structural.rs | 11 +- 23 files changed, 189 insertions(+), 80 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 2d007a76..fe1bf845 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -758,6 +758,12 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + [[package]] name = "bitflags" version = "2.13.0" @@ -827,7 +833,7 @@ checksum = "87a52479c9237eb04047ddb94788c41ca0d26eaff8b697ecfbb4c32f7fdc3b1b" dependencies = [ "async-stream", "base64 0.22.1", - "bitflags", + "bitflags 2.13.0", "bollard-buildkit-proto", "bollard-stubs", "bytes", @@ -1338,7 +1344,6 @@ dependencies = [ "aes-gcm", "argon2", "base64 0.22.1", - "chrono", "ciborium", "ed25519-dalek", "file-format", @@ -1349,6 +1354,7 @@ dependencies = [ "hkdf", "hmac", "indexmap 2.14.0", + "jiff", "jpeg-encoder", "kamadak-exif", "log", @@ -1982,6 +1988,37 @@ version = "2.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror 2.0.18", +] + [[package]] name = "der" version = "0.7.10" @@ -3355,6 +3392,48 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jiff" +version = "0.2.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "961d16382652bfdd8c6f68b223b26a8c93e0d475c672f414411db31c6c5c900e" +dependencies = [ + "defmt", + "jiff-static", + "jiff-tzdb-platform", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "windows-link 0.2.1", +] + +[[package]] +name = "jiff-static" +version = "0.2.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0879bd39df99c4c5e2c6615ccc026391a423dde10532c573e6086eb94a802cc" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + [[package]] name = "jobserver" version = "0.1.34" @@ -3478,7 +3557,7 @@ version = "0.1.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f02ab6bace2054fb888a3c16f990117b579d14a3088e472d63c6011fa185c9d3" dependencies = [ - "bitflags", + "bitflags 2.13.0", "libc", "plain", "redox_syscall 0.8.1", @@ -3770,7 +3849,7 @@ version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "22f9786d56d972959e1408b6a93be6af13b9c1392036c5c1fafa08a1b0c6ee87" dependencies = [ - "bitflags", + "bitflags 2.13.0", "byteorder", "derive_builder", "getset", @@ -3799,7 +3878,7 @@ version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" dependencies = [ - "bitflags", + "bitflags 2.13.0", "cfg-if", "cfg_aliases", "libc", @@ -3812,7 +3891,7 @@ version = "0.30.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6" dependencies = [ - "bitflags", + "bitflags 2.13.0", "cfg-if", "cfg_aliases", "libc", @@ -3975,7 +4054,7 @@ version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536" dependencies = [ - "bitflags", + "bitflags 2.13.0", ] [[package]] @@ -4039,7 +4118,7 @@ version = "0.10.80" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a45fa2aa886c42762255da344f0a0d313e254066c46aad76f300c3d3da62d967" dependencies = [ - "bitflags", + "bitflags 2.13.0", "cfg-if", "foreign-types", "libc", @@ -4485,6 +4564,15 @@ version = "1.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + [[package]] name = "potential_utf" version = "0.1.5" @@ -4712,7 +4800,7 @@ version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e9f068eba8e7071c5f9511831b44f32c740d5adf574e990f946ddb53db2f314e" dependencies = [ - "bitflags", + "bitflags 2.13.0", "memchr", "unicase", ] @@ -4907,7 +4995,7 @@ version = "0.5.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" dependencies = [ - "bitflags", + "bitflags 2.13.0", ] [[package]] @@ -4916,7 +5004,7 @@ version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5b44b894f2a6e36457d665d1e08c3866add6ed5e70050c1b4ba8a8ddedb02ce7" dependencies = [ - "bitflags", + "bitflags 2.13.0", ] [[package]] @@ -5112,7 +5200,7 @@ version = "0.32.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7753b721174eb8ff87a9a0e799e2d7bc3749323e773db92e0984debb00019d6e" dependencies = [ - "bitflags", + "bitflags 2.13.0", "fallible-iterator", "fallible-streaming-iterator", "hashlink 0.9.1", @@ -5207,7 +5295,7 @@ version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" dependencies = [ - "bitflags", + "bitflags 2.13.0", "errno", "libc", "linux-raw-sys", @@ -5767,7 +5855,7 @@ version = "3.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" dependencies = [ - "bitflags", + "bitflags 2.13.0", "core-foundation 0.10.1", "core-foundation-sys", "libc", @@ -6270,7 +6358,7 @@ dependencies = [ "atoi", "base64 0.22.1", "bigdecimal", - "bitflags", + "bitflags 2.13.0", "byteorder", "bytes", "chrono", @@ -6317,7 +6405,7 @@ dependencies = [ "atoi", "base64 0.22.1", "bigdecimal", - "bitflags", + "bitflags 2.13.0", "byteorder", "chrono", "crc", @@ -6537,7 +6625,7 @@ version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b" dependencies = [ - "bitflags", + "bitflags 2.13.0", "core-foundation 0.9.4", "system-configuration-sys", ] @@ -7067,7 +7155,7 @@ version = "0.6.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" dependencies = [ - "bitflags", + "bitflags 2.13.0", "bytes", "futures-util", "http", @@ -7845,7 +7933,7 @@ version = "0.244.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" dependencies = [ - "bitflags", + "bitflags 2.13.0", "hashbrown 0.15.5", "indexmap 2.14.0", "semver", @@ -8479,7 +8567,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" dependencies = [ "anyhow", - "bitflags", + "bitflags 2.13.0", "indexmap 2.14.0", "log", "serde", diff --git a/Cargo.toml b/Cargo.toml index 1e116371..18869d49 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -57,6 +57,7 @@ derive_more = "2.1.1" eyre = "0.6.12" futures-util = "0.3.32" indexmap = { version = "2.14.0", features = ["serde"] } +jiff = { version = "0.2", features = ["serde"] } jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] } log = { version = "0.4.32", features = ["release_max_level_debug"] } file-format = "0.28" diff --git a/capsule-core/Cargo.toml b/capsule-core/Cargo.toml index d7ec67a4..6b450dab 100644 --- a/capsule-core/Cargo.toml +++ b/capsule-core/Cargo.toml @@ -45,11 +45,11 @@ media = [ tpm = ["dep:tss-esapi"] [dependencies] -chrono = { workspace = true } ciborium = "0.2" half = "2" # IEEE-754 f16 for RFC 8949 §4.2 shortest-float canonical CBOR globset = "0.4.18" indexmap = { workspace = true } +jiff = { workspace = true } kamadak-exif = "0.5" log = { workspace = true } rusqlite = { version = "0.32", features = ["bundled"] } diff --git a/capsule-core/src/crypto/verify_asset.rs b/capsule-core/src/crypto/verify_asset.rs index 6635a91c..8debf2ae 100644 --- a/capsule-core/src/crypto/verify_asset.rs +++ b/capsule-core/src/crypto/verify_asset.rs @@ -80,8 +80,8 @@ impl VerifyOutcome { } fn rfc3339_le(a: &str, b: &str) -> Option { - let pa = chrono::DateTime::parse_from_rfc3339(a).ok()?; - let pb = chrono::DateTime::parse_from_rfc3339(b).ok()?; + let pa = a.parse::().ok()?; + let pb = b.parse::().ok()?; Some(pa <= pb) } diff --git a/capsule-core/src/exif/extract.rs b/capsule-core/src/exif/extract.rs index 801ece19..5a577d36 100644 --- a/capsule-core/src/exif/extract.rs +++ b/capsule-core/src/exif/extract.rs @@ -2,12 +2,12 @@ use std::fs::File; use std::io::BufReader; use std::path::Path; -use chrono::NaiveDateTime; use exif::{In, Reader, Tag, Value}; +use jiff::civil; #[derive(Debug, Clone, PartialEq, Default)] pub struct ExifExtract { - pub date_time_original: Option, + pub date_time_original: Option, pub offset_time_original: Option, // e.g. "+09:00" pub gps_lat: Option, pub gps_lon: Option, @@ -44,7 +44,7 @@ pub fn extract_exif(path: &Path) -> Result TimezoneResolution { let capture_timestamp = extract .date_time_original - .map(|dt| dt.and_utc().timestamp()); + .and_then(|dt| dt.to_zoned(TimeZone::UTC).ok()) + .map(|z| z.timestamp().as_second()); // Case 1: OffsetTimeOriginal present if let (Some(dt), Some(offset_str)) = (extract.date_time_original, &extract.offset_time_original) && let Some(offset) = parse_offset(offset_str) { - let local = offset.from_local_datetime(&dt).single(); - let capture_utc = local.map(|t| t.timestamp()); + // A fixed offset is never ambiguous, so this either resolves or the datetime + // itself is out of range. + let local = dt.to_zoned(TimeZone::fixed(offset)).ok(); + let capture_utc = local.map(|t| t.timestamp().as_second()); return TimezoneResolution { capture_timestamp, capture_utc, @@ -39,7 +42,7 @@ pub fn resolve_timezone(extract: &ExifExtract) -> TimezoneResolution { { return TimezoneResolution { capture_timestamp, - capture_utc: None, // Would need chrono-tz to apply IANA tz; leave as None + capture_utc: None, // Applying the IANA tz is deferred; leave as None capture_tz: Some(tz_name), capture_tz_source: Some(CaptureTzSource::GpsLookup), tz_db_version: Some(tz_db_ver), @@ -56,7 +59,7 @@ pub fn resolve_timezone(extract: &ExifExtract) -> TimezoneResolution { } } -fn parse_offset(s: &str) -> Option { +fn parse_offset(s: &str) -> Option { // Parse "+HH:MM" or "-HH:MM" let s = s.trim(); if s.len() < 6 { @@ -70,7 +73,7 @@ fn parse_offset(s: &str) -> Option { let hours: i32 = parts[0].parse().ok()?; let minutes: i32 = parts[1].parse().ok()?; let total_secs = sign * (hours * 3600 + minutes * 60); - FixedOffset::east_opt(total_secs) + Offset::from_seconds(total_secs).ok() } fn lookup_timezone(lat: f64, lon: f64) -> Option<(String, String)> { @@ -86,7 +89,7 @@ fn lookup_timezone(lat: f64, lon: f64) -> Option<(String, String)> { #[cfg(test)] mod tests { - use chrono::NaiveDateTime; + use jiff::civil; use super::*; use crate::domain::CaptureTzSource; @@ -94,7 +97,7 @@ mod tests { fn extract_with_offset(dt: &str, offset: &str) -> ExifExtract { ExifExtract { - date_time_original: NaiveDateTime::parse_from_str(dt, "%Y:%m:%d %H:%M:%S").ok(), + date_time_original: civil::DateTime::strptime("%Y:%m:%d %H:%M:%S", dt).ok(), offset_time_original: Some(offset.to_string()), gps_lat: None, gps_lon: None, @@ -109,7 +112,7 @@ mod tests { fn extract_with_gps(dt: &str, lat: f64, lon: f64) -> ExifExtract { ExifExtract { - date_time_original: NaiveDateTime::parse_from_str(dt, "%Y:%m:%d %H:%M:%S").ok(), + date_time_original: civil::DateTime::strptime("%Y:%m:%d %H:%M:%S", dt).ok(), offset_time_original: None, gps_lat: Some(lat), gps_lon: Some(lon), @@ -124,7 +127,7 @@ mod tests { fn extract_floating(dt: &str) -> ExifExtract { ExifExtract { - date_time_original: NaiveDateTime::parse_from_str(dt, "%Y:%m:%d %H:%M:%S").ok(), + date_time_original: civil::DateTime::strptime("%Y:%m:%d %H:%M:%S", dt).ok(), offset_time_original: None, gps_lat: None, gps_lon: None, @@ -214,13 +217,13 @@ mod tests { #[test] fn test_parse_offset_positive() { let offset = parse_offset("+09:00").unwrap(); - assert_eq!(offset.utc_minus_local(), -9 * 3600); + assert_eq!(offset.seconds(), 9 * 3600); } #[test] fn test_parse_offset_negative() { let offset = parse_offset("-05:30").unwrap(); - assert_eq!(offset.utc_minus_local(), 5 * 3600 + 30 * 60); + assert_eq!(offset.seconds(), -(5 * 3600 + 30 * 60)); } #[test] diff --git a/capsule-core/src/library/paths.rs b/capsule-core/src/library/paths.rs index 3c701d75..caba7d7b 100644 --- a/capsule-core/src/library/paths.rs +++ b/capsule-core/src/library/paths.rs @@ -1,6 +1,5 @@ use std::path::{Path, PathBuf}; -use chrono::{DateTime, Datelike}; use uuid::Uuid; #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -39,8 +38,14 @@ fn shard_dirs(uuid: &Uuid) -> (String, String) { fn date_parts(capture_utc: Option) -> (u32, u8) { match capture_utc { Some(ts) => { - let dt = DateTime::from_timestamp(ts, 0).unwrap_or(DateTime::UNIX_EPOCH); - (dt.year() as u32, dt.month() as u8) + let date = jiff::Timestamp::from_second(ts) + .unwrap_or(jiff::Timestamp::UNIX_EPOCH) + .to_zoned(jiff::tz::TimeZone::UTC) + .date(); + ( + u32::try_from(date.year()).unwrap_or(1970), + u8::try_from(date.month()).unwrap_or(1), + ) } None => (1970, 1), } diff --git a/capsule-core/src/lifecycle.rs b/capsule-core/src/lifecycle.rs index 61de9585..7e73f1c4 100644 --- a/capsule-core/src/lifecycle.rs +++ b/capsule-core/src/lifecycle.rs @@ -27,7 +27,7 @@ use std::collections::{BTreeMap, HashMap}; use std::fs; use std::path::{Path, PathBuf}; -use chrono::{Datelike, Utc}; +use jiff::Timestamp; use thiserror::Error; use uuid::Uuid; @@ -135,7 +135,7 @@ pub struct Workspace { } fn now_rfc3339() -> String { - Utc::now().to_rfc3339() + Timestamp::now().to_string() } fn content_type_for(ext: &str) -> String { @@ -151,10 +151,13 @@ fn content_type_for(ext: &str) -> String { } fn media_dir(root: &Path, capture_utc: i64) -> PathBuf { - let dt = chrono::DateTime::from_timestamp(capture_utc, 0).unwrap_or_default(); + let date = Timestamp::from_second(capture_utc) + .unwrap_or(Timestamp::UNIX_EPOCH) + .to_zoned(jiff::tz::TimeZone::UTC) + .date(); root.join("media") - .join(format!("{:04}", dt.year())) - .join(format!("{:04}-{:02}", dt.year(), dt.month())) + .join(format!("{:04}", date.year())) + .join(format!("{:04}-{:02}", date.year(), date.month())) } fn asset_type_for(content_type: &str) -> String { @@ -167,7 +170,7 @@ fn asset_type_for(content_type: &str) -> String { } fn rfc3339_to_secs(s: &str) -> i64 { - chrono::DateTime::parse_from_rfc3339(s).map_or(0, |d| d.timestamp()) + s.parse::().map_or(0, Timestamp::as_second) } /// Map a managed asset's in-memory state to its queryable `assets` index row. Deletion state is @@ -461,7 +464,7 @@ impl Workspace { format: Some(asset.ext.clone()), bytes: bytes as i64, path: self.media_path(asset).to_string_lossy().into_owned(), - last_accessed_at: Utc::now().timestamp(), + last_accessed_at: Timestamp::now().as_second(), pinned: false, is_owned_original: true, }) @@ -477,7 +480,7 @@ impl Workspace { .extension() .map_or_else(|| "bin".into(), |e| e.to_string_lossy().to_lowercase()); let asset_id = Uuid::now_v7(); - let capture_utc = Utc::now().timestamp(); + let capture_utc = Timestamp::now().as_second(); let album = self.album(&album_id)?; let file_key = self.file_key(album, album.current_epoch, &asset_id); @@ -677,7 +680,11 @@ impl Workspace { /// Soft-delete: emit a `delete` record carrying a signed retention window. pub fn soft_delete(&mut self, asset_id: &Uuid, retain_days: i64) -> Result<()> { - let until = (Utc::now() + chrono::Duration::days(retain_days)).to_rfc3339(); + // Timestamp arithmetic is absolute, so a retention "day" is exactly 24 h — the + // correct semantic for a UTC retention window. + let until = (Timestamp::now() + + jiff::SignedDuration::from_hours(retain_days.saturating_mul(24))) + .to_string(); self.append_lifecycle(asset_id, Action::Delete, Some(until), |_, _| {}) } @@ -770,7 +777,7 @@ impl Workspace { .last() .expect("restored provenance is never empty") .manifest; - let capture_utc = Utc::now().timestamp(); + let capture_utc = Timestamp::now().as_second(); let mut chain = ProvenanceChain::new(); for rec in &restored.provenance { chain diff --git a/capsule-core/src/media/image/formats/avif.rs b/capsule-core/src/media/image/formats/avif.rs index f9c05745..49637c7a 100644 --- a/capsule-core/src/media/image/formats/avif.rs +++ b/capsule-core/src/media/image/formats/avif.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct AvifImage {} impl ImageMetadataExtractor for AvifImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/bmp.rs b/capsule-core/src/media/image/formats/bmp.rs index b99b27f1..f0c98318 100644 --- a/capsule-core/src/media/image/formats/bmp.rs +++ b/capsule-core/src/media/image/formats/bmp.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct BmpImage {} impl ImageMetadataExtractor for BmpImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/dng.rs b/capsule-core/src/media/image/formats/dng.rs index 80ac7f8f..a544caf3 100644 --- a/capsule-core/src/media/image/formats/dng.rs +++ b/capsule-core/src/media/image/formats/dng.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct DngImage {} impl ImageMetadataExtractor for DngImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/gif.rs b/capsule-core/src/media/image/formats/gif.rs index a42cd657..cc465c8e 100644 --- a/capsule-core/src/media/image/formats/gif.rs +++ b/capsule-core/src/media/image/formats/gif.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct GifImage {} impl ImageMetadataExtractor for GifImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/heif.rs b/capsule-core/src/media/image/formats/heif.rs index b22393ae..b3d81533 100644 --- a/capsule-core/src/media/image/formats/heif.rs +++ b/capsule-core/src/media/image/formats/heif.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct HeifImage {} impl ImageMetadataExtractor for HeifImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/jpeg.rs b/capsule-core/src/media/image/formats/jpeg.rs index 50c090f8..396f5ebe 100644 --- a/capsule-core/src/media/image/formats/jpeg.rs +++ b/capsule-core/src/media/image/formats/jpeg.rs @@ -31,7 +31,7 @@ pub struct JpegImage { } impl ImageMetadataExtractor for JpegImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { None } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/jxl.rs b/capsule-core/src/media/image/formats/jxl.rs index f734b7d8..ab740645 100644 --- a/capsule-core/src/media/image/formats/jxl.rs +++ b/capsule-core/src/media/image/formats/jxl.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct JxlImage {} impl ImageMetadataExtractor for JxlImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/png.rs b/capsule-core/src/media/image/formats/png.rs index 46cad4da..3933079c 100644 --- a/capsule-core/src/media/image/formats/png.rs +++ b/capsule-core/src/media/image/formats/png.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct PngImage {} impl ImageMetadataExtractor for PngImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/raw.rs b/capsule-core/src/media/image/formats/raw.rs index ba47f1bc..3a104614 100644 --- a/capsule-core/src/media/image/formats/raw.rs +++ b/capsule-core/src/media/image/formats/raw.rs @@ -44,7 +44,7 @@ impl RawImage { } impl ImageMetadataExtractor for RawImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/tiff.rs b/capsule-core/src/media/image/formats/tiff.rs index 364fb0ae..f15f1c4a 100644 --- a/capsule-core/src/media/image/formats/tiff.rs +++ b/capsule-core/src/media/image/formats/tiff.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct TiffImage {} impl ImageMetadataExtractor for TiffImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/formats/webp.rs b/capsule-core/src/media/image/formats/webp.rs index c853da9c..62bc7afc 100644 --- a/capsule-core/src/media/image/formats/webp.rs +++ b/capsule-core/src/media/image/formats/webp.rs @@ -19,7 +19,7 @@ use crate::media::metadata::{ColorSpace, DeviceMetadata}; pub struct WebpImage {} impl ImageMetadataExtractor for WebpImage { - fn get_date_taken(&self) -> Option> { + fn get_date_taken(&self) -> Option { unimplemented!() } fn get_dimensions(&self) -> (u32, u32) { diff --git a/capsule-core/src/media/image/metadata/mod.rs b/capsule-core/src/media/image/metadata/mod.rs index f1fbfc79..0e210a4c 100644 --- a/capsule-core/src/media/image/metadata/mod.rs +++ b/capsule-core/src/media/image/metadata/mod.rs @@ -1,4 +1,4 @@ -use chrono::{DateTime, Utc}; +use jiff::civil; use serde::{Deserialize, Serialize}; use crate::media::core::types::ImageFormat; @@ -28,7 +28,7 @@ pub trait ImageMetadataExtractor { fn get_file_size(&self) -> u64; // Generic typed metadata - fn get_date_taken(&self) -> Option>; + fn get_date_taken(&self) -> Option; fn get_device_metadata(&self) -> Option; fn get_capture_settings(&self) -> Option; fn get_location(&self) -> Option; @@ -93,7 +93,7 @@ pub struct ImageMetadata { // --- Extracted Typed Metadata (From your original struct) --- #[serde(skip_serializing_if = "Option::is_none")] - pub date_taken: Option>, + pub date_taken: Option, #[serde(skip_serializing_if = "Option::is_none")] pub device: Option, #[serde(skip_serializing_if = "Option::is_none")] diff --git a/capsule-core/src/metadata/crdt/lww.rs b/capsule-core/src/metadata/crdt/lww.rs index 0b0b6aa2..a39ad114 100644 --- a/capsule-core/src/metadata/crdt/lww.rs +++ b/capsule-core/src/metadata/crdt/lww.rs @@ -38,8 +38,8 @@ pub struct Lww { /// Order two candidates: later timestamp wins; ties break on the larger device id. /// `None` if a timestamp is unparseable (caller treats as a structural reject upstream). fn beats(a: &Stamped, b: &Stamped) -> Option { - let ta = chrono::DateTime::parse_from_rfc3339(&a.ts).ok()?; - let tb = chrono::DateTime::parse_from_rfc3339(&b.ts).ok()?; + let ta = a.ts.parse::().ok()?; + let tb = b.ts.parse::().ok()?; Some((ta, a.by) > (tb, b.by)) } diff --git a/capsule-core/src/metadata/file.rs b/capsule-core/src/metadata/file.rs index b0a732f2..8d7b882f 100644 --- a/capsule-core/src/metadata/file.rs +++ b/capsule-core/src/metadata/file.rs @@ -2,7 +2,7 @@ use std::ops::Deref; use std::path::Path; use std::{fs, io}; -use chrono::{DateTime, Utc}; +use jiff::Timestamp; use serde::{Deserialize, Serialize}; use crate::utils::hash::get_file_hash; @@ -41,12 +41,12 @@ pub struct FileMetadata { /// Original file name pub original_filename: String, /// Creation timestamp - pub created_timestamp: DateTime, + pub created_timestamp: Timestamp, /// Last modified timestamp - pub modified_timestamp: DateTime, + pub modified_timestamp: Timestamp, /// Import timestamp - pub import_timestamp: DateTime, + pub import_timestamp: Timestamp, // pub size: u64, // pub modified: u64, } @@ -72,13 +72,17 @@ impl FileMetadata { // Get timestamps let created_timestamp = metadata .created() - .map_or_else(|_| Utc::now(), DateTime::::from); + .ok() + .and_then(|t| Timestamp::try_from(t).ok()) + .unwrap_or_else(Timestamp::now); let modified_timestamp = metadata .modified() - .map_or_else(|_| Utc::now(), DateTime::::from); + .ok() + .and_then(|t| Timestamp::try_from(t).ok()) + .unwrap_or_else(Timestamp::now); - let import_timestamp = Utc::now(); + let import_timestamp = Timestamp::now(); // Detect media (MIME) type // let media_type = ...; diff --git a/capsule-core/src/validation/structural.rs b/capsule-core/src/validation/structural.rs index 8aa80918..cd5a7f1d 100644 --- a/capsule-core/src/validation/structural.rs +++ b/capsule-core/src/validation/structural.rs @@ -34,8 +34,8 @@ pub fn album_pin_matches(request_protocol: &str, album_pin: &str) -> bool { /// Invariant 7 (time half): the device's directory `added_at` precedes the manifest /// `timestamp`. `None` on an unparseable timestamp (the caller rejects with `400`). pub fn device_added_before(added_at: &str, timestamp: &str) -> Option { - let a = chrono::DateTime::parse_from_rfc3339(added_at).ok()?; - let t = chrono::DateTime::parse_from_rfc3339(timestamp).ok()?; + let a = added_at.parse::().ok()?; + let t = timestamp.parse::().ok()?; Some(a <= t) } @@ -46,9 +46,10 @@ pub fn timestamp_within_drift( server_clock: &str, drift_days: i64, ) -> Option { - let t = chrono::DateTime::parse_from_rfc3339(timestamp).ok()?; - let now = chrono::DateTime::parse_from_rfc3339(server_clock).ok()?; - let delta = (t - now).num_days().abs(); + let t = timestamp.parse::().ok()?; + let now = server_clock.parse::().ok()?; + // Absolute-time days: 24 h, matching the drift guard's UTC semantics. + let delta = (t.as_second() - now.as_second()).abs() / 86_400; Some(delta <= drift_days) } From cac3ecd2d5891a3f6e5c43afdb6d67c14972f6ca Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:05:27 -0400 Subject: [PATCH 47/58] refactor(api): convert chrono to jiff at the entity boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entity crate gains entity::time — the sanctioned chrono <-> jiff seam (lossless second+nanosecond conversion, now_entity() for column writes). model DTOs, service mutations, auth (JWT epoch seconds, password-reset expiry), and upload (Redis session RFC 3339 fields, EXIF capture-time extraction) now speak jiff; chrono remains only as the sea-orm column type in the entity crate and in the frozen capsule-api-library. Upload-session OpenAPI schemas pin the timestamp fields as value_type String (RFC 3339), matching what jiff's serde emits. Stored RFC 3339 values parse identically under jiff (Z vs +00:00 both accepted). Pin: design/dependencies.md (datetime row). --- Cargo.lock | 9 +-- capsule-api/auth/Cargo.toml | 2 +- capsule-api/auth/src/claims/mod.rs | 10 +-- capsule-api/auth/src/models/mod.rs | 4 +- capsule-api/auth/src/models/responses.rs | 4 +- capsule-api/auth/src/routes/password.rs | 2 +- capsule-api/auth/src/service/password.rs | 4 +- capsule-api/auth/src/session/mod.rs | 2 +- capsule-api/entity/Cargo.toml | 1 + capsule-api/entity/src/lib.rs | 1 + capsule-api/entity/src/time.rs | 64 +++++++++++++++++++ capsule-api/model/Cargo.toml | 2 +- capsule-api/model/src/asset.rs | 8 +-- capsule-api/model/src/passkey.rs | 10 +-- capsule-api/model/src/user.rs | 14 ++-- capsule-api/service/Cargo.toml | 2 +- capsule-api/service/src/asset/mutation.rs | 19 +++--- .../service/src/friendship/mutation.rs | 6 +- capsule-api/service/src/passkey/mutation.rs | 8 +-- capsule-api/service/src/stack/mutation.rs | 12 ++-- capsule-api/service/src/user/mutation.rs | 6 +- capsule-api/upload/Cargo.toml | 2 +- capsule-api/upload/src/models/session.rs | 15 +++-- capsule-api/upload/src/service/owner.rs | 7 +- capsule-api/upload/src/service/processing.rs | 11 +++- capsule-api/upload/src/service/upload.rs | 6 +- capsule-api/upload/src/session/mod.rs | 16 ++--- 27 files changed, 161 insertions(+), 86 deletions(-) create mode 100644 capsule-api/entity/src/time.rs diff --git a/Cargo.lock b/Cargo.lock index fe1bf845..913a9a51 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1062,9 +1062,9 @@ dependencies = [ "capsule-api-model", "capsule-api-service", "capsule-i18n", - "chrono", "derive_more", "eyre", + "jiff", "jsonwebtoken", "mime", "nanoid", @@ -1096,6 +1096,7 @@ name = "capsule-api-entity" version = "0.1.0" dependencies = [ "chrono", + "jiff", "nanoid", "sea-orm", "serde_json", @@ -1180,8 +1181,8 @@ version = "0.1.0" dependencies = [ "argon2", "capsule-api-entity", - "chrono", "eyre", + "jiff", "jsonwebtoken", "redis", "salvo", @@ -1200,8 +1201,8 @@ dependencies = [ "capsule-api-entity", "capsule-api-model", "capsule-core", - "chrono", "data-encoding", + "jiff", "nanoid", "sea-orm", "serde", @@ -1270,10 +1271,10 @@ dependencies = [ "capsule-api-service", "capsule-core", "capsule-i18n", - "chrono", "eyre", "futures-util", "indexmap 2.14.0", + "jiff", "jsonwebtoken", "nanoid", "ring", diff --git a/capsule-api/auth/Cargo.toml b/capsule-api/auth/Cargo.toml index c038ea4c..d05ccdbb 100644 --- a/capsule-api/auth/Cargo.toml +++ b/capsule-api/auth/Cargo.toml @@ -16,7 +16,7 @@ capsule-i18n = { path = "../../capsule-i18n" } argon2 = { workspace = true } bb8 = { workspace = true } bb8-redis = { workspace = true } -chrono = { workspace = true } +jiff = { workspace = true } derive_more = { workspace = true, features = ["from"] } eyre = { workspace = true } jsonwebtoken = { workspace = true } diff --git a/capsule-api/auth/src/claims/mod.rs b/capsule-api/auth/src/claims/mod.rs index d8acc754..d2fb3cd9 100644 --- a/capsule-api/auth/src/claims/mod.rs +++ b/capsule-api/auth/src/claims/mod.rs @@ -1,8 +1,8 @@ use std::sync::OnceLock; use std::time::Duration; -use chrono::Utc; use environment::constants::{ACCESS_TOKEN_EXPIRY, REFRESH_TOKEN_EXPIRY}; +use jiff::Timestamp; use jsonwebtoken::{Algorithm, DecodingKey, EncodingKey, TokenData, Validation, decode, encode}; use serde::{Deserialize, Serialize}; @@ -58,7 +58,7 @@ impl Claims { sid: Option, ) -> Self { let expiry_duration = expiry_duration.into(); - let iat = Utc::now().timestamp() as u64; + let iat = Timestamp::now().as_second() as u64; let exp: u64 = iat + expiry_duration.as_secs(); Self { @@ -110,7 +110,7 @@ impl Claims { /// Returns true if the token has not expired pub fn has_expired(&self) -> bool { - self.exp <= Utc::now().timestamp() as u64 + self.exp <= Timestamp::now().as_second() as u64 } /// Returns true if the token has valid issuer @@ -238,8 +238,8 @@ mod tests { let claims = Claims { sub: user_id, - exp: Utc::now().timestamp() as u64 - 100, - iat: Utc::now().timestamp() as u64, + exp: Timestamp::now().as_second() as u64 - 100, + iat: Timestamp::now().as_second() as u64, jti: uuid::Uuid::new_v4().to_string(), iss: ISSUER.to_string(), sid: None, // Added sid diff --git a/capsule-api/auth/src/models/mod.rs b/capsule-api/auth/src/models/mod.rs index e91c46c6..09e107e4 100644 --- a/capsule-api/auth/src/models/mod.rs +++ b/capsule-api/auth/src/models/mod.rs @@ -97,8 +97,8 @@ impl From for UserProfile { email, profile_image_url, is_admin, - created_at: created_at.to_rfc3339(), - updated_at: modified_at.to_rfc3339(), + created_at: created_at.to_string(), + updated_at: modified_at.to_string(), registered_via, } } diff --git a/capsule-api/auth/src/models/responses.rs b/capsule-api/auth/src/models/responses.rs index db599697..225b174b 100644 --- a/capsule-api/auth/src/models/responses.rs +++ b/capsule-api/auth/src/models/responses.rs @@ -995,8 +995,8 @@ impl From for PasskeyModel { Self { id: passkey.id, name: passkey.name, - created_at: passkey.created_at.to_rfc3339(), - last_used_at: passkey.last_used_at.map(|t| t.to_rfc3339()), + created_at: passkey.created_at.to_string(), + last_used_at: passkey.last_used_at.map(|t| t.to_string()), } } } diff --git a/capsule-api/auth/src/routes/password.rs b/capsule-api/auth/src/routes/password.rs index b406cdf7..a7beaa91 100644 --- a/capsule-api/auth/src/routes/password.rs +++ b/capsule-api/auth/src/routes/password.rs @@ -119,7 +119,7 @@ pub async fn reset_password( // Check expiry if user .password_reset_expires_at - .is_none_or(|exp| exp < chrono::Utc::now()) + .is_none_or(|exp| exp.timestamp() < jiff::Timestamp::now().as_second()) { return PasswordResetResponses::InvalidToken; } diff --git a/capsule-api/auth/src/service/password.rs b/capsule-api/auth/src/service/password.rs index c09d5760..bff7090b 100644 --- a/capsule-api/auth/src/service/password.rs +++ b/capsule-api/auth/src/service/password.rs @@ -1,6 +1,6 @@ use std::time::Instant; -use chrono::{Duration, Utc}; +use jiff::{SignedDuration, Timestamp}; use model::errors::InternalServerError; use sea_orm::DatabaseConnection; use service::user as UserService; @@ -30,7 +30,7 @@ impl PasswordService { Ok(Some(user)) => { // Generate token let token = nanoid::nanoid!(); - let expires_at = Utc::now() + Duration::hours(1); + let expires_at = Timestamp::now() + SignedDuration::from_hours(1); // Update user with token UserService::Mutation::update_password_reset_token( diff --git a/capsule-api/auth/src/session/mod.rs b/capsule-api/auth/src/session/mod.rs index 732ace12..7502c949 100644 --- a/capsule-api/auth/src/session/mod.rs +++ b/capsule-api/auth/src/session/mod.rs @@ -56,7 +56,7 @@ impl SessionManager { ip_address: Option, ) -> Result { let sid = nanoid::nanoid!(); - let now = chrono::Utc::now().timestamp(); + let now = jiff::Timestamp::now().as_second(); let session = Session { user_id: user_id.clone(), created_at: now, diff --git a/capsule-api/entity/Cargo.toml b/capsule-api/entity/Cargo.toml index 9bf0e512..9d86c173 100644 --- a/capsule-api/entity/Cargo.toml +++ b/capsule-api/entity/Cargo.toml @@ -10,6 +10,7 @@ path = "src/lib.rs" [dependencies] chrono = { workspace = true } +jiff = { workspace = true } nanoid = { workspace = true } sea-orm = { workspace = true, features = [ "sqlx-postgres", diff --git a/capsule-api/entity/src/lib.rs b/capsule-api/entity/src/lib.rs index 94b6fb45..b65f5927 100644 --- a/capsule-api/entity/src/lib.rs +++ b/capsule-api/entity/src/lib.rs @@ -13,4 +13,5 @@ pub mod person; pub mod share_link; pub mod smart_tag; pub mod stack_member; +pub mod time; pub mod user; diff --git a/capsule-api/entity/src/time.rs b/capsule-api/entity/src/time.rs new file mode 100644 index 00000000..e36d8831 --- /dev/null +++ b/capsule-api/entity/src/time.rs @@ -0,0 +1,64 @@ +//! The chrono ⇄ jiff conversion seam. +//! +//! Per the [Dependencies design doc](../../../capsule-docs/src/content/docs/design/dependencies.md), +//! `jiff` is the canonical datetime library for all domain logic; `chrono` survives only +//! as the sea-orm column type inside the entity crates. Every value crossing the entity +//! boundary converts through these helpers, losslessly at second + nanosecond precision +//! for any real wall-clock value. + +use jiff::Timestamp; +use sea_orm::prelude::{DateTimeUtc, DateTimeWithTimeZone}; + +/// Convert a sea-orm UTC column value to a jiff [`Timestamp`]. +/// +/// Falls back to the Unix epoch for values outside jiff's representable range +/// (year 9999+ / pre-(-9999) — unreachable for real row timestamps). +pub fn entity_to_ts(dt: DateTimeUtc) -> Timestamp { + let nanos = i32::try_from(dt.timestamp_subsec_nanos()).unwrap_or(0); + Timestamp::new(dt.timestamp(), nanos).unwrap_or(Timestamp::UNIX_EPOCH) +} + +/// Convert a sea-orm timezone-aware column value to a jiff [`Timestamp`]. +pub fn entity_tz_to_ts(dt: DateTimeWithTimeZone) -> Timestamp { + entity_to_ts(dt.to_utc()) +} + +/// Convert a jiff [`Timestamp`] to a sea-orm UTC column value. +pub fn ts_to_entity(ts: Timestamp) -> DateTimeUtc { + let nanos = u32::try_from(ts.subsec_nanosecond()).unwrap_or(0); + DateTimeUtc::from_timestamp(ts.as_second(), nanos).unwrap_or_default() +} + +/// Convert a jiff [`Timestamp`] to a sea-orm timezone-aware column value (UTC offset). +pub fn ts_to_entity_tz(ts: Timestamp) -> DateTimeWithTimeZone { + ts_to_entity(ts).into() +} + +/// The current instant as a sea-orm UTC column value — the sanctioned `now()` for +/// entity writes. +pub fn now_entity() -> DateTimeUtc { + ts_to_entity(Timestamp::now()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn round_trips_losslessly() { + let ts: Timestamp = "2026-07-10T12:34:56.123456789Z".parse().expect("parses"); + assert_eq!(entity_to_ts(ts_to_entity(ts)), ts); + } + + #[test] + fn tz_round_trip_preserves_instant() { + let ts: Timestamp = "2026-07-10T12:34:56.5Z".parse().expect("parses"); + assert_eq!(entity_tz_to_ts(ts_to_entity_tz(ts)), ts); + } + + #[test] + fn now_is_representable_both_ways() { + let now = now_entity(); + assert!(entity_to_ts(now).as_second() > 0); + } +} diff --git a/capsule-api/model/Cargo.toml b/capsule-api/model/Cargo.toml index 6db7c5f9..0689c1ff 100644 --- a/capsule-api/model/Cargo.toml +++ b/capsule-api/model/Cargo.toml @@ -11,7 +11,7 @@ path = "src/lib.rs" [dependencies] capsule-api-entity = { path = "../entity" } argon2 = { workspace = true } -chrono = { workspace = true } +jiff = { workspace = true } jsonwebtoken = { workspace = true } salvo = { workspace = true } serde = { workspace = true } diff --git a/capsule-api/model/src/asset.rs b/capsule-api/model/src/asset.rs index 39af35ce..d6d52615 100644 --- a/capsule-api/model/src/asset.rs +++ b/capsule-api/model/src/asset.rs @@ -1,4 +1,4 @@ -use chrono::{DateTime, Utc}; +use jiff::Timestamp; use serde::{Deserialize, Serialize}; #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -35,7 +35,7 @@ pub struct CreateAsset { pub file_size: i64, pub file_hash: String, pub content_type: String, - pub date: Option>, + pub date: Option, pub uploaded: bool, } @@ -49,7 +49,7 @@ pub struct UpdateAsset { pub file_size: Option, pub file_hash: Option, pub content_type: Option, - pub date: Option>>, + pub date: Option>, pub uploaded: Option, - pub deleted_at: Option>>, + pub deleted_at: Option>, } diff --git a/capsule-api/model/src/passkey.rs b/capsule-api/model/src/passkey.rs index 21a3a079..d0e7b795 100644 --- a/capsule-api/model/src/passkey.rs +++ b/capsule-api/model/src/passkey.rs @@ -1,4 +1,4 @@ -use chrono::{DateTime, Utc}; +use jiff::Timestamp; use serde::{Deserialize, Serialize}; #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -22,10 +22,10 @@ pub struct Passkey { pub name: String, /// When this key was created. - pub created_at: DateTime, + pub created_at: Timestamp, /// When this key was last used, if ever. - pub last_used_at: Option>, + pub last_used_at: Option, /// The AAGUID identifying the authenticator model. pub aaguid: Option, @@ -46,8 +46,8 @@ impl From for Passkey { public_key: model.public_key, counter: model.counter, name: model.name, - created_at: model.created_at, - last_used_at: model.last_used_at, + created_at: entity::time::entity_to_ts(model.created_at), + last_used_at: model.last_used_at.map(entity::time::entity_to_ts), aaguid: model.aaguid, backup_eligible: model.backup_eligible, backup_state: model.backup_state, diff --git a/capsule-api/model/src/user.rs b/capsule-api/model/src/user.rs index 386e5ddc..1ea00f82 100644 --- a/capsule-api/model/src/user.rs +++ b/capsule-api/model/src/user.rs @@ -1,4 +1,4 @@ -use chrono::{DateTime, Utc}; +use jiff::Timestamp; use serde::{Deserialize, Serialize}; pub struct CreateUser { @@ -44,13 +44,13 @@ pub struct User { pub is_admin: bool, /// Timestamp of when the user record was originally created. - pub created_at: DateTime, + pub created_at: Timestamp, /// Timestamp of the last time the user record was updated. - pub modified_at: DateTime, + pub modified_at: Timestamp, /// Timestamp of when the user was soft-deleted. If None, the user is active. - pub deleted_at: Option>, + pub deleted_at: Option, /// How the account was created (e.g. 'invitation'); None for seeded/legacy/unknown. pub registered_via: Option, @@ -89,9 +89,9 @@ impl From for User { profile_image_url, needs_onboarding, is_admin, - created_at, - modified_at, - deleted_at, + created_at: entity::time::entity_to_ts(created_at), + modified_at: entity::time::entity_to_ts(modified_at), + deleted_at: deleted_at.map(entity::time::entity_to_ts), registered_via, } } diff --git a/capsule-api/service/Cargo.toml b/capsule-api/service/Cargo.toml index ed5692fd..73d864c7 100644 --- a/capsule-api/service/Cargo.toml +++ b/capsule-api/service/Cargo.toml @@ -15,7 +15,7 @@ capsule-core = { path = "../../capsule-core" } serde = { workspace = true } serde_json = { workspace = true } uuid = { workspace = true } -chrono = { workspace = true } +jiff = { workspace = true } data-encoding = { workspace = true } nanoid = { workspace = true } sea-orm = { workspace = true, features = [ diff --git a/capsule-api/service/src/asset/mutation.rs b/capsule-api/service/src/asset/mutation.rs index 806cda47..1d88874b 100644 --- a/capsule-api/service/src/asset/mutation.rs +++ b/capsule-api/service/src/asset/mutation.rs @@ -1,5 +1,6 @@ use ::entity::asset::{self, AssetType}; -use chrono::{DateTime, Utc}; +use ::entity::time; +use jiff::Timestamp; use nanoid::nanoid; use sea_orm::*; @@ -18,7 +19,7 @@ impl Mutation { file_size: i64, file_hash: String, content_type: String, - captured_at: Option>, + captured_at: Option, ) -> Result { let model = asset::ActiveModel { id: Set(nanoid!()), @@ -32,9 +33,9 @@ impl Mutation { file_size: Set(file_size), file_hash: Set(file_hash), content_type: Set(content_type), - captured_at: Set(captured_at), - uploaded_at: Set(Utc::now()), - modified_at: Set(Utc::now().into()), + captured_at: Set(captured_at.map(time::ts_to_entity)), + uploaded_at: Set(time::now_entity()), + modified_at: Set(time::now_entity().into()), uploaded: Set(false), ..Default::default() }; @@ -47,7 +48,7 @@ impl Mutation { asset_id: &str, width: i32, height: i32, - captured_at: Option>, + captured_at: Option, ) -> Result { let asset = asset::Entity::find_by_id(asset_id) .one(db) @@ -59,9 +60,9 @@ impl Mutation { model.width = Set(width); model.height = Set(height); if captured_at.is_some() { - model.captured_at = Set(captured_at); + model.captured_at = Set(captured_at.map(time::ts_to_entity)); } - model.modified_at = Set(Utc::now().into()); + model.modified_at = Set(time::now_entity().into()); model.update(db).await } @@ -76,7 +77,7 @@ impl Mutation { .ok_or_else(|| DbErr::Custom("Asset not found".to_string()))?; let mut model: asset::ActiveModel = asset.into(); - model.deleted_at = Set(Some(Utc::now())); + model.deleted_at = Set(Some(time::now_entity())); model.update(db).await } diff --git a/capsule-api/service/src/friendship/mutation.rs b/capsule-api/service/src/friendship/mutation.rs index e567ee10..d13efc49 100644 --- a/capsule-api/service/src/friendship/mutation.rs +++ b/capsule-api/service/src/friendship/mutation.rs @@ -1,5 +1,5 @@ use ::entity::friendship::{self, FriendshipStatus}; -use chrono::Utc; +use ::entity::time; use sea_orm::*; use super::error::{ @@ -45,7 +45,7 @@ impl Mutation { user_id: Set(user_id.to_string()), friend_id: Set(friend_id.to_string()), status: Set(FriendshipStatus::Pending), - created_at: Set(Utc::now()), + created_at: Set(time::now_entity()), ..Default::default() }; @@ -84,7 +84,7 @@ impl Mutation { let mut model: friendship::ActiveModel = request.into(); model.status = Set(FriendshipStatus::Accepted); - model.accepted_at = Set(Some(Utc::now())); + model.accepted_at = Set(Some(time::now_entity())); let updated = model.update(db).await?; Ok(AcceptRequestResult::Accepted(updated)) } diff --git a/capsule-api/service/src/passkey/mutation.rs b/capsule-api/service/src/passkey/mutation.rs index bb9938e1..0b1ba31d 100644 --- a/capsule-api/service/src/passkey/mutation.rs +++ b/capsule-api/service/src/passkey/mutation.rs @@ -1,4 +1,4 @@ -use chrono::Utc; +use ::entity::time; use entity::passkey; use sea_orm::{ ActiveModelTrait, ColumnTrait, DatabaseConnection, DbErr, EntityTrait, QueryFilter, Set, @@ -30,8 +30,8 @@ impl Mutation { public_key: Set(args.public_key), counter: Set(args.counter), name: Set(args.name), - created_at: Set(Utc::now()), - last_used_at: Set(Some(Utc::now())), + created_at: Set(time::now_entity()), + last_used_at: Set(Some(time::now_entity())), aaguid: Set(args.aaguid), backup_eligible: Set(args.backup_eligible), backup_state: Set(args.backup_state), @@ -54,7 +54,7 @@ impl Mutation { let mut active: passkey::ActiveModel = passkey.into(); active.counter = Set(new_counter); - active.last_used_at = Set(Some(Utc::now())); + active.last_used_at = Set(Some(time::now_entity())); active.update(conn).await } diff --git a/capsule-api/service/src/stack/mutation.rs b/capsule-api/service/src/stack/mutation.rs index 3260d1ba..ed6ba6db 100644 --- a/capsule-api/service/src/stack/mutation.rs +++ b/capsule-api/service/src/stack/mutation.rs @@ -1,7 +1,7 @@ use ::entity::asset::{self, Entity as Asset}; use ::entity::asset_stack::{self, Entity as AssetStack, StackType}; use ::entity::stack_member::{self, Entity as StackMember, MemberRole}; -use chrono::Utc; +use ::entity::time; use nanoid::nanoid; use sea_orm::prelude::Expr; use sea_orm::*; @@ -39,8 +39,8 @@ impl Mutation { metadata: Set(metadata_json), is_collapsed: Set(true), is_auto_generated: Set(true), - created_at: Set(Utc::now()), - modified_at: Set(Utc::now()), + created_at: Set(time::now_entity()), + modified_at: Set(time::now_entity()), }; let stack = stack.insert(db).await?; @@ -109,7 +109,7 @@ impl Mutation { sequence_order: Set(sequence_order), member_role: Set(member_role), metadata: Set(metadata_json), - created_at: Set(Utc::now()), + created_at: Set(time::now_entity()), }; member.insert(db).await } @@ -176,7 +176,7 @@ impl Mutation { // Update stack's primary let mut stack_active: asset_stack::ActiveModel = stack.into(); stack_active.primary_asset_id = Set(new_primary_id.to_string()); - stack_active.modified_at = Set(Utc::now()); + stack_active.modified_at = Set(time::now_entity()); stack_active.update(db).await } @@ -193,7 +193,7 @@ impl Mutation { let mut stack_active: asset_stack::ActiveModel = stack.into(); stack_active.is_collapsed = Set(is_collapsed); - stack_active.modified_at = Set(Utc::now()); + stack_active.modified_at = Set(time::now_entity()); stack_active.update(db).await } diff --git a/capsule-api/service/src/user/mutation.rs b/capsule-api/service/src/user/mutation.rs index ee107c27..053122f6 100644 --- a/capsule-api/service/src/user/mutation.rs +++ b/capsule-api/service/src/user/mutation.rs @@ -74,7 +74,7 @@ impl Mutation { .ok_or(DbErr::RecordNotFound("User not found".to_string()))?; let mut user: user::ActiveModel = user_model.into(); - user.last_login_at = Set(Some(chrono::Utc::now())); + user.last_login_at = Set(Some(::entity::time::now_entity())); user.failed_login_attempts = Set(0); user.update(db).await @@ -100,7 +100,7 @@ impl Mutation { db: &DbConn, id: &str, token: &str, - expires_at: chrono::DateTime, + expires_at: jiff::Timestamp, ) -> Result { let user_model = User::find_by_id(id) .one(db) @@ -109,7 +109,7 @@ impl Mutation { let mut user: user::ActiveModel = user_model.into(); user.password_reset_token = Set(Some(token.to_string())); - user.password_reset_expires_at = Set(Some(expires_at)); + user.password_reset_expires_at = Set(Some(::entity::time::ts_to_entity(expires_at))); user.update(db).await } diff --git a/capsule-api/upload/Cargo.toml b/capsule-api/upload/Cargo.toml index 9e247d32..a6be8d7f 100644 --- a/capsule-api/upload/Cargo.toml +++ b/capsule-api/upload/Cargo.toml @@ -31,7 +31,7 @@ thiserror = { workspace = true } tokio = { workspace = true } tracing = { workspace = true } uuid = { workspace = true } -chrono = { workspace = true } +jiff = { workspace = true } ring = { workspace = true } base64 = { workspace = true } secrecy = { workspace = true } diff --git a/capsule-api/upload/src/models/session.rs b/capsule-api/upload/src/models/session.rs index 2c88bef5..3cdee40e 100644 --- a/capsule-api/upload/src/models/session.rs +++ b/capsule-api/upload/src/models/session.rs @@ -1,4 +1,4 @@ -use chrono::{DateTime, Utc}; +use jiff::Timestamp; use salvo::oapi::ToSchema; use serde::{Deserialize, Serialize}; @@ -67,14 +67,17 @@ pub(crate) struct UploadSession { pub total_size: u64, pub status: UploadSessionStatus, - /// Creation timestamp - pub created_at: DateTime, + /// Creation timestamp (RFC 3339) + #[salvo(schema(value_type = String))] + pub created_at: Timestamp, /// Timestamp of the last accepted chunk (creation time if none). Anchors the /// ≥1-hour survival floor; the pressure sweeper (S-C1) evicts /// least-recently-progressed first. - pub last_progress_at: DateTime, - /// Expiration timestamp (the 24-hour cap) - pub expires_at: DateTime, + #[salvo(schema(value_type = String))] + pub last_progress_at: Timestamp, + /// Expiration timestamp (the 24-hour cap, RFC 3339) + #[salvo(schema(value_type = String))] + pub expires_at: Timestamp, } #[derive(Debug, Copy, Clone, Eq, PartialEq, Serialize, Deserialize, ToSchema)] diff --git a/capsule-api/upload/src/service/owner.rs b/capsule-api/upload/src/service/owner.rs index 803526c5..461a1ea9 100644 --- a/capsule-api/upload/src/service/owner.rs +++ b/capsule-api/upload/src/service/owner.rs @@ -1,5 +1,4 @@ -use chrono::Utc; -use entity::{owner, owner_member}; +use entity::{owner, owner_member, time}; use nanoid::nanoid; use sea_orm::{ ActiveModelTrait, ColumnTrait, DatabaseConnection, DatabaseTransaction, EntityTrait, @@ -74,7 +73,7 @@ impl OwnerService { let owner_id = nanoid!(); let owner = owner::ActiveModel { id: Set(owner_id.clone()), - created_at: Set(Utc::now()), + created_at: Set(time::now_entity()), }; owner.insert(txn).await?; @@ -83,7 +82,7 @@ impl OwnerService { let member = owner_member::ActiveModel { owner_id: Set(owner_id.clone()), user_id: Set(uid.clone()), - created_at: Set(Utc::now()), + created_at: Set(time::now_entity()), ..Default::default() }; member.insert(txn).await?; diff --git a/capsule-api/upload/src/service/processing.rs b/capsule-api/upload/src/service/processing.rs index 8836724c..a4429d72 100644 --- a/capsule-api/upload/src/service/processing.rs +++ b/capsule-api/upload/src/service/processing.rs @@ -1,7 +1,7 @@ use std::path::Path; use capsule_core::media::fs::{ImageParseError, load_image}; -use chrono::{DateTime, Utc}; +use jiff::Timestamp; /// Service for processing uploaded assets #[derive(Clone)] @@ -10,7 +10,7 @@ pub(crate) struct ProcessingService; pub(crate) struct ExtractedMetadata { pub width: i32, pub height: i32, - pub date: Option>, + pub date: Option, } impl ProcessingService { @@ -24,7 +24,12 @@ impl ProcessingService { ) -> Result { let image = load_image(path).await?; let metadata = image.get_metadata(); - let date = metadata.date_taken; + // EXIF capture time carries no offset; interpret the civil datetime as UTC, matching + // the prior behavior of this extraction path. + let date = metadata + .date_taken + .and_then(|dt| dt.to_zoned(jiff::tz::TimeZone::UTC).ok()) + .map(|z| z.timestamp()); Ok(ExtractedMetadata { width: metadata.width as i32, diff --git a/capsule-api/upload/src/service/upload.rs b/capsule-api/upload/src/service/upload.rs index 6c741baf..3a8435b5 100644 --- a/capsule-api/upload/src/service/upload.rs +++ b/capsule-api/upload/src/service/upload.rs @@ -1,8 +1,8 @@ use std::clone::Clone; use capsule_core::utils::hash::get_file_hash; -use chrono::Utc; use entity::asset; +use jiff::{SignedDuration, Timestamp}; use nanoid::nanoid; use sea_orm::{DatabaseConnection, TransactionTrait}; use service::{album as AlbumService, asset as AssetService}; @@ -111,7 +111,7 @@ impl UploadService { .await .map_err(|e| UploadError::Unknown(e.to_string()))?; - let now = Utc::now(); + let now = Timestamp::now(); let session = UploadSession { id: upload_id.clone(), asset_id: asset.id.clone(), @@ -130,7 +130,7 @@ impl UploadService { status: UploadSessionStatus::Pending, created_at: now, last_progress_at: now, - expires_at: now + chrono::Duration::hours(24), + expires_at: now + SignedDuration::from_hours(24), }; // Create session in Redis (atomic HSET) diff --git a/capsule-api/upload/src/session/mod.rs b/capsule-api/upload/src/session/mod.rs index e5252cb2..bef897a8 100644 --- a/capsule-api/upload/src/session/mod.rs +++ b/capsule-api/upload/src/session/mod.rs @@ -4,7 +4,7 @@ use std::time::Duration; use bb8_redis::RedisConnectionManager; use bb8_redis::bb8::Pool; use bb8_redis::redis::AsyncCommands; -use chrono::{DateTime, Utc}; +use jiff::Timestamp; use crate::error::UploadError; use crate::models::session::{BlobRole, UploadSession, UploadSessionStatus}; @@ -72,12 +72,12 @@ impl UploadSessionManager { .unwrap_or_else(|_| "\"Pending\"".to_string()) .into_bytes(), ), - ("created_at", session.created_at.to_rfc3339().into_bytes()), + ("created_at", session.created_at.to_string().into_bytes()), ( "last_progress_at", - session.last_progress_at.to_rfc3339().into_bytes(), + session.last_progress_at.to_string().into_bytes(), ), - ("expires_at", session.expires_at.to_rfc3339().into_bytes()), + ("expires_at", session.expires_at.to_string().into_bytes()), ]; // Store optional fields if present @@ -141,7 +141,7 @@ impl UploadSessionManager { let mut conn = self.pool.get().await?; let key = self.key(upload_id); let _: () = conn - .hset(&key, "last_progress_at", Utc::now().to_rfc3339()) + .hset(&key, "last_progress_at", Timestamp::now().to_string()) .await?; Ok(()) } @@ -258,15 +258,15 @@ impl UploadSessionManager { let status: UploadSessionStatus = serde_json::from_str(&status_str) .map_err(|e| UploadError::Unknown(format!("Invalid status '{status_str}': {e}")))?; - let created_at: DateTime = get_string("created_at")? + let created_at: Timestamp = get_string("created_at")? .parse() .map_err(|e| UploadError::Unknown(format!("Invalid created_at: {e}")))?; - let last_progress_at: DateTime = get_string("last_progress_at")? + let last_progress_at: Timestamp = get_string("last_progress_at")? .parse() .map_err(|e| UploadError::Unknown(format!("Invalid last_progress_at: {e}")))?; - let expires_at: DateTime = get_string("expires_at")? + let expires_at: Timestamp = get_string("expires_at")? .parse() .map_err(|e| UploadError::Unknown(format!("Invalid expires_at: {e}")))?; From a4a32f646588401d1358f054be5fbf0d5e3d265a Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:06:10 -0400 Subject: [PATCH 48/58] refactor(cli): move status datetimes to jiff capsule-cli/entity keeps chrono as its sea-orm column type per the entity-boundary rule in design/dependencies.md; the CLI binary itself is now chrono-free. --- Cargo.lock | 2 +- capsule-cli/Cargo.toml | 2 +- capsule-cli/src/status.rs | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 913a9a51..cac8cfe8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1298,7 +1298,6 @@ dependencies = [ "capsule-cli-entity", "capsule-cli-migration", "capsule-core", - "chrono", "clap", "color-eyre", "colored", @@ -1308,6 +1307,7 @@ dependencies = [ "futures", "humansize", "indexmap 2.14.0", + "jiff", "nanoid", "sea-orm", "serde", diff --git a/capsule-cli/Cargo.toml b/capsule-cli/Cargo.toml index 0bf178d4..d46d6f85 100644 --- a/capsule-cli/Cargo.toml +++ b/capsule-cli/Cargo.toml @@ -13,7 +13,7 @@ capsule-cli-migration = { path = "./migration" } capsule-core = { path = "../capsule-core" } base64 = { workspace = true } capitalize = "0.3.4" -chrono = { workspace = true } +jiff = { workspace = true } clap = { version = "4.6.1", features = ["derive"] } color-eyre = "0.6.5" colored = "3.1.1" diff --git a/capsule-cli/src/status.rs b/capsule-cli/src/status.rs index 8383b8f9..ed867a33 100644 --- a/capsule-cli/src/status.rs +++ b/capsule-cli/src/status.rs @@ -1,7 +1,7 @@ use std::path::PathBuf; use std::time::SystemTime; -use chrono::{DateTime, Utc}; +use jiff::Timestamp; use colored::*; use eyre::{Result, eyre}; use humansize::{BINARY, format_size}; @@ -25,7 +25,7 @@ pub(crate) struct AuthStatus { /// Whether token is valid according to server pub token_valid: bool, /// Expiry time of the authentication token - pub token_expires_at: Option>, + pub token_expires_at: Option, } pub(crate) struct LocalEnvStatus { @@ -119,7 +119,7 @@ impl AuthStatus { let username = config.user_id.clone(); // TODO: Fetch from server, otherwise use cache let token_valid = config.auth_token.is_some(); // Assume token is valid if it exists let token_expires_at = if token_valid { - Some(Utc::now() + chrono::Duration::days(30)) // Mock expiry 30 days from now + Some(Timestamp::now() + jiff::SignedDuration::from_hours(30 * 24)) // Mock expiry 30 days from now } else { None }; @@ -148,7 +148,7 @@ impl AuthStatus { println!( " {} {}", "Expires:".dimmed(), - expires.to_rfc3339().dimmed() + expires.to_string().dimmed() ); } } else { From 5a483cb6175fac08c14ffd16a1ee4e348ccddb99 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:06:13 -0400 Subject: [PATCH 49/58] build(sdk): drop the unused chrono dependency capsule-sdk declared chrono but no source consumed it; the SDK's future datetime needs are jiff per design/dependencies.md. --- Cargo.lock | 1 - capsule-sdk/Cargo.toml | 1 - 2 files changed, 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index cac8cfe8..e21cce86 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1414,7 +1414,6 @@ dependencies = [ name = "capsule-sdk" version = "0.1.0" dependencies = [ - "chrono", "log", "reqwest", "serde", diff --git a/capsule-sdk/Cargo.toml b/capsule-sdk/Cargo.toml index 1669073b..d431abc1 100644 --- a/capsule-sdk/Cargo.toml +++ b/capsule-sdk/Cargo.toml @@ -4,7 +4,6 @@ version.workspace = true edition.workspace = true [dependencies] -chrono = { workspace = true } log = { workspace = true } serde = { workspace = true } thiserror = { workspace = true } From a5336103d764de873d53dc0a69def5868fef9008 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:07:01 -0400 Subject: [PATCH 50/58] refactor(xtask): adopt eyre for error handling anyhow is retired per the errors row in design/dependencies.md: thiserror in libraries, eyre/color-eyre in binaries. --- Cargo.lock | 2 +- xtask/Cargo.toml | 2 +- xtask/src/i18n.rs | 2 +- xtask/src/main.rs | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index e21cce86..46789754 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -8661,7 +8661,7 @@ checksum = "636f85e5ca6488e96401b61eb7de54f4e44755c988af0f52cf90230c312a1a89" name = "xtask" version = "0.1.0" dependencies = [ - "anyhow", + "eyre", "regex", "semver", "serde_json", diff --git a/xtask/Cargo.toml b/xtask/Cargo.toml index 8343cdae..2e8343c4 100644 --- a/xtask/Cargo.toml +++ b/xtask/Cargo.toml @@ -5,7 +5,7 @@ edition.workspace = true publish.workspace = true [dependencies] -anyhow = "1" +eyre = { workspace = true } regex = "1" semver = "1" serde_json = "1" diff --git a/xtask/src/i18n.rs b/xtask/src/i18n.rs index 0e0960b6..44af081f 100644 --- a/xtask/src/i18n.rs +++ b/xtask/src/i18n.rs @@ -17,7 +17,7 @@ use std::fmt::Write as _; use std::fs; use std::path::{Path, PathBuf}; -use anyhow::{Context, Result, bail}; +use eyre::{Context, ContextCompat, Result, bail}; use regex::Regex; use serde_json::{Map, Value}; diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 84a48212..154bab6e 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -15,7 +15,7 @@ mod i18n; use std::fs; use std::path::{Path, PathBuf}; -use anyhow::{Context, Result, bail}; +use eyre::{Context, ContextCompat, Result, bail}; use regex::{Captures, Regex}; use semver::Version; use toml_edit::Item; From ee2d797f5f0a10a3d6f1fb259e8f492f9c7ab18d Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:08:25 -0400 Subject: [PATCH 51/58] docs(design): store the GPS datum verbatim and fold BD-09 at input MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the design half of issue #325: the sidecar gps value gains a closed datum tag (wgs84 | gcj02, wire-absent when wgs84 so existing sidecars and vectors stay byte-identical). User-inputted GCJ-02 is stored losslessly as entered — the inverse conversion has no exact solution, so converting on input would destroy ground truth. BD-09 is never stored: the exact closed-form fold to GCJ-02 runs at the input edge. Display conversion stays client-side behind the geocoordinates-rs gate. Implementation is indexed as slice S-A7. --- SLICES.md | 19 ++++++++++++++++++- .../src/content/docs/design/metadata.md | 9 ++++++--- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/SLICES.md b/SLICES.md index 2c3c9da5..2f055aab 100644 --- a/SLICES.md +++ b/SLICES.md @@ -66,6 +66,7 @@ its slice. | S-A4 | P-256 hybrid DSK variant | core-crypto | — | L | ready | | S-A5 | Share-link crypto (`capsule_core::sharing`) | core-crypto | — | M | ready | | S-A6 | Drop crypto (`capsule_core::drop`, incl. WASM build) | core-crypto | S-A1 | L | ready | +| S-A7 | `gps.datum` sidecar field + BD-09 input fold | core-crypto | geocoordinates-rs (fold only) | S | ready | | S-B1 | Thumbnail/LQIP generation | media/import | — | L | ready | | S-B2 | Signed-path import-executor rewrite | media/import | S-B1 | L | ready | | S-B3 | Streaming import (probe, `total_size`, drive mode) | media/import | S-D1, S-D4 | L | ready | @@ -182,7 +183,7 @@ pass until the gate lifts. | --- | --- | --- | | `rawshift` (in-house RAW decode; git submodule, alpha, consumed by nothing yet) | stabilizing | Full RAW support in S-B1/S-B2. v1 ships the zune-jpeg format set; the `media::image::formats::raw` stub is the integration point. | | `spargen` (in-house OpenAPI **3.1** client generator) | in development | S-D8 (typed REST client + `AuthenticatedClient` revival). Progenitor is gone — we do not downgrade schemas to 3.0. | -| `geocoordinates-rs` (in-house WGS-84 ↔ GCJ-02/BD-09 conversions) | planned | The deterministic client-side coordinate conversion named in [Metadata — Geolocation](capsule-docs/src/content/docs/design/metadata.md); consumed by map display and S-H3 geo features. Until it lands, WGS-84 storage is unaffected (conversion is display-only). | +| `geocoordinates-rs` (in-house WGS-84 ↔ GCJ-02/BD-09 conversions) | planned | The deterministic client-side coordinate conversion named in [Metadata — Geolocation](capsule-docs/src/content/docs/design/metadata.md); consumed by map display, S-H3 geo features, and S-A7's exact BD-09→GCJ-02 input fold. Until it lands, verbatim datum storage is unaffected (display conversion and the BD-09 fold are the gated pieces). | | `openmls` X-Wing/SHA-512 ciphersuite (codepoint pending) | blocked upstream | S-X1 → S-X2 → S-X3 (tracked in Lane X). | ## Lane A — core crypto @@ -267,6 +268,22 @@ pass until the gate lifts. Validation bullets pass; the sealing path compiles to `wasm32-unknown-unknown`. - **Tier:** Unit (seal round-trip + adoption rewrap). +### S-A7 — `gps.datum` sidecar field + BD-09 input fold + +- **Contract:** [Metadata — Geolocation](capsule-docs/src/content/docs/design/metadata.md), + [Metadata — Closed Enum Value Sets](capsule-docs/src/content/docs/design/metadata.md). +- **Deliverable:** the closed `GpsDatum` enum (`wgs84 | gcj02`) in + `capsule-core::domain`; the optional `datum` key on the sidecar `gps` value + (wire-absent = `wgs84`, byte-identity regression-tested against the existing + known-answer vectors, plus a new populated-`datum` vector); the exact BD-09 → GCJ-02 + fold applied at the input edge (from `geocoordinates-rs` — the only gated piece; + verbatim storage of either datum needs no conversion code). The lossy display + conversions stay behind the `geocoordinates-rs` gate and are **not** this slice. +- **Done when:** the metadata doc's datum-verbatim-storage Validation bullet passes + (GCJ-02 round-trips unconverted; BD-09 folds exactly; WGS-84 stays wire-absent and + byte-identical); `mise run check-rust` green. +- **Tier:** Unit. + ## Lane B — media / import ### S-B1 — Thumbnail/LQIP generation diff --git a/capsule-docs/src/content/docs/design/metadata.md b/capsule-docs/src/content/docs/design/metadata.md index 8d540e02..bb9fb043 100644 --- a/capsule-docs/src/content/docs/design/metadata.md +++ b/capsule-docs/src/content/docs/design/metadata.md @@ -49,7 +49,8 @@ SidecarV1 { session_id: UUIDv7, // geolocation (see Geolocation below) - gps: Option<{ lat: f64, lon: f64, source: GpsSource }>, + gps: Option<{ lat: f64, lon: f64, source: GpsSource, + datum: GpsDatum /* wire-absent ⇒ wgs84 */ }>, // provenance binding — the PRIOR chain head; see Provenance Binding and Sealing Order below provenance_chain_hash: Option<[u8; 32]>, // hash of the provenance record PRECEDING the write that seals this @@ -73,12 +74,13 @@ SidecarV1 { ### Closed Enum Value Sets -Two sidecar fields are closed enums whose authoritative value sets live here (the blanket closed-enum rule is [Threat Model — Schema Rules](/design/threat-model/schema-rules/); the code mirror is a closed Rust enum in `capsule-core::domain`, and adding a value requires a new, later-dated `protocol_version`): +Three sidecar fields are closed enums whose authoritative value sets live here (the blanket closed-enum rule is [Threat Model — Schema Rules](/design/threat-model/schema-rules/); the code mirror is a closed Rust enum in `capsule-core::domain`, and adding a value requires a new, later-dated `protocol_version`): - **`content_type`** — MIME syntax, exactly **one canonical value per format** (never an alias like `image/jpg`). The v1 set: - images: `image/jpeg`, `image/png`, `image/webp`, `image/gif`, `image/tiff`, `image/heic`, `image/avif`, `image/jxl`, `image/x-adobe-dng` - video: `video/mp4`, `video/quicktime`, `video/x-matroska`, `video/webm` - **`gps.source` (`GpsSource`)** — `exif` (written by the capturing device), `manual` (set by the user), `inferred` (client-derived, e.g. from a paired device's location or an ML suggestion). An `inferred` value is written to the canonical `gps` field only on **explicit user confirmation** — the same promotion rule as `tags_ai` → `tags_user`, so an automated guess can never silently overwrite capture truth. +- **`gps.datum` (`GpsDatum`)** — `wgs84 | gcj02`. The coordinate is stored **verbatim in the datum the source supplied**, never converted at rest: GCJ-02 → WGS-84 has no exact inverse, so converting on input would destroy the user's ground truth (the raw-input-is-truth principle). **BD-09 is never a storable datum** — BD-09 input is folded to GCJ-02 at the input edge (that transform is closed-form and exact) and stored as `datum = gcj02`. The field is **wire-absent when `wgs84`**, so every existing sidecar and known-answer vector stays byte-identical; it is an additive optional key within sidecar schema v1 (older v1 readers preserve-and-ignore it per the [request-side Postel rule](/design/threat-model/schema-rules/), no `sidecar_schema` bump — if the implementing slice finds the nested `gps` decoder strict rather than tolerant, its documented fallback is a schema bump). The value set is closed: a third datum requires a new `protocol_version`. `gps` is a single atomic value under CRDT merge — `datum` travels with `lat`/`lon` in one write, so no merge rule changes. ### Canonical CBOR Encoding @@ -206,7 +208,7 @@ The same dual-namespace structure applies to any future ML-derived metadata fiel ## Geolocation -GPS is stored canonically in **WGS-84** (`gps.lat` / `gps.lon`), the near-universal camera format. Some jurisdictions mandate obfuscated coordinates for display — notably China's **GCJ-02**, and Baidu's **BD-09** (a second obfuscation layer over GCJ-02). Capsule always stores WGS-84 and converts to the required system **deterministically and client-side** (in `capsule-core`) at plot time; the stored coordinate is never the obfuscated one. Per-platform map-provider selection is a client/deployment concern, not part of this schema. +GPS is stored in **the coordinate datum the source supplied**, tagged by [`gps.datum`](#closed-enum-value-sets) — WGS-84 (the near-universal camera format, and the wire-absent default) or GCJ-02 (China's legally mandated obfuscated datum, which user-entered coordinates from Chinese maps arrive in). The stored value is never converted at rest; conversion between datums for display or search happens **deterministically and client-side** (in `capsule-core`, via the in-house `geocoordinates-rs` library gated in the repo-root `SLICES.md`), with the lossy GCJ-02 → WGS-84 inverse marked approximate wherever it surfaces. Baidu's **BD-09** (a second obfuscation layer over GCJ-02) exists only at the input edge: it is folded exactly to GCJ-02 on entry and never stored. Per-platform map-provider selection is a client/deployment concern, not part of this schema. Implementation is slice `S-A7`. ## Validation @@ -219,6 +221,7 @@ The sidecar schema is the contract; validation focuses on serde determinism + CR - **Add-id rejection (unit).** Issue a remove with an `add_id` never observed locally; assert rejection (not silent no-op). - **LWW with superseded capture (unit).** Two devices write captions within milliseconds; merge; assert the winner is the lexicographic-tiebreak chosen, and the loser appears in `superseded_captions`. - **Privacy-on-export stripping (unit).** Each row of the privacy table is a fixture test: assert the field is stripped by default, retained when opt-in is set, and that the local sidecar is unchanged either way. +- **Datum verbatim storage (unit).** A GCJ-02 input round-trips unconverted with `datum = gcj02`; a BD-09 input asserts the exact fold to GCJ-02; a WGS-84 write asserts `datum` is wire-absent and the encoded sidecar is byte-identical to the pre-`datum` vector. - **Local–server metadata equivalence (unit).** Seal a sidecar into a metadata blob; assert that decrypting it is byte-identical to the signed sidecar and that the blob's content hash equals the manifest's `metadata_blob_hash`. Mutate the local sidecar by one byte; assert the round-trip check rejects it rather than persisting a divergent copy. - **Concurrent-edit reconciliation (smoke).** Two test clients edit the same album offline; merge over MLS; assert convergence with no manual conflict resolution needed. From e4e6c0ec0b18d4c3efaf022c1f36b1aa446e5486 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:10:17 -0400 Subject: [PATCH 52/58] docs(design): declare the generic lifecycle-write endpoint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validation invariants 16-18 (non-upload writes) had no owning server surface: no api-surfaces transport row and no slice. Authorization now owns POST /albums/{album_id}/ops — one generic signed-op endpoint for delete/metadata-update/derivative-add-replace/trash-restore, enforcing the envelope gate + invariants 16-18 + 25 before any write, content-hash replay idempotency, and provenance + sync_seq in one transaction. A new action is a manifest-schema change, never a new endpoint. Indexed as slice S-C16 (owns E2E case 7's server half with S-C11). --- SLICES.md | 22 +++++++++++++++++++ .../src/content/docs/design/api-surfaces.md | 1 + .../src/content/docs/design/authorization.md | 13 +++++++++++ .../src/content/docs/design/module-map.md | 3 ++- .../docs/design/threat-model/validation.md | 2 ++ 5 files changed, 40 insertions(+), 1 deletion(-) diff --git a/SLICES.md b/SLICES.md index 2f055aab..c5bda0f2 100644 --- a/SLICES.md +++ b/SLICES.md @@ -86,6 +86,7 @@ its slice. | S-C13 | Session device-cohort storage + grouping | server | — | S | ready | | S-C14 | Server integrity scrub (Postgres⇄blob-store) | server | S-C1 | M | ready | | S-C15 | Custody receipts + signed storage attestation | server | S-C1, S-C3 | M | ready | +| S-C16 | Generic lifecycle-write endpoint (`/albums/{id}/ops`) | server | S-C1 | M | ready | | S-D1 | SDK upload client (hand-written, stateful protocol) | sdk/clients | S-C1 | M | ready | | S-D2 | SDK sync/download client + connection-class budget | sdk/clients | S-C2, S-C9 | L | ready | | S-D3 | Web guest drop client (WASM) | sdk/clients | S-A6, S-C5 | L | ready | @@ -151,6 +152,7 @@ graph LR C3 --> C15 C15 --> D4 C1 --> C14[S-C14 integrity scrub] + C1 --> C16[S-C16 lifecycle writes] D1 --> B3 D1 --> D5 D2 --> E3 @@ -518,6 +520,26 @@ pass until the gate lifts. against testcontainer Postgres + a real blob tree; clean-store idempotency holds. - **Tier:** Unit + Smoke. +### S-C16 — Generic lifecycle-write endpoint + +- **Contract:** [Authorization — The Lifecycle Write Surface](capsule-docs/src/content/docs/design/authorization.md), + [Validation invariants 16–18 + 25](capsule-docs/src/content/docs/design/threat-model/validation.md), + [API Surfaces — transport row](capsule-docs/src/content/docs/design/api-surfaces.md). +- **Deliverable:** `POST /albums/{album_id}/ops` in `capsule-api-upload::ops` — the + signed manifest bundle (opaque canonical-CBOR manifest + encrypted metadata blob + when the action carries one) through S-C1's `EnvelopeGate` before any write; + invariants 16 (closed action set), 17 (`prior_provenance_hash` chain match, + `409` stale-revival), 18 (monotonic + MLS-attested `amk_version`), and 25 + (metadata-blob hash binding) each rejecting with its `error.*` code; content-hash + replay idempotency returning the byte-identical prior response; provenance append + + per-album `sync_seq` mint in one transaction (the sync feed's finalization rule). +- **Depends on:** S-C1 (envelope gate + finalization transaction shape). +- **Done when:** invariants 16/17/18 each have a rejecting test (status **and** + `error.*` code) against testcontainer Postgres; the replay test returns + byte-identical responses; a delete → restore round-trip smoke passes and appears on + the sync feed in order. +- **Tier:** Unit + Smoke + E2E case 7 (with S-C11, S-D2). + ## Lane D — SDK / clients `capsule-sdk` is the **sanctioned network path**: it owns the session/token store and diff --git a/capsule-docs/src/content/docs/design/api-surfaces.md b/capsule-docs/src/content/docs/design/api-surfaces.md index ccbdf887..a9663d09 100644 --- a/capsule-docs/src/content/docs/design/api-surfaces.md +++ b/capsule-docs/src/content/docs/design/api-surfaces.md @@ -12,6 +12,7 @@ Capsule's server exposes exactly **two** wire transports, chosen per surface: ** | --------------------------------------------------------- | -------------------------------------------- | ------------------------------------ | ----------------------------------------------------------------------- | | Authentication (sessions, passkeys, TOTP, OIDC) | REST | `capsule-api-auth` | [Authentication](/design/authentication/) | | Resumable upload (`POST/HEAD/PATCH /upload`) | REST | `capsule-api-upload` | [Import — Upload Protocol](/design/import/upload-protocol/) | +| Lifecycle writes (`POST /albums/{album_id}/ops`) | REST | `capsule-api-upload::ops` (planned) | [Authorization](/design/authorization/#the-lifecycle-write-surface) | | Blob fetch (`GET /blob/{hash}`, HTTP `Range`) | REST | `capsule-api-media` | [Import — Download & Sync](/design/import/download-sync/) | | Share serving (`/s/{opaque-id}`) | REST | `capsule-api-media::shares` (planned) | [Share Links](/design/share-links/) | | Guest drops (`/u/{opaque-id}/drop`, inbox, adoption) | REST | `capsule-api-media::drops` (planned) | [Web Upload](/design/web-upload/) | diff --git a/capsule-docs/src/content/docs/design/authorization.md b/capsule-docs/src/content/docs/design/authorization.md index 71bca1b0..e806b483 100644 --- a/capsule-docs/src/content/docs/design/authorization.md +++ b/capsule-docs/src/content/docs/design/authorization.md @@ -35,6 +35,19 @@ Authorization is established exactly as for a write: A `delete` or `replace` is therefore authorized by the same proof as the original `create`: there is no weaker path to destroy data than to add it. Similarly, a `derivative-replace` is authorized as strongly as the original `derivative-add` — a buggy client cannot quietly poison a thumbnail. +## The Lifecycle Write Surface + +Every non-upload lifecycle write — `delete`, `metadata-update`, `derivative-add`/`derivative-replace` when the manifest references already-stored blobs, and `trash-restore` — travels through **one generic REST endpoint**: `POST /albums/{album_id}/ops`. (`create` and `replace`, and any derivative action carrying new bytes, ride the [upload protocol](/design/import/upload-protocol/) instead — a write that moves blob bytes is an upload by definition.) The body is the signed manifest bundle: the opaque canonical-CBOR manifest plus the encrypted metadata blob when the action carries one. + +The endpoint is deliberately singular — one closed enum, one gate, one transaction shape: + +- The structural envelope check above, [invariants 16–18](/design/threat-model/validation/#server-side-validation-invariants) and the metadata-blob binding ([invariant 25](/design/threat-model/validation/#server-side-validation-invariants)), runs **before any row is written**, for every action uniformly. A new action is a manifest-schema change under a new `protocol_version`, never a new endpoint. +- Accepted ops are **idempotent by content hash**: replaying a manifest whose content hash the server has already accepted returns the byte-identical prior response and writes nothing. +- The accepted op appends the provenance record and mints the per-album `sync_seq` in **one transaction**, the same finalization rule the [sync feed](/design/import/download-sync/) relies on — an op is visible on the feed exactly when it is durable. +- Rejections carry an [`error.*` code](/design/i18n/#server-error-codes) and write nothing; the rejection itself is logged. + +The transport row lives in [API Surfaces](/design/api-surfaces/#surface--transport-map). Implemented in `capsule-api-upload::ops` (planned — slice `S-C16`, reusing the upload server's envelope gate). + ## The Server Executes But Never Authorizes Per the principle of [trusting the server for storage, never for authorization](/design/cryptography/), the server **carries out** a remote delete or replace but is **never** the authority that permits it. A server-asserted lifecycle change with no valid write-tier signature is rejected by every client. This bounds the damage a compromised or buggy server can do: it can refuse to store data, but it cannot forge its destruction. diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 4b08e4cd..53cbc615 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -83,6 +83,7 @@ Hardware-key adapters do **not** live in `capsule-sdk`: the `Signer`/`HardwareSi | `capsule-api-library::schema::*` (legacy, retiring) | [API Surfaces](/design/api-surfaces/#legacy-graphql-retiring) | frozen (no new surface) | | `capsule-api-library::loaders` (legacy, retiring) | [API Surfaces](/design/api-surfaces/#legacy-graphql-retiring) | frozen (no new surface) | | `capsule-api-upload` | [Import — Upload Protocol](/design/import/upload-protocol/) | Unit + Smoke + 1 E2E | +| `capsule-api-upload::ops` (planned) | [Authorization](/design/authorization/#the-lifecycle-write-surface), [Threat Model — Validation](/design/threat-model/validation/) | Unit + Smoke (+ E2E case 7) | | `capsule-api-media::routes` | [Filesystem — Server](/design/filesystem/server/), [Thumbnails](/design/thumbnails/) | Smoke | | `capsule-api-media::verify` (planned) | [Import — Storage Verification](/design/import/storage-verification/) | Unit + Smoke | | `capsule-api-media::shares` (planned) | [Share Links](/design/share-links/) | Unit + Smoke | @@ -124,7 +125,7 @@ Navigation from a design doc back to where the code lives. | [MLS Resilience](/design/mls-resilience/) | `capsule-core::crypto::mls` (extends main MLS module; blocked with it) | | [Device Enrollment](/design/device-enrollment/) | `capsule-core::crypto::keys`, `capsule-api-auth::devices` | | [Authentication](/design/authentication/) | `capsule-api-auth::{oidc,session,claims}` | -| [Authorization](/design/authorization/) | `capsule-api-auth::roles`, `capsule-core::crypto::provenance` (verify_asset) | +| [Authorization](/design/authorization/) | `capsule-api-auth::roles`, `capsule-core::crypto::provenance` (verify_asset), `capsule-api-upload::ops` (lifecycle-write surface, planned) | | [Clients](/design/clients/) | `capsule-sdk` + per-platform native code | | [Internationalization](/design/i18n/) | `capsule-i18n` (runtime) + `xtask::i18n` (codegen) + `locales/` source + per-platform generated catalogs | | [Versioning](/design/versioning/) | Cross-cutting: `capsule-api` (header enforcement), `capsule-core::crypto::mls` (upgrade ceremony), `capsule-api-migration` | diff --git a/capsule-docs/src/content/docs/design/threat-model/validation.md b/capsule-docs/src/content/docs/design/threat-model/validation.md index 03741564..d19342a0 100644 --- a/capsule-docs/src/content/docs/design/threat-model/validation.md +++ b/capsule-docs/src/content/docs/design/threat-model/validation.md @@ -44,6 +44,8 @@ Session TTL, the ≥ 1-hour survival floor, and pressure-discard semantics are s ### On non-upload writes (lifecycle action manifest, metadata-update, derivative-add/replace, trash-restore) +These checks run at the single lifecycle-write surface, `POST /albums/{album_id}/ops`, owned by [Authorization — The Lifecycle Write Surface](/design/authorization/#the-lifecycle-write-surface) (transport row in [API Surfaces](/design/api-surfaces/#surface--transport-map); slice `S-C16`). + - **16.** `action` is in the closed enum. Otherwise `400`. - **17.** `prior_provenance_hash` equals the last accepted manifest's content hash for this `asset_id`. Otherwise `409` (stale-revival). - **18.** `amk_version` is monotonic per album (never regresses) **and within the range the album's admin-signed MLS commit chain attests**. The server's stored counter is a structural backstop; the authoritative ceiling is MLS, so a server cannot fabricate a future epoch a client will honor — see [Write Authorization](/design/cryptography/keys/#write-authorization). Otherwise `400`. From 142619755b8daf647478053cb90f793739331070 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:11:56 -0400 Subject: [PATCH 53/58] docs(design): contract third-party importers in the import pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the design half of issue #296: an importer is a source adapter on the scanner's contract — structure-aware metadata folding (embedded EXIF wins except for exporter-authoritative constructs) at extraction, pure planner untouched, idempotent re-runs. Google Takeout is the v1 adapter (S-B6); iCloud and Immich are post-v1 (S-B7/S-B8); user-facing migration guides live outside design/ and gate on importer stabilization (S-Z2). --- SLICES.md | 54 +++++++++++++++++++ .../content/docs/design/import/pipeline.md | 15 ++++++ .../src/content/docs/design/module-map.md | 2 +- 3 files changed, 70 insertions(+), 1 deletion(-) diff --git a/SLICES.md b/SLICES.md index c5bda0f2..fdcccae2 100644 --- a/SLICES.md +++ b/SLICES.md @@ -71,6 +71,9 @@ its slice. | S-B2 | Signed-path import-executor rewrite | media/import | S-B1 | L | ready | | S-B3 | Streaming import (probe, `total_size`, drive mode) | media/import | S-D1, S-D4 | L | ready | | S-B4 | Staged uploads (low-data tier ladder) | media/import | S-C1, S-C2, S-D1 | M | ready | +| S-B6 | Google Takeout importer | media/import | S-B2 | M | ready | +| S-B7 | iCloud export importer | media/import | S-B6 | M | post-v1 | +| S-B8 | Immich importer | media/import | S-B6 | M | post-v1 | | S-C1 | Upload-server hardening (envelope gate + invariants) | server | — | L | ready | | S-C2 | Key-free sync feed | server | S-C1 | L | ready | | S-C3 | Storage-verification endpoint | server | — | M | ready | @@ -124,6 +127,7 @@ its slice. | S-X2 | MLS membership + Welcome/history delivery | blocked-external | S-X1 | L | blocked | | S-X3 | Album upgrade ceremony + MLS resilience | blocked-external | S-X2 | L | blocked | | S-Z1 | Library-settings document schema (design) | design | — | S | ready | +| S-Z2 | Provider migration user guides (docs site) | design | S-B6 | S | ready | Lanes are independent by construction; within a lane, "Depends on" is the only ordering. `blocked` = a dependency (or an upstream project) gates the start, not review priority. @@ -157,6 +161,9 @@ graph LR D1 --> D5 D2 --> E3 B1[S-B1 thumbnails] --> B2[S-B2 executor rewrite] --> G4[S-G4 executor retire] + B2 --> B6[S-B6 takeout] --> B7[S-B7 icloud] + B6 --> B8[S-B8 immich] + B6 --> Z2[S-Z2 migration guides] H1[S-H1 embeddings] --> H2[S-H2 registry] H1 --> H3[S-H3 semantic/face] --> G3 X1[S-X1 openmls] --> X2[S-X2 membership] --> X3[S-X3 upgrade ceremony] @@ -340,6 +347,37 @@ pass until the gate lifts. resume-from-server-truth, staged×streaming exclusion). - **Tier:** Unit + Smoke. +### S-B6 — Google Takeout importer + +- **Contract:** [Import — Third-Party Importers](capsule-docs/src/content/docs/design/import/pipeline.md). +- **Deliverable:** the Takeout source adapter in + `capsule-core::import::importers::takeout` (and the adapter trait it defines, + shared by S-B7/S-B8/S-B9): archive walk, JSON-sidecar pairing (taken-time, GPS, + description, favorites, album JSONs), the EXIF-over-exporter precedence fold at + extraction, and the known Takeout quirks (truncated filenames, `(1)` duplicates, + edited/original pairs, split archives) as fixture-covered adapter concerns. The + planner and executor are untouched. +- **Depends on:** S-B2 (imports must land on the signed path). **Blocks:** S-B7, + S-B8, S-Z2. +- **Done when:** the pipeline doc's Takeout mapping-table Validation bullet passes; + a fixture-archive import is deterministic across runs and skips completed work on + re-run. +- **Tier:** Unit (mapping table, determinism) + Smoke (end-to-end archive import). + +### S-B7 — iCloud export importer (post-v1) + +- **Contract:** [Import — Third-Party Importers](capsule-docs/src/content/docs/design/import/pipeline.md). +- **Deliverable:** the iCloud Photos export adapter (originals + CSV metadata) on the + S-B6 adapter trait. **Depends on:** S-B6. **Status: post-v1** — indexed so the + contract has an owner. **Tier:** Unit + Smoke. + +### S-B8 — Immich importer (post-v1) + +- **Contract:** [Import — Third-Party Importers](capsule-docs/src/content/docs/design/import/pipeline.md). +- **Deliverable:** the Immich adapter (export/API surface fixed when the slice + starts) on the S-B6 adapter trait. **Depends on:** S-B6. **Status: post-v1**. +- **Tier:** Unit + Smoke. + ## Lane C — server (key-free surfaces) ### S-C1 — Upload-server hardening @@ -973,3 +1011,19 @@ its design here. a design-doc addition. The scope-override map's rows and grammar are now specified in [Organization — Scope Grammar](capsule-docs/src/content/docs/design/organization.md); what remains is the smart-album predicate schema and the document's envelope/versioning. + +### S-Z2 — Provider migration user guides + +- **Contract:** [Import — Third-Party Importers](capsule-docs/src/content/docs/design/import/pipeline.md) + (the guides describe shipped importer behavior only); issue #296's requirements. +- **Deliverable:** user-facing migration guides under + `capsule-docs/src/content/docs/guides/` — **outside `design/`**, non-normative — + one per provider as its importer lands (Google Photos first, with S-B6): export + walkthrough, import steps, an end-to-end verification checklist (counts, spot + hashes, metadata sampling), and the robustness disclaimer (transfer, verify + everything end-to-end, run both systems in parallel for a period, follow + deployment best practices). +- **Depends on:** S-B6 (guides ship only for stabilized importers). +- **Done when:** the Google Photos guide is published and its steps round-trip + against a real Takeout archive; `mise run check-docs` green. +- **Tier:** docs build. diff --git a/capsule-docs/src/content/docs/design/import/pipeline.md b/capsule-docs/src/content/docs/design/import/pipeline.md index 6eaae669..108722f5 100644 --- a/capsule-docs/src/content/docs/design/import/pipeline.md +++ b/capsule-docs/src/content/docs/design/import/pipeline.md @@ -75,6 +75,20 @@ Peak local disk is therefore bounded to the in-flight window, not the whole impo **Quota.** Streaming creates one upload session per asset, so server [quota](/design/quota/#enforcement-points) is enforced exactly as for any upload — at session creation, no new enforcement point. If quota is exhausted mid-stream the next session creation is refused; the pipeline pauses and surfaces the remediable state rather than failing the whole import, and no local original is released for an asset that did not upload. +## Third-Party Importers + +**Status: planned** (slices `S-B6` v1, `S-B7`/`S-B8` post-v1). Migration from another photo service is an import, not a new pipeline: an importer is a **source adapter** implementing the same contract as the filesystem scanner — it yields `(bytes-source, extracted-metadata)` entries into [Scan & Extract](#scan--extract), and the pure planner and executor are unchanged. Third-party migration adds no new pipeline stage and no new server surface. + +What an adapter owns is **structure awareness**: metadata the exporting service carries out-of-band is folded into the extracted metadata *before* planning, under a fixed precedence — **embedded EXIF wins over exporter-side records, except for constructs the exporter is authoritative for** (its album membership, favorites/rating flags, and user-typed descriptions that the file bytes never carried). The fold happens at extraction, so the planner's determinism contract holds: a given archive yields the same `ImportPlan` on every run, and re-running an importer over the same archive skips completed work exactly like a [resumed import](#validation). + +The committed adapters: + +- **Google Takeout (`S-B6`, v1).** Walks a Takeout archive; pairs each media file with its JSON sidecar (photo-taken time, GPS, description, favorites, album JSONs); folds per the precedence rule. Takeout's known quirks — truncated filenames, `(1)` duplicates, edited/original pairs, JSON files split across archive parts — are adapter concerns with fixture coverage, never planner special cases. +- **iCloud Photos export (`S-B7`, post-v1).** The iCloud export layout (originals + CSV metadata). +- **Immich (`S-B8`, post-v1).** Immich's export/API surface; format fixed when the slice starts. + +Tethered cameras plug into this same adapter seam — see [Import — Camera Import](/design/import/camera-import/). User-facing migration guides (per-provider export walkthroughs, verification checklists, robustness disclaimers) live in the docs site outside `design/` and ship gated on each importer stabilizing (slice `S-Z2`). + ## Upload Prioritization When many files are processed simultaneously, the order they are *started* is decided by these heuristics: @@ -105,5 +119,6 @@ What the rest of the system depends on this module for: - **Streaming auto-detect (unit).** With `available_bytes()` mocked below and above `total_size + headroom`, assert `streaming_recommended` is set in the constrained case and clear otherwise. - **Streaming release gating (smoke).** Run a streaming import with `/storage/verify` mocked: assert each local original (and Move-mode source) is released *only* after its `durable` verdict, and that a non-`durable` verdict leaves the local copy in place. - **Streaming halt-on-disconnect (smoke).** Drop the server connection mid-stream; assert the pipeline stops admitting new source files into the library (no unbounded local growth) and resumes uploading via `HEAD` on reconnect without re-importing completed assets. +- **Takeout mapping table (unit).** A fixture Takeout archive → expected plan, one row per metadata-mapping rule (taken-time vs EXIF precedence, GPS fold, description, favorites, album membership, truncated-filename pairing, duplicate suffixes). Re-running over the same fixture yields a plan that skips completed work. The cross-module case — pipeline → upload protocol → server finalization → assets visible in the sync feed — is bounded E2E surface listed in [Module Map](/design/module-map/#e2e-test-surface). diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 53cbc615..58b123f7 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -134,7 +134,7 @@ Navigation from a design doc back to where the code lives. | [Filesystem — Server](/design/filesystem/server/) | `capsule-api`, `capsule-api-entity`, blob store glue | | [Filesystem — Client](/design/filesystem/client/) | `capsule-core::{library,db}`, per-platform native code | | [Filesystem — Maintenance](/design/filesystem/maintenance/) | `capsule-core::library::{scrub,rebuild,trash}`, server-side scrub in `capsule-api-upload` | -| [Import — Pipeline](/design/import/pipeline/) | `capsule-core::import::*`; the streaming import-upload loop and the `capsule-core::library::available_bytes` free-space probe are planned | +| [Import — Pipeline](/design/import/pipeline/) | `capsule-core::import::*`; the streaming import-upload loop, the `capsule-core::library::available_bytes` free-space probe, and the `import::importers` third-party source adapters are planned | | [Import — Upload Protocol](/design/import/upload-protocol/) | `capsule-sdk::upload` (client) + `capsule-api-upload` (server) | | [Import — Download & Sync](/design/import/download-sync/) | `capsule-sdk` (client) + `capsule-api-sync` (server) | | [Import — Storage Verification](/design/import/storage-verification/) | `capsule-api-media::verify` (route) + `capsule-sdk` (client) + `capsule-core` (verify-before-destroy predicate) | From 1be966bb1bb7565f98d2d9f664838c26bb53c0f8 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:13:06 -0400 Subject: [PATCH 54/58] docs(design): specify tethered camera import over PTP/IP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the design half of issue #365: a connected camera is a source adapter on the pipeline's scanner contract, spoken through ptpip-rs — a new in-house portable Rust PTP/IP library gate (libgphoto2 is ruled out for portability and licensing). Deterministic handle enumeration keeps the planner pure; hash-on-receipt is the end-to-end integrity story (PTP/IP has none of its own); dedupe-on-reimport falls out of the planner's content-hash rule; camera storage is read-only. Post-v1, indexed as slice S-B9; rides E2E case 2 once live. --- SLICES.md | 17 ++++++++++ capsule-docs/astro.config.mjs | 1 + .../docs/design/import/camera-import.md | 32 +++++++++++++++++++ .../src/content/docs/design/module-map.md | 1 + .../src/content/docs/design/principles.md | 1 + 5 files changed, 52 insertions(+) create mode 100644 capsule-docs/src/content/docs/design/import/camera-import.md diff --git a/SLICES.md b/SLICES.md index fdcccae2..081d1b44 100644 --- a/SLICES.md +++ b/SLICES.md @@ -74,6 +74,7 @@ its slice. | S-B6 | Google Takeout importer | media/import | S-B2 | M | ready | | S-B7 | iCloud export importer | media/import | S-B6 | M | post-v1 | | S-B8 | Immich importer | media/import | S-B6 | M | post-v1 | +| S-B9 | Tethered camera import (PTP/IP) | media/import | S-B2, ptpip-rs | L | post-v1 | | S-C1 | Upload-server hardening (envelope gate + invariants) | server | — | L | ready | | S-C2 | Key-free sync feed | server | S-C1 | L | ready | | S-C3 | Storage-verification endpoint | server | — | M | ready | @@ -164,6 +165,7 @@ graph LR B2 --> B6[S-B6 takeout] --> B7[S-B7 icloud] B6 --> B8[S-B8 immich] B6 --> Z2[S-Z2 migration guides] + B2 --> B9[S-B9 camera import] H1[S-H1 embeddings] --> H2[S-H2 registry] H1 --> H3[S-H3 semantic/face] --> G3 X1[S-X1 openmls] --> X2[S-X2 membership] --> X3[S-X3 upgrade ceremony] @@ -193,6 +195,7 @@ pass until the gate lifts. | `rawshift` (in-house RAW decode; git submodule, alpha, consumed by nothing yet) | stabilizing | Full RAW support in S-B1/S-B2. v1 ships the zune-jpeg format set; the `media::image::formats::raw` stub is the integration point. | | `spargen` (in-house OpenAPI **3.1** client generator) | in development | S-D8 (typed REST client + `AuthenticatedClient` revival). Progenitor is gone — we do not downgrade schemas to 3.0. | | `geocoordinates-rs` (in-house WGS-84 ↔ GCJ-02/BD-09 conversions) | planned | The deterministic client-side coordinate conversion named in [Metadata — Geolocation](capsule-docs/src/content/docs/design/metadata.md); consumed by map display, S-H3 geo features, and S-A7's exact BD-09→GCJ-02 input fold. Until it lands, verbatim datum storage is unaffected (display conversion and the BD-09 fold are the gated pieces). | +| `ptpip-rs` (in-house PTP/IP camera protocol; repo not yet created) | planned | S-B9 (post-v1). Portable Rust PTP/IP (ISO 15740 over TCP/IP) with a vendor-extension seam (Sony first) — see [Import — Camera Import](capsule-docs/src/content/docs/design/import/camera-import.md). | | `openmls` X-Wing/SHA-512 ciphersuite (codepoint pending) | blocked upstream | S-X1 → S-X2 → S-X3 (tracked in Lane X). | ## Lane A — core crypto @@ -378,6 +381,20 @@ pass until the gate lifts. starts) on the S-B6 adapter trait. **Depends on:** S-B6. **Status: post-v1**. - **Tier:** Unit + Smoke. +### S-B9 — Tethered camera import (post-v1) + +- **Contract:** [Import — Camera Import](capsule-docs/src/content/docs/design/import/camera-import.md). +- **Deliverable:** the PTP/IP source adapter in `capsule-core::import::camera` over + the gated in-house `ptpip-rs` crate — deterministic handle enumeration, + hash-on-receipt integrity, per-object resume, read-only camera storage, mDNS + + manual discovery/pairing — feeding the unmodified pipeline; Sony extension quirks + stay behind the crate. +- **Depends on:** S-B2 (signed path) + the `ptpip-rs` library gate. +- **Status: post-v1** — indexed so the adapter seam has an owner now. +- **Done when:** the camera-import doc's mock-responder unit suite passes; the bench + smoke pulls a real card's worth from hardware. **Tier:** Unit + Smoke (bench + hardware lane); rides E2E case 2 once live. + ## Lane C — server (key-free surfaces) ### S-C1 — Upload-server hardening diff --git a/capsule-docs/astro.config.mjs b/capsule-docs/astro.config.mjs index 2fb1f45c..ff795f15 100644 --- a/capsule-docs/astro.config.mjs +++ b/capsule-docs/astro.config.mjs @@ -102,6 +102,7 @@ export default defineConfig({ { slug: 'design/import/upload-protocol' }, { slug: 'design/import/download-sync' }, { slug: 'design/import/storage-verification' }, + { slug: 'design/import/camera-import' }, { slug: 'design/backup-recovery' }, { slug: 'design/versioning' }, ], diff --git a/capsule-docs/src/content/docs/design/import/camera-import.md b/capsule-docs/src/content/docs/design/import/camera-import.md new file mode 100644 index 00000000..e5e8360f --- /dev/null +++ b/capsule-docs/src/content/docs/design/import/camera-import.md @@ -0,0 +1,32 @@ +--- +title: Import — Camera Import +description: Tethered and wireless in-camera import over PTP/IP +status: draft +--- + +**Status: planned, post-v1** (slice `S-B9`, gated on the in-house `ptpip-rs` library — see the repo-root `SLICES.md`). This doc is the owner contract for tethered/wireless camera ingestion so the seam is fixed now; no code exists. + +Professional cameras expose their storage over **PTP/IP (ISO 15740 over TCP/IP)**. Capsule speaks it through `ptpip-rs`, an in-house Rust implementation (a library gate like `rawshift` and `spargen`) — portable across every client platform including mobile, which rules out binding libgphoto2 (C, LGPL, desktop-oriented). Vendor extensions (Sony first, per issue #365) live behind the crate; Capsule consumes only the adapter surface below. + +## The Camera Is a Source Adapter + +A connected camera is an import **source adapter** on the same contract as the filesystem scanner and the [third-party importers](/design/import/pipeline/#third-party-importers): it enumerates media into [Scan & Extract](/design/import/pipeline/#scan--extract), and everything downstream — plan, confirm, execute, upload — is the unmodified [import pipeline](/design/import/pipeline/). Camera import adds no pipeline stage and no server surface. + +- **Lives in:** `capsule-core::import::camera` (planned, post-v1) over the gated `ptpip-rs` crate. +- **Public surface:** the shared source-adapter trait; a discovery/pairing surface for clients (list cameras, connect, session state). +- **Defers to:** [Import — Pipeline](/design/import/pipeline/) (planner/executor), [Metadata](/design/metadata/) (extraction schema), [Import — Upload Protocol](/design/import/upload-protocol/) (upload), and the pipeline's dedup rules (below). + +## Contract + +- **Deterministic enumeration.** PTP object handles are enumerated in a fixed order — by storage id, then object handle id, with capture date as the tiebreaker — so the pure planner's determinism contract holds: the same camera state yields the same `ImportPlan`. +- **Dedupe on re-import is by content, not by source.** The planner's existing content-hash dedup is what prevents a re-connected card or camera from re-importing prior assets; the adapter does not track "seen" state of its own. Pulling the same card out of the camera and importing it over the filesystem scanner dedups identically. +- **Integrity end-to-end.** PTP/IP carries no checksums of its own; TCP integrity is treated as transport-grade only. Every transferred object is hashed on receipt, and that hash *is* the content hash the [manifest](/design/cryptography/provenance/#asset-manifest) commits to — a corrupted transfer surfaces as a hash mismatch at import, before anything is signed. +- **Resumable per object.** A dropped connection resumes at the object boundary (partial-object ranged transfer where the camera supports `GetPartialObject`); completed objects are never re-pulled (dedup covers the crash window). +- **Read-only by default.** The adapter never deletes or mutates camera storage. A delete-after-verified-import option (Sony's extension surface) is a client feature that may come later; it must gate on the same [verify-before-destroy](/design/import/storage-verification/#verify-before-destroy) rule as every destructive path. +- **Discovery and pairing.** mDNS (`_ptp._tcp`) where the camera advertises, manual IP entry otherwise; the PTP/IP pairing handshake (GUID + friendly name) is remembered per camera. Pairing state is client-local config, not library metadata. + +## Validation + +- **Unit** — the adapter against a mock PTP/IP responder: deterministic enumeration order, resume-at-object-boundary, hash-mismatch rejection, read-only invariant (no write/delete op ever issued in a standard import). +- **Smoke** — a bench lane against real hardware (Sony first), exercised where hardware CI allows, mirroring the hardware-signer smoke posture. +- **No new E2E case:** once live, camera import rides bounded E2E case 2 (full import + upload + finalize) with the camera adapter as the source — the [Module Map](/design/module-map/#e2e-test-surface) bound is unchanged. diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 58b123f7..7448e324 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -135,6 +135,7 @@ Navigation from a design doc back to where the code lives. | [Filesystem — Client](/design/filesystem/client/) | `capsule-core::{library,db}`, per-platform native code | | [Filesystem — Maintenance](/design/filesystem/maintenance/) | `capsule-core::library::{scrub,rebuild,trash}`, server-side scrub in `capsule-api-upload` | | [Import — Pipeline](/design/import/pipeline/) | `capsule-core::import::*`; the streaming import-upload loop, the `capsule-core::library::available_bytes` free-space probe, and the `import::importers` third-party source adapters are planned | +| [Import — Camera Import](/design/import/camera-import/) | `capsule-core::import::camera` (planned, post-v1) over the gated in-house `ptpip-rs` crate | | [Import — Upload Protocol](/design/import/upload-protocol/) | `capsule-sdk::upload` (client) + `capsule-api-upload` (server) | | [Import — Download & Sync](/design/import/download-sync/) | `capsule-sdk` (client) + `capsule-api-sync` (server) | | [Import — Storage Verification](/design/import/storage-verification/) | `capsule-api-media::verify` (route) + `capsule-sdk` (client) + `capsule-core` (verify-before-destroy predicate) | diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 8d489d00..bf1ae294 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -70,6 +70,7 @@ The owner docs are: | API surface ↔ transport map + cross-transport negotiation/rejection mapping | [API Surfaces](/design/api-surfaces/) | | Storage durability verdict + verify-before-destroy rule | [Import — Storage Verification](/design/import/storage-verification/) | | Canonical dependency + tooling pins (non-crypto libraries per platform) | [Dependencies](/design/dependencies/) | +| Tethered camera import (PTP/IP source adapter) | [Import — Camera Import](/design/import/camera-import/) | **Permitted secondary mentions.** Mechanism-explanatory phrasing inside a non-owner doc is fine — for example, "STREAM tags catch chunk reordering" inside [Peering](/design/peering/) is explaining a *behavior*, not declaring a *choice*. What the rule forbids is restating the choice itself ("we use SHA-256") outside the owner doc. When in doubt, link. From 8c359f052bbe33afbd55596ef9dfbbdfd5173ae6 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:14:36 -0400 Subject: [PATCH 55/58] docs(design): name the official language set and the README translation pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the design halves of issues #366 and #385: i18n.md now owns the committed twelve-language set (zh-Hans, zh-Hant, ja, ko, fr, de, es, pt-BR, it, ru, hi, ar — RTL in scope via Arabic; zh-Hant falls back to en, never zh-Hans) while locales/config.json stays the mechanical truth of what ships, and the structure-aware xtask translate-readme contract (glossary-pinned LLM translation, committed outputs, key-less structural drift check mirroring i18n --check). New Lane I indexes the work: S-I1 string migration, S-I2 locale rollout, S-I3 README pipeline. --- SLICES.md | 48 ++++++++++++++++++ capsule-docs/src/content/docs/design/i18n.md | 49 ++++++++++++++++++- .../src/content/docs/design/principles.md | 2 +- 3 files changed, 97 insertions(+), 2 deletions(-) diff --git a/SLICES.md b/SLICES.md index 081d1b44..ea523631 100644 --- a/SLICES.md +++ b/SLICES.md @@ -124,6 +124,9 @@ its slice. | S-H2 | Model registry + version regen | ML | S-H1 | M | ready | | S-H3 | Semantic/face features | ML | S-H1 | L | ready | | S-H4 | Group-scoped evaluations (best shot/framing/exposure) | ML | S-H3 | M | post-v1 | +| S-I1 | Hardcoded-string migration to catalog keys | i18n | — | M | ready | +| S-I2 | Official language-set rollout (12 locales + RTL) | i18n | — | L | ready | +| S-I3 | `xtask translate-readme` + CI drift check | i18n | S-I2 | M | ready | | S-X1 | OpenMLS backend → `OpenMlsAuthority` | blocked-external | upstream | L | blocked | | S-X2 | MLS membership + Welcome/history delivery | blocked-external | S-X1 | L | blocked | | S-X3 | Album upgrade ceremony + MLS resilience | blocked-external | S-X2 | L | blocked | @@ -182,6 +185,7 @@ graph LR E2 --> E4[S-E4 aggregated albums] D2 --> E4 H3 --> H4[S-H4 group evaluations] + I2[S-I2 locale rollout] --> I3[S-I3 readme translate] ``` ## In-House and External Library Gates @@ -989,6 +993,50 @@ its design here. - **Depends on:** S-H3 (and the stacking surfaces). **Status: post-v1** — indexed now so the sequencing contract has an owner; not part of the v1 cut. +## Lane I — i18n + +Catalog + client work over the [i18n contract](capsule-docs/src/content/docs/design/i18n.md); +no new server surface. The infrastructure (catalogs, codegen, `capsule-i18n` +runtime, error-code scheme) already ships — this lane is the content and rollout. + +### S-I1 — Hardcoded-string migration + +- **Contract:** [i18n — Canonical source](capsule-docs/src/content/docs/design/i18n.md); + the no-hardcoded-strings rule in `AGENTS.md`. +- **Deliverable:** every user-facing literal in web JSX, SwiftUI `Text`, and + Compose moved onto catalog keys (`locales/en.json` grows the keys; + `mise run i18n` regenerates the per-platform files), plus a per-platform + lint/grep gate that fails on new user-facing literals so the migration cannot + regress. +- **Done when:** the gate runs clean on all three surfaces; `mise run i18n-check` + green; the touched screens render from the catalogs. +- **Tier:** Unit/Smoke per platform. + +### S-I2 — Official language-set rollout + +- **Contract:** [i18n — Supported Languages](capsule-docs/src/content/docs/design/i18n.md). +- **Deliverable:** the twelve locales (`zh-Hans`, `zh-Hant`, `ja`, `ko`, `fr`, + `de`, `es`, `pt-BR`, `it`, `ru`, `hi`, `ar`) added to `locales/config.json` + + full catalogs (machine-seeded entries flagged for human review in the + translator `context` field); fallbacks direct-to-`en` (explicitly **no** + `zh-Hant → zh-Hans`); RTL support for `ar` (web `dir` attribute wiring, native + layout mirroring). +- **Done when:** `mise run i18n-check` green with thirteen catalogs carrying the + full key set; an RTL smoke renders the web app mirrored under `ar`. +- **Tier:** Unit + Smoke. **Blocks:** S-I3. + +### S-I3 — README translation pipeline + +- **Contract:** [i18n — README Translation](capsule-docs/src/content/docs/design/i18n.md). +- **Deliverable:** `xtask translate-readme` — block segmentation (code/links/badges + pass through), glossary-pinned LLM translation, committed `README..md` + with the do-not-edit banner — plus the key-less structural `--check` drift gate + in CI; languages mirror `locales/config.json`. +- **Depends on:** S-I2 (the locale list it mirrors). +- **Done when:** every non-source locale has a committed translation; mutating a + source segment makes `--check` fail; segmentation has golden tests. +- **Tier:** Unit (segmentation goldens) + Smoke. + ## Lane X — blocked on upstream ### S-X1 — OpenMLS backend → `OpenMlsAuthority` diff --git a/capsule-docs/src/content/docs/design/i18n.md b/capsule-docs/src/content/docs/design/i18n.md index 634fe8bb..83fe4ac1 100644 --- a/capsule-docs/src/content/docs/design/i18n.md +++ b/capsule-docs/src/content/docs/design/i18n.md @@ -51,6 +51,24 @@ iOS String Catalogs, and Android all compile it natively — so codegen to those targets is near-mechanical. The Rust runtime is the one target without a native ICU formatter and carries a small interpreter instead (see below). +## Supported Languages + +The committed official language set is **English (the source) plus twelve +translations**: `zh-Hans`, `zh-Hant`, `ja`, `ko`, `fr`, `de`, `es`, `pt-BR`, +`it`, `ru`, `hi`, and `ar`. Including Arabic puts **RTL layout support in scope +for every client** (web `dir` handling, native mirroring). Fallbacks are direct +to the source locale — in particular **`zh-Hant` falls back to `en`, never to +`zh-Hans`**: showing auto-Simplified text to Traditional-script readers is worse +than showing English. + +This section owns the *decision*; `locales/config.json` stays the **mechanical +truth of what ships** — a locale is supported in a build exactly when its catalog +lands and its tag enters `supportedLocales`. Today that is `en` only; the twelve +catalogs are the rollout slice `S-I2` in the repo-root `SLICES.md`. Everything +that enumerates languages (per-platform catalogs, [README +translations](#readme-translation)) derives from `config.json`, so the set is +never declared twice. + ## Codegen `mise run i18n` (`cargo run -p xtask -- i18n`) compiles the catalogs into: @@ -127,6 +145,30 @@ docs: `error.protocol.version_unsupported` (the `426` class), `error.quota.exceeded`, `error.moderation.account_suspended`, `error.auth.invalid_credentials`. +## README Translation + +The repo-root `README.md` ships translated as `README..md` for **every +non-source locale in `locales/config.json`** — the language list is never +declared outside that file. Generation is `xtask translate-readme` +(slice `S-I3`), structure-aware rather than whole-document: + +- The markdown is segmented by block (headings, paragraphs, list items, tables); + **code blocks, link targets, and badge/image URLs pass through untouched**. +- Each translatable segment goes through an LLM translation API with a pinned + project glossary (product terms — "Capsule", "sidecar", "album", "drop" — and + their per-language renderings), so terminology stays consistent across + regenerations. +- Output files are **committed** with the same do-not-edit banner as the other + generated i18n artifacts, and `xtask translate-readme --check` is the CI drift + gate: it re-segments the source and fails when a committed translation's + segment structure no longer matches `README.md` — the same pattern as + `i18n --check`. (The check is structural, not semantic — it needs no API key + in CI.) + +American English (`README.md`) is the base version; translations are +regenerated, never hand-edited (hand improvements go into the glossary or the +source). + ## Contributing translations Translators edit the JSON catalogs in `locales/` and open a pull request — no code @@ -157,7 +199,12 @@ a cross-module integration test. ## Future work -- Migrate existing hardcoded strings (web JSX, SwiftUI `Text`, Compose) onto catalog keys. +Items with a slice are indexed in the repo-root `SLICES.md`; the rest remain +unowned future work. + +- Hardcoded-string migration (web JSX, SwiftUI `Text`, Compose → catalog keys): **slice `S-I1`**. +- The twelve-locale catalog rollout + RTL support: **slice `S-I2`**. +- README translation pipeline: **slice `S-I3`** (see [README Translation](#readme-translation)). - Full ICU `plural`/`select`/`number`/`date` fidelity in the Rust runtime and the Android generator. - Wire the iOS `.xcstrings` target into the Xcode project; add the desktop target once its framework is chosen. - Retrofit the remaining server error variants with codes; regenerate the OpenAPI spec / SDK. diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index bf1ae294..3ff714fb 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -65,7 +65,7 @@ The owner docs are: | Moderation policy + federated reporting + blocklists | [Moderation](/design/moderation/) | | Quota accounting + enforcement points | [Quota](/design/quota/) | | Client validation duties + sandboxed decoder | [Clients](/design/clients/) | -| Translation catalog format + locale resolution + error-code scheme | [Internationalization](/design/i18n/) | +| Translation catalog format + locale resolution + error-code scheme + supported language set + README translation | [Internationalization](/design/i18n/) | | Code module → design doc mapping + bounded E2E test surface | [Module Map](/design/module-map/) | | API surface ↔ transport map + cross-transport negotiation/rejection mapping | [API Surfaces](/design/api-surfaces/) | | Storage durability verdict + verify-before-destroy rule | [Import — Storage Verification](/design/import/storage-verification/) | From ebe9f9c82f416f8569029d43755c260481e01993 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:15:38 -0400 Subject: [PATCH 56/58] docs(design): own the client test and performance tooling strategy Absorbs the still-valid substance of issues #9, #12, and #17 into clients.md: swift-testing is the sole unit/smoke framework on Apple platforms (XCTest only inside XCUITest bundles; core-swift's XCTest smoke migrates as slice S-F7), Instruments + MetricKit are the performance stack with no on-device third-party APM, and Kotlin/web pins ride the same one-framework-per-platform principle. --- SLICES.md | 12 ++++++++++++ capsule-docs/src/content/docs/design/clients.md | 9 +++++++++ capsule-docs/src/content/docs/design/principles.md | 2 +- 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/SLICES.md b/SLICES.md index ea523631..477250f9 100644 --- a/SLICES.md +++ b/SLICES.md @@ -116,6 +116,7 @@ its slice. | S-F4 | Windows TPM (TBS) backend | platform/FFI | S-A4 | M | ready | | S-F5 | Hardware DEK binding | platform/FFI | S-F2 | M | ready | | S-F6 | `log` → `tracing` migration (core + core-ffi) | platform/FFI | — | S | ready | +| S-F7 | core-swift XCTest → swift-testing migration | platform/FFI | — | S | ready | | S-G1 | GraphQL retirement | legacy-retire | S-C2, S-D6 | M | blocked | | S-G2 | Legacy plaintext proto/service removal | legacy-retire | S-C2, S-D2 | S | blocked | | S-G3 | Plaintext entity retirement (face/person/smart_tag) | legacy-retire | S-G1, S-H3 | M | blocked | @@ -921,6 +922,17 @@ SDK; they never hand-roll network flows. unit suites pass unchanged; `mise run check-rust` green. - **Tier:** Unit (existing suites). +### S-F7 — core-swift XCTest → swift-testing migration + +- **Contract:** [Clients — Test and Performance Tooling](capsule-docs/src/content/docs/design/clients.md). +- **Deliverable:** `capsule-core-swift`'s XCTest smoke suite rewritten on + swift-testing (`@Suite`/`@Test`), matching the framework the `capsule-swift` + app already uses; after this, XCTest imports exist only inside XCUITest + UI-automation bundles. +- **Done when:** `swift test` in `capsule-core-swift` runs green with no + `import XCTest` outside UI-automation targets. +- **Tier:** Smoke per platform. + ## Lane G — legacy retirement (frozen until preconditions) Every `LEGACY-PLAINTEXT (frozen)` marker in the tree names its slice here. Frozen code diff --git a/capsule-docs/src/content/docs/design/clients.md b/capsule-docs/src/content/docs/design/clients.md index b81aa379..0fac61e6 100644 --- a/capsule-docs/src/content/docs/design/clients.md +++ b/capsule-docs/src/content/docs/design/clients.md @@ -59,6 +59,15 @@ The defense is structural isolation: This is the canonical declaration of the sandbox; [Federation — Security Against Malicious Files](/design/federation/#security-against-malicious-files) references it for the federated-asset case, and [Backup & Recovery — Backup Verification](/design/backup-recovery/#backup-verification) references it for dry-run decode sanity checks. +## Test and Performance Tooling + +This section owns the per-platform test-framework and performance-tooling pins (the cross-platform library pins live in [Dependencies](/design/dependencies/)). + +- **Apple platforms:** **swift-testing** (`@Suite`/`@Test`) is the sole unit/smoke framework. XCTest is sanctioned **only** inside XCUITest UI-automation bundles, where no swift-testing analogue exists. The `capsule-core-swift` harness's existing XCTest smoke migrates (slice `S-F7` in the repo-root `SLICES.md`); new XCTest unit tests are not accepted. +- **Apple performance work:** **Instruments** (Time Profiler, Allocations, Leaks, Network, Core Animation) for interactive profiling, and **MetricKit** for field metrics (launch time, hangs, memory, disk writes) — no third-party APM runs on-device. +- **Kotlin/Android:** JUnit 5 (Jupiter) is the current harness for the `capsule-core-kotlin` bindings; the canonical pin for the Compose app is recorded here when its test harness stabilizes, under the same one-framework-per-platform principle. +- **Web:** the test runner rides the toolchain pinned in [Dependencies — Web](/design/dependencies/#web) (bun's built-in runner today). + ## Validation The validation duties above translate directly to test surface. Most live in `capsule-core` (so they apply uniformly to every client); the per-platform pieces are the sandbox harness. diff --git a/capsule-docs/src/content/docs/design/principles.md b/capsule-docs/src/content/docs/design/principles.md index 3ff714fb..adc45999 100644 --- a/capsule-docs/src/content/docs/design/principles.md +++ b/capsule-docs/src/content/docs/design/principles.md @@ -64,7 +64,7 @@ The owner docs are: | Web upload — upload links, guest drops, adoption | [Web Upload](/design/web-upload/) | | Moderation policy + federated reporting + blocklists | [Moderation](/design/moderation/) | | Quota accounting + enforcement points | [Quota](/design/quota/) | -| Client validation duties + sandboxed decoder | [Clients](/design/clients/) | +| Client validation duties + sandboxed decoder + client test/perf tooling | [Clients](/design/clients/) | | Translation catalog format + locale resolution + error-code scheme + supported language set + README translation | [Internationalization](/design/i18n/) | | Code module → design doc mapping + bounded E2E test surface | [Module Map](/design/module-map/) | | API surface ↔ transport map + cross-transport negotiation/rejection mapping | [API Surfaces](/design/api-surfaces/) | From c123a287b7d29e493870d091cf91f236050e037d Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:18:46 -0400 Subject: [PATCH 57/58] docs(slices): split video derivatives and reconcile slices with the docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SLICES-vs-docs audit findings, closed out: video derivative generation gets its own contracted slice (S-B5 — thumbnails.md names the seam; S-B1 is stills-only), the three unclaimed bounded E2E cases get owners (case 1 -> S-D5, case 6 -> S-C12, case 9 -> S-D1, case 7 annotated to S-C16+S-C11), S-X3 gains its missing Contract line (mls-resilience.md + versioning.md), and every slice ID referenced anywhere now resolves to an index row and block. --- SLICES.md | 47 ++++++++++++++----- .../src/content/docs/design/module-map.md | 8 ++-- .../src/content/docs/design/thumbnails.md | 4 ++ 3 files changed, 44 insertions(+), 15 deletions(-) diff --git a/SLICES.md b/SLICES.md index 477250f9..1e3e87ed 100644 --- a/SLICES.md +++ b/SLICES.md @@ -71,6 +71,7 @@ its slice. | S-B2 | Signed-path import-executor rewrite | media/import | S-B1 | L | ready | | S-B3 | Streaming import (probe, `total_size`, drive mode) | media/import | S-D1, S-D4 | L | ready | | S-B4 | Staged uploads (low-data tier ladder) | media/import | S-C1, S-C2, S-D1 | M | ready | +| S-B5 | Video derivatives (first-frame still + H.264 preview) | media/import | S-B1 | M | ready | | S-B6 | Google Takeout importer | media/import | S-B2 | M | ready | | S-B7 | iCloud export importer | media/import | S-B6 | M | post-v1 | | S-B8 | Immich importer | media/import | S-B6 | M | post-v1 | @@ -166,6 +167,7 @@ graph LR D1 --> D5 D2 --> E3 B1[S-B1 thumbnails] --> B2[S-B2 executor rewrite] --> G4[S-G4 executor retire] + B1 --> B5[S-B5 video derivatives] B2 --> B6[S-B6 takeout] --> B7[S-B7 icloud] B6 --> B8[S-B8 immich] B6 --> Z2[S-Z2 migration guides] @@ -306,13 +308,15 @@ pass until the gate lifts. ### S-B1 — Thumbnail/LQIP generation - **Contract:** [Thumbnails](capsule-docs/src/content/docs/design/thumbnails.md). -- **Deliverable:** thumbnail/preview generation over `capsule_core::media` (the folded - former `capsule-media` crate, behind the non-default `media` feature; today it decodes - JPEG only — format decoders grow as needed), chromahash LQIP + `dominant_color` into - the sidecar `lqip` field, `DerivativeManifest`-signed outputs. -- **Done when:** generation produces the committed formats with signed derivative - manifests; LQIP lands in the sidecar and renders as the fallback tier. -- **Tier:** Unit + Smoke. **Blocks:** S-B2. +- **Deliverable:** **still-image** thumbnail/preview generation over + `capsule_core::media` (the folded former `capsule-media` crate, behind the + non-default `media` feature; today it decodes JPEG only — format decoders grow as + needed), chromahash LQIP + `dominant_color` into the sidecar `lqip` field, + `DerivativeManifest`-signed outputs. Video tiers are split to S-B5 (distinct + transcode toolchain). +- **Done when:** generation produces the committed still formats with signed + derivative manifests; LQIP lands in the sidecar and renders as the fallback tier. +- **Tier:** Unit + Smoke. **Blocks:** S-B2, S-B5. ### S-B2 — Signed-path import-executor rewrite @@ -355,6 +359,19 @@ pass until the gate lifts. resume-from-server-truth, staged×streaming exclusion). - **Tier:** Unit + Smoke. +### S-B5 — Video derivatives + +- **Contract:** [Thumbnails — Video Previews](capsule-docs/src/content/docs/design/thumbnails.md) + (formats fixed by the tier table). +- **Deliverable:** the video derivative path behind the `media` feature — first-frame + JXL/AVIF still for the thumbnail tier, H.264 baseline preview transcode (original + resolution capped to 1080p, CRF 23, 30 fps cap, AAC audio) for the preview tier — + signed through the same `DerivativeManifest` path as S-B1's stills. +- **Depends on:** S-B1 (derivative plumbing + signing path). +- **Done when:** a fixture video yields both tiers with signed manifests; the + closed-format rejection covers the video rows of the tier table. +- **Tier:** Unit + Smoke. + ### S-B6 — Google Takeout importer - **Contract:** [Import — Third-Party Importers](capsule-docs/src/content/docs/design/import/pipeline.md). @@ -553,7 +570,9 @@ pass until the gate lifts. surfaced client-side. - **Done when:** escrow round-trips through the server and unwraps with the passphrase path already tested in core; after a replace, the prior blob is gone and unwraps - nothing. **Tier:** Smoke. **Blocks:** S-D12. + nothing. **Tier:** Smoke + E2E case 6 (backup → restore on a fresh device: this + slice's escrow fetch bootstraps the passphrase path over the implemented core + restore). **Blocks:** S-D12. ### S-C13 — Session device-cohort storage + grouping @@ -625,7 +644,8 @@ SDK; they never hand-roll network flows. - **Depends on:** S-C1. **Blocks:** S-B3, S-B4, S-D5. **Done when:** the upload doc's client-side Validation bullets pass against a real server; the recovery matrix has a mocked-HTTP test per code; E2E case 2 lives. -- **Tier:** Unit + Smoke. +- **Tier:** Unit + Smoke + E2E case 9 (the cross-version protocol gate — this slice's + `426` abort-with-upgrade path against the real handshake). ### S-D2 — SDK sync/download client @@ -672,8 +692,10 @@ SDK; they never hand-roll network flows. - **Contract:** [Clients](capsule-docs/src/content/docs/design/clients.md) (the CLI is a client). - **Deliverable:** the `todo!()` CLI commands (`auth login/logout`, `sync`, `list`) over the SDK clients. -- **Depends on:** S-D1, S-D2, S-D7 (the token store — the CLI never hand-rolls auth). **Done when:** `capsule auth login && capsule sync && - capsule list` round-trips against a dev server. **Tier:** Smoke. +- **Depends on:** S-D1, S-D2, S-D7 (the token store — the CLI never hand-rolls auth). + **Done when:** `capsule auth login && capsule sync && capsule list` round-trips + against a dev server. **Tier:** Smoke + E2E case 1 (auth → sync → client-side + library query — the CLI round-trip *is* the case). ### S-D6 — Web server gateway @@ -1074,6 +1096,9 @@ runtime, error-code scheme) already ships — this lane is the content and rollo ### S-X3 — Album upgrade ceremony + MLS resilience +- **Contract:** [MLS Resilience](capsule-docs/src/content/docs/design/mls-resilience.md) + (state divergence, lost commits, the re-keying ceremony, `ReconcileOutcome`), + [Versioning](capsule-docs/src/content/docs/design/versioning.md) (upgrade ceremony). - **Deliverable:** the tombstone-plus-fork ceremony, re-keying, reconciliation (`ReconcileOutcome`). **Depends on:** S-X2; E2E case 8. diff --git a/capsule-docs/src/content/docs/design/module-map.md b/capsule-docs/src/content/docs/design/module-map.md index 7448e324..e3d7b4d6 100644 --- a/capsule-docs/src/content/docs/design/module-map.md +++ b/capsule-docs/src/content/docs/design/module-map.md @@ -161,15 +161,15 @@ Target count: ≤ 13 cases. Each one is named by what it proves — not "test X" **Status:** none of the 13 cases is runnable today — every one crosses at least one planned surface (the networked server/client wiring, MLS, or ML), even where its `capsule-core` half is implemented and unit/smoke-tested. The bound starts governing with the first implementation slice that makes a case live; until then this list is the contract the slices build toward. -1. **Auth → sync → client-side library query.** Log in via OIDC → access token → gRPC `Sync` returns the account's album entries → the client applies them and a local `library.sqlite` query lists the expected albums (rich queries are client-side per [API Surfaces](/design/api-surfaces/)). Covers `capsule-api-auth::oidc + session` × `capsule-api-sync` × `capsule-core::library`. +1. **Auth → sync → client-side library query.** Log in via OIDC → access token → gRPC `Sync` returns the account's album entries → the client applies them and a local `library.sqlite` query lists the expected albums (rich queries are client-side per [API Surfaces](/design/api-surfaces/)). Covers `capsule-api-auth::oidc + session` × `capsule-api-sync` × `capsule-core::library`. Made live by slice `S-D5`. 2. **Full import + upload + finalize.** Local scan → plan → execute → upload session → finalize → blob present at `blobs/{hash}` + index row marked uploaded. Covers `capsule-core::import` × `capsule-sdk::upload` × `capsule-api-upload` × `capsule-api-entity`. 3. **Sync feed pickup.** Upload from device A → device B's sync feed advances → device B fetches metadata blob and (per scope) the original. Covers `capsule-api-sync` × `capsule-sdk` download path × `capsule-core::library` write. 4. **Federation cross-server pull.** Alice on `home.tld` shares to Bob on `other.tld` → capability token → Bob's server pulls metadata + blobs → Bob's client renders. Covers `capsule-api-sync::federation` (both sides) × `capsule-api-auth` (capability issue). 5. **LAN peering A→B.** Two devices on the same LAN; mDNS discovery → TLS handshake → delta-scoped artifact → restore on receiver → byte-equal libraries. Covers `capsule-sdk::peering` × `capsule-core::backup` × `capsule-core::library`. -6. **Backup → restore on a fresh device.** Export full backup → bootstrap new device via passphrase + escrow → import backup → assert every asset present and verifiable. Covers `capsule-core::backup` × `capsule-core::crypto::keys` × `capsule-core::library`. -7. **Full lifecycle.** Create → metadata-update → trash → restore → re-delete → hard-purge after retention. Provenance chain advances through every transition; server refuses purge before `retention_until`. Covers `capsule-api-auth::roles` × `capsule-core::crypto::provenance` × server purge worker. +6. **Backup → restore on a fresh device.** Export full backup → bootstrap new device via passphrase + escrow → import backup → assert every asset present and verifiable. Covers `capsule-core::backup` × `capsule-core::crypto::keys` × `capsule-core::library`. Made live by slice `S-C12` (the escrow surface — the core halves already ship). +7. **Full lifecycle.** Create → metadata-update → trash → restore → re-delete → hard-purge after retention. Provenance chain advances through every transition; server refuses purge before `retention_until`. Covers `capsule-api-upload::ops` (lifecycle-write surface) × `capsule-core::crypto::provenance` × server purge worker. Made live by slices `S-C16` + `S-C11` (with `S-D2`). 8. **Album upgrade ceremony.** Multi-member album; admin initiates upgrade → quiesce → drain → tombstone → fork → queued writes replay. Includes one resume-from-crash mid-ceremony. Covers `capsule-core::crypto::mls` × `capsule-api` × client UI. -9. **Cross-version protocol gate.** Client with `protocol_version` outside server's range attempts upload; receives `426`; UI surfaces actionable error. Covers `capsule-api` handshake × `capsule-sdk` error handling. +9. **Cross-version protocol gate.** Client with `protocol_version` outside server's range attempts upload; receives `426`; UI surfaces actionable error. Covers `capsule-api` handshake × `capsule-sdk` error handling. Made live by slice `S-D1` (its `426` abort-with-upgrade recovery path). 10. **Model regen after version bump.** Bump canonical model version; assert stale embeddings excluded from queries; background regen produces fresh embeddings; queries return correct results post-regen. Covers `capsule-core::ml` × `capsule-core::db` vector index. 11. **Server crash mid-finalization.** Inject crash between blob rename and Postgres transaction commit; restart; assert session moves to `FailedProcessing` cleanly, no orphaned blob, no zombie pending row. Covers `capsule-api-upload` × `capsule-api-entity` × `capsule-api`'s startup scrub. 12. **Cross-device enrollment.** Existing device A authorizes new device B over a verified channel (enrollment code + safety-code check) → B generates hardware keys → A cross-signs B into the device directory → B joins each album's MLS group → B's library matches A's. Includes one MITM-on-relay abort. Covers `capsule-api-auth::devices` × `capsule-core::crypto::keys` × `capsule-sdk::hardware-keys`. diff --git a/capsule-docs/src/content/docs/design/thumbnails.md b/capsule-docs/src/content/docs/design/thumbnails.md index 6e9cf973..6f801e86 100644 --- a/capsule-docs/src/content/docs/design/thumbnails.md +++ b/capsule-docs/src/content/docs/design/thumbnails.md @@ -29,6 +29,10 @@ Two derivative tiers per photo asset and one preview tier for video assets: - **WebP** is the last-resort fallback for the rare client lacking AVIF. We deliberately do not fall back to JPEG — WebP covers everything JPEG would. - **H.264 baseline** for video previews — universally decodable, cheap to decode on every platform. AV1 was considered but mobile encode cost is still high in 2026. +### Video Previews + +The table above stays the SSoT for the video formats; this section only names the implementation seam. Video derivative generation — the first-frame still and the H.264 baseline preview transcode — is its own implementation slice (`S-B5` in the repo-root `SLICES.md`), split from still-image generation (`S-B1`) because transcode brings a distinct toolchain (demux, video decode, H.264/AAC encode) the still path never touches. Both slices sign their outputs identically through the [`DerivativeManifest`](#derivative-provenance) path. + If an original asset is lower-resolution than the highest thumbnail tier, that tier references the original instead of generating a redundant derivative. This is **distinct** from a missing derivative (an unintentional generation failure): the tier's [`DerivativeManifest`](/design/cryptography/provenance/#derivative-provenance) carries the recognized sentinel `format = "original"` — an explicit, signed marker the receiver trusts — whereas a simply-absent derivative is treated as rebuildable from the original (recovery-first). ## LQIP From e7168041daf5006fc8f9d85c09e2dd0aded103f1 Mon Sep 17 00:00:00 2001 From: Justin Chung <20733699+justin13888@users.noreply.github.com> Date: Fri, 10 Jul 2026 04:19:38 -0400 Subject: [PATCH 58/58] style(cli): apply rustfmt to the jiff status change --- capsule-cli/src/status.rs | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/capsule-cli/src/status.rs b/capsule-cli/src/status.rs index ed867a33..97708b1c 100644 --- a/capsule-cli/src/status.rs +++ b/capsule-cli/src/status.rs @@ -1,10 +1,10 @@ use std::path::PathBuf; use std::time::SystemTime; -use jiff::Timestamp; use colored::*; use eyre::{Result, eyre}; use humansize::{BINARY, format_size}; +use jiff::Timestamp; use crate::config::Config; use crate::utils::directories::{ @@ -145,11 +145,7 @@ impl AuthStatus { println!(" {} {}", "Token:".dimmed(), "Invalid".red()); } if let Some(expires) = &self.token_expires_at { - println!( - " {} {}", - "Expires:".dimmed(), - expires.to_string().dimmed() - ); + println!(" {} {}", "Expires:".dimmed(), expires.to_string().dimmed()); } } else { println!(" {} {}", "Status:".dimmed(), "Not logged in".red());