diff --git a/README.md b/README.md index 3a092bf21..9934286ed 100644 --- a/README.md +++ b/README.md @@ -445,6 +445,7 @@ Read: - [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md) - [`docs/ECONOMICS.md`](docs/ECONOMICS.md) +- [`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md) Use a dedicated low-balance operator wallet. Do not use a treasury wallet as the heartbeat key. diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md new file mode 100644 index 000000000..16e205fcc --- /dev/null +++ b/docs/STORAGE-AND-PINNING.md @@ -0,0 +1,89 @@ +# Storage and pinning + +How a gitlawb node stores git objects and keeps them available. This documents +the behavior implemented in `crates/gitlawb-node/src/ipfs_pin.rs` and +`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. + +## Two-tier model + +After a push lands, new git objects are pinned to up to two independent sinks. +Both are **opt-in and independent** — a node runs fine with neither, either, or +both configured. + +| Tier | Sink | Module | Enabled by | Purpose | +|------|------|--------|-----------|---------| +| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon | +| Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | + +If a sink's config value is empty, its **pin** paths are no-ops — so leaving +`GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*, though not the tail's +cost: the post-push Pinata tail still takes a global pin permit and runs the full +object-list re-derivation walk before the upload step returns empty, since its +gate is the announce/walk state, not the JWT. Note the no-op applies to pinning +only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error rather +than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the +encrypted-blob read endpoint needs Kubo configured. + +## Configuration + +| Env var | Default | Meaning | +|---------|---------|---------| +| `GITLAWB_IPFS_API` | `""` (disabled) | Base URL of the local Kubo HTTP API, e.g. `http://127.0.0.1:5001` | +| `GITLAWB_PINATA_JWT` | `""` (disabled) | Pinata bearer JWT enabling the warm tier | +| `GITLAWB_PINATA_UPLOAD_URL` | `https://uploads.pinata.cloud/v3/files` | Pinata v3 upload endpoint | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | `8` | Cap on concurrent post-push pin loops across all repos | + +## How pinning runs + +Pinning happens **after** a push is accepted, not on the push's critical path: + +- The **hot** tier pins inline in the post-push encrypt/pin task. +- The **warm** (Pinata) tier runs in a spawned replication tail, so a slow or + unreachable Pinata never blocks the pusher. + +Both tiers share a single global **pin admission semaphore** +(`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is +saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is +bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot +indefinitely. The Pinata tail re-derives its object list only *after* acquiring a +slot, which bounds *its* outstanding memory to O(refs) rather than +O(pushes × objects) — but that bound is the warm tail's alone. A **hot-tier** pin +loop takes an owned object list materialized by its caller *before* it waits for +a slot, so every hot loop parked on a saturated pool is holding its full list: +the hot side of the parked-list memory is O(pushes × objects), capped per repo by +`EncryptInflight` but not across repos. Do not size a node's memory from the +semaphore knob (README says the same). +Note the budget covers the pin batch, not the preceding object-list re-derivation +walk: that walk holds the slot too, and bounds only each child git process +individually (no aggregate deadline). + +De-duplication is per sink and best-effort, backed by the single `pinned_cids` +table: the hot tier keys on its `cid`/`sha256_hex` rows and the warm tier on the +nullable `pinata_cid` column, so later pushes normally skip objects whose +successful pin is already recorded. The isolation is one-directional: a hot-only +pin does not suppress the warm tier (`pinata_cid` stays null), but the hot tier's +skip check is row *presence*, and a warm-tier pin inserts the row — so an object +warm-pinned while `GITLAWB_IPFS_API` was unset is skipped by the hot tier after +Kubo is configured, until something re-pins it. The check-upload-record sequence is not atomic, +so concurrent post-push tasks for the same object, or a failure to record after a +successful upload, can still cause a repeat upload attempt. + +## Durability notes + +- With only the **hot** tier, availability depends on the node (and any IPFS + peers that have fetched the CIDs). If the node is down and no peer holds the + objects, they are unreachable until it returns. +- The **warm** tier adds off-node durability via Pinata's Filecoin-backed + storage and makes objects reachable through the public IPFS gateway. +- Running **both** gives a node-local hot copy plus an off-node warm copy. + +> The two sinks are currently invoked as separate call paths. Unifying them +> behind a single pluggable backend interface (to add providers such as direct +> Filecoin deals or self-hosted clusters without touching push logic) is tracked +> separately. + +## See also + +- [RUN-A-NODE.md](RUN-A-NODE.md) — provisioning and running a node +- `crates/gitlawb-node/src/ipfs_pin.rs` — hot-tier implementation +- `crates/gitlawb-node/src/pinata.rs` — warm-tier implementation