From 25f8b716e795631d51b3ea215352561ae2644cce Mon Sep 17 00:00:00 2001 From: Gravirei Date: Fri, 28 Aug 2026 22:49:11 +0600 Subject: [PATCH 01/22] fix(node): durable post-receive outbox at the DB layer (#26 split 1/4) Reviewer 2 closed PR #224 on 2026-08-28 with a directive: split the work into four narrow PRs. This is Split PR 1 (durable post-receive lifecycle) at the DB layer; the handler refactor in crates/gitlawb-node/src/api/repos.rs:2007 (git_receive_pack) lands in the next slice so the test can drive the failure injection end-to-end. The pre-outbox crash window the reviewer flagged: receive_pack can apply a ref to disk and return Ok, and a process exit, a dropped future, or a DB failure before the bookkeeping at crates/gitlawb-node/src/api/repos.rs:2361 (push event + cert + webhook) loses the recovery record. Startup drain enumerates only sources written from that bookkeeping, so it cannot reconstruct the missing work. The partial fallback that re-derives from a row present in the bookkeeping substitutes did:key:recovered and an empty attestation, which is not equivalent to the original authenticated push. This commit adds the durable boundary the handler will lean on. NEW TABLE pending_ref_transitions (migration v27): - Written by the handler BEFORE smart_http::receive_pack, in state 'prepared', carrying the verified pusher DID, the raw RFC 9421 signature header, signature-input, and content-digest that authorized the push, the request id, and the parsed ref update. - The handler transitions the row to 'applied' on receive_pack Ok or 'cancelled' on Err. The drain reads only 'applied'. - A failed or cancelled receive-pack therefore leaves the row in 'prepared' or 'cancelled', which the drain never promotes. This is what closes the reviewer's second proof ("a failed or cancelled receive-pack does not turn a prepared intent into completed accounting or anchoring"). NEW TABLE anchor_jobs (migration v27, owned by PR 1, consumed by PR 2): - One row per (repo_id, ref_name, old_sha, new_sha) transition. PR 1 inserts it on 'applied'; PR 2 reads it and updates claimed_at. - ON CONFLICT (id) DO NOTHING makes the insert idempotent on the deterministic id, so a recovery re-pass cannot create a second upload request. This is the handoff boundary; the bundler call itself is PR 2. NEW DB METHODS on Db: - insert_pending_ref_transitions: writes one 'prepared' row per ref update, returns the persisted rows. - mark_pending_ref_transitions_applied / _cancelled: state flip, gated on the FROM state, idempotent. - list_pending_ref_transitions_applied: drain query, oldest first. - delete_pending_ref_transition: called by recovery after the artifacts land; a third pass is a no-op. - record_push_with_id: ON CONFLICT (id) DO NOTHING on the deterministic id. - insert_ref_certificate_idempotent: ON CONFLICT (repo_id, ref_name) DO NOTHING, returns None if a live-path cert already exists. - insert_anchor_job_idempotent: ON CONFLICT (id) DO NOTHING on the deterministic per-transition id. NEW HELPERS in db/mod.rs: - deterministic_id: SHA-256 hex with an ASCII Unit Separator between fields so two distinct tuples never collide on prefix overlap. - push_event_id_for, ref_cert_id_for, anchor_job_id_for: the derived ids above, one helper per artifact so a caller cannot derive a wrong id by mistake. NEW STRUCTS: - PendingRefTransition: the row shape. - AnchorJob: the handoff row shape. - pending_state: const strings ('prepared' / 'applied' / 'cancelled') shared by tests, the producer, and the drain so a typo on one side cannot silently mismatch the other. NEW TESTS in db::pending_ref_transition_tests (8 tests, all green): - insert_then_mark_applied_flips_state_for_every_ref: producer contract. - mark_applied_is_idempotent_on_repeat: re-fire is a no-op. - cancelled_rows_are_not_returned_by_the_drain: reviewer's second proof at the DB layer. - prepared_rows_are_not_returned_by_the_drain: same proof for the pre-flip state (handler crashed before reaching post-Ok). - mark_cancelled_is_idempotent_on_repeat: counterpart. - drain_then_re_derive_is_idempotent: reviewer's first proof at the DB layer. Inserts a row in 'applied' state directly via insert_pending_ref_transition_for_test, drains it, derives the artifact ids twice, exercises record_push_with_id and insert_anchor_job_idempotent directly, asserts exactly one push event row and exactly one anchor job row regardless of how many times the drain runs. - deterministic_id_avoids_prefix_overlap_collisions: the separator regression test. - push_event_id_for_is_stable: derived ids match across calls and differ on each varied input. OTHER: - Make RefUpdate and its fields pub(crate) so the DB methods can iterate the parsed ref updates. No public API change. NOT IN THIS SLICE (the handler refactor, next commit): - The receive-pack handler does not yet call insert_pending_ref_ transitions before the receive_pack call, nor mark_applied / mark_cancelled after. The DB layer is in place for it; the handler will call these methods and the startup drain will be wired in main.rs. - The startup drain in main.rs is not yet called; it will iterate list_pending_ref_transitions_applied, re-derive the artifacts, and delete the row. - The cert/push event issuance in cert.rs and the bookkeeping in api/repos.rs:2361 are not yet changed to use the deterministic ids. The helper functions exist and are tested; the callers follow. Compiles clean, clippy clean under -D warnings, fmt clean. --- crates/gitlawb-node/src/api/repos.rs | 8 +- crates/gitlawb-node/src/db/mod.rs | 971 +++++++++++++++++++++++++++ 2 files changed, 975 insertions(+), 4 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 4e327c429..896b80e13 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -3147,10 +3147,10 @@ pub async fn get_icaptcha_proof( /// replication tail at the durability boundary while the certificate and webhook /// loops below still iterate their own copy (#174 U5). #[derive(Clone)] -struct RefUpdate { - old_sha: String, - new_sha: String, - ref_name: String, +pub(crate) struct RefUpdate { + pub(crate) old_sha: String, + pub(crate) new_sha: String, + pub(crate) ref_name: String, } /// Parse git receive-pack pkt-line ref updates from the request body. diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index cc2cf0bd7..88b8e5297 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -1,6 +1,7 @@ use anyhow::{Context, Result}; use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; use sqlx::{postgres::PgPoolOptions, PgPool, Row}; use std::time::Duration; use tracing::info; @@ -153,6 +154,122 @@ pub struct RefCertificate { pub issued_at: String, } +/// The lifecycle states of a row in `pending_ref_transitions`. Persisted as a +/// TEXT column with one of these string values; the constants are the canonical +/// spellings, and tests + the recovery drain all use them so a typo on one +/// side or the other cannot silently mismatch the other. +#[allow(dead_code)] // constants are used by tests + the next-slice handler +pub mod pending_state { + #[allow(dead_code)] + pub const PREPARED: &str = "prepared"; + #[allow(dead_code)] + pub const APPLIED: &str = "applied"; + #[allow(dead_code)] + pub const CANCELLED: &str = "cancelled"; +} + +/// #26 Split PR 1 — durable intent row for a single (request, ref) transition. +/// +/// One row is written BEFORE `smart_http::receive_pack` runs, in state +/// `prepared`, carrying the verified pusher DID, the raw RFC 9421 signature +/// header that authorized the push, the request id, and the parsed ref +/// update. The handler then transitions the row to `applied` on Ok or +/// `cancelled` on Err. Startup recovery drains only `applied` rows. +/// +/// `request_id` is the per-handler UUID. It is the deterministic key for +/// the push event, the ref certificate, and the anchor job — those +/// artifacts derive their ids from `(request_id, ref_name)` (cert and push) +/// or `(repo_id, ref_name, old_sha, new_sha)` (anchor) so a recovery pass +/// that re-fires the same transition cannot create a second row of any +/// of them. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[allow(dead_code)] // wired by the handler refactor in the next slice +pub struct PendingRefTransition { + pub id: String, + pub request_id: String, + pub repo_id: String, + pub ref_name: String, + pub old_sha: String, + pub new_sha: String, + pub pusher_did: String, + pub node_did: String, + pub signature_header: String, + pub signature_input: String, + pub content_digest: String, + pub state: String, + pub created_at: String, + pub applied_at: Option, + pub cancelled_at: Option, +} + +/// #26 Split PR 1 — anchor handoff row, owned by PR 1, consumed by PR 2. +/// +/// One row per `(repo_id, ref_name, old_sha, new_sha)` transition. The +/// recovery path inserts it on `applied` using `ON CONFLICT (id) DO +/// NOTHING` (id derived from the tuple) so re-running the drain is +/// idempotent. Split PR 2 reads the row, calls the bundler, and updates +/// `claimed_at` to take it. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[allow(dead_code)] // wired by the handler refactor in the next slice +pub struct AnchorJob { + pub id: String, + pub repo_id: String, + pub ref_name: String, + pub old_sha: String, + pub new_sha: String, + pub pusher_did: String, + pub created_at: String, + pub claimed_at: Option, +} + +/// SHA-256 hex of an arbitrary tuple, used as the deterministic id for the +/// artifacts that recovery inserts idempotently. Returns 64 lowercase hex +/// characters. The input is concatenated with `\x1f` (ASCII Unit Separator) +/// as the field separator so two distinct tuples can never collide by +/// accidental prefix overlap, e.g. `(a, bc)` and `(ab, c)` would otherwise +/// produce the same hash input. +#[allow(dead_code)] // called from tests + the next-slice handler refactor +pub fn deterministic_id(parts: &[&str]) -> String { + let mut hasher = Sha256::new(); + for part in parts { + hasher.update(b"\x1f"); + hasher.update(part.as_bytes()); + } + hasher.update(b"\x1e"); // end-of-record terminator; never appears in any field + let digest = hasher.finalize(); + hex::encode(digest) +} + +/// Deterministic id for a push event row. Derived from +/// `(request_id, ref_name)` so a recovery pass re-firing the same +/// transition produces the same id and the ON CONFLICT collapses to a +/// no-op rather than creating a second push event. +#[allow(dead_code)] // wired by the handler refactor in the next slice +pub fn push_event_id_for(request_id: &str, ref_name: &str) -> String { + deterministic_id(&["push_event", request_id, ref_name]) +} + +/// Deterministic id for a ref certificate row. Derived from +/// `(request_id, ref_name)` for the same idempotency reason as +/// `push_event_id_for`. The certificate's `id` column is the primary +/// key; the unique index on `(repo_id, ref_name)` still applies, so +/// the recovery path must additionally check for an existing cert +/// before inserting to avoid the upsert replacing a live-path cert. +#[allow(dead_code)] // wired by the handler refactor in the next slice +pub fn ref_cert_id_for(request_id: &str, ref_name: &str) -> String { + deterministic_id(&["ref_cert", request_id, ref_name]) +} + +/// Deterministic id for an anchor job. The anchor's uniqueness contract +/// is per-transition, not per-request, because two different pushes to +/// the same ref (different `request_id`) should still produce ONE +/// anchor per landed state. The key is the transition tuple +/// `(repo_id, ref_name, old_sha, new_sha)`. +#[allow(dead_code)] // wired by the handler refactor in the next slice +pub fn anchor_job_id_for(repo_id: &str, ref_name: &str, old_sha: &str, new_sha: &str) -> String { + deterministic_id(&["anchor_job", repo_id, ref_name, old_sha, new_sha]) +} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct PeerRecord { pub did: String, @@ -1123,6 +1240,95 @@ const MIGRATIONS: &[Migration] = &[ "ALTER TABLE pin_repair_sweep ADD COLUMN IF NOT EXISTS discovery_cursor_id TEXT NOT NULL DEFAULT ''", ], }, + Migration { + version: 27, + name: "pending_ref_transitions_durable_outbox", + stmts: &[ + // #26 Split PR 1 — durable post-receive lifecycle. + // + // The pre-outbox crash window the reviewer flagged: receive_pack can + // apply a ref to disk and return Ok, and a process exit, a dropped + // future, or a DB failure before the bookkeeping at + // crates/gitlawb-node/src/api/repos.rs:2361 (push event + cert + + // webhook) loses the recovery record. Startup drain enumerates only + // sources written from that bookkeeping, so it cannot reconstruct + // the missing work. The partial fallback that re-derives from a row + // present in the bookkeeping substitutes `did:key:recovered` and an + // empty attestation — not equivalent to the original authenticated + // push. + // + // The fix is to persist the authentic intent BEFORE the receive_pack + // call lands the ref. The row carries the verified pusher DID, the + // raw RFC 9421 signature header that authorized this push, the + // request id, and the parsed ref updates. The receive_pack call + // then transitions the row `prepared` → `applied` on Ok, or + // `prepared` → `cancelled` on Err. Startup recovery drains only + // `applied` rows, re-deriving the push event, the per-ref + // certificate (carrying the ORIGINAL pusher DID, not a placeholder), + // and the anchor handoff — exactly once per transition. + // + // `cancelled` rows are NEVER promoted. A failed or dropped + // receive_pack leaves the row in `prepared`; only the post-Ok code + // flips to `applied`, and only that state is drained. This is what + // closes the reviewer's second proof: a prepared intent that never + // lands cannot become a push event, a certificate, or an anchor. + // + // `request_id` is a per-handler UUID. It is the producer of the + // deterministic ids for the push event, the certificate, and the + // anchor job, so re-running recovery is idempotent on + // `(request_id, ref_name)` — the unique key. + // + // `signature_header` is the raw `Signature` request header value, + // the `keyid` is the pusher DID (already extracted to `pusher_did`). + // It is kept for audit, not re-verified on recovery: the + // `require_signature` middleware already verified it before the + // handler ran, and the route is gated by it. + r#"CREATE TABLE IF NOT EXISTS pending_ref_transitions ( + id TEXT NOT NULL PRIMARY KEY, + request_id TEXT NOT NULL, + repo_id TEXT NOT NULL, + ref_name TEXT NOT NULL, + old_sha TEXT NOT NULL, + new_sha TEXT NOT NULL, + pusher_did TEXT NOT NULL, + node_did TEXT NOT NULL, + signature_header TEXT NOT NULL, + signature_input TEXT NOT NULL, + content_digest TEXT NOT NULL, + state TEXT NOT NULL, + created_at TEXT NOT NULL, + applied_at TEXT, + cancelled_at TEXT + )"#, + // The drain order is by `applied_at ASC NULLS LAST, id ASC` so a + // crashed node that re-runs the drain processes transitions in the + // order they were applied. The `id` tiebreaker keeps the order + // stable when many transitions land in the same `applied_at` tick. + "CREATE INDEX IF NOT EXISTS idx_pending_ref_transitions_state_applied_at ON pending_ref_transitions (state, applied_at, id)", + "CREATE INDEX IF NOT EXISTS idx_pending_ref_transitions_request_ref ON pending_ref_transitions (request_id, ref_name)", + "CREATE INDEX IF NOT EXISTS idx_pending_ref_transitions_repo_ref ON pending_ref_transitions (repo_id, ref_name, old_sha, new_sha)", + // The anchor handoff for Split PR 2 to consume. Split PR 1 owns + // the durable queue: one row per (repo, ref, old, new) transition + // whose row in pending_ref_transitions is `applied`. ON CONFLICT + // DO NOTHING on the unique key makes the recovery re-derivation + // idempotent — a second drain pass cannot create a second anchor + // upload request. Split PR 2 owns the actual transport and the + // three-outcome probe; this PR only proves the handoff is + // exactly-once. + r#"CREATE TABLE IF NOT EXISTS anchor_jobs ( + id TEXT NOT NULL PRIMARY KEY, + repo_id TEXT NOT NULL, + ref_name TEXT NOT NULL, + old_sha TEXT NOT NULL, + new_sha TEXT NOT NULL, + pusher_did TEXT NOT NULL, + created_at TEXT NOT NULL, + claimed_at TEXT + )"#, + "CREATE UNIQUE INDEX IF NOT EXISTS idx_anchor_jobs_repo_ref_transition ON anchor_jobs (repo_id, ref_name, old_sha, new_sha)", + "CREATE INDEX IF NOT EXISTS idx_anchor_jobs_claimed_at ON anchor_jobs (claimed_at, id)", + ], + }, ]; /// Max distinct source repos recorded per pinned object (F1, #173 jatmn round 8). @@ -2367,6 +2573,370 @@ impl Db { Ok(row_to_cert(row)) } + // ── #26 Split PR 1: durable post-receive outbox ──────────────────────── + // + // The methods below own the producer/persistence/restore boundary the + // reviewer flagged: every externally visible ref transition has a row + // here before the receive_pack call, the row's state reflects the + // outcome (`applied` on Ok, `cancelled` on Err), and a startup drain + // re-derives the push event, ref certificate, and anchor handoff for + // any `applied` row. Recovery is idempotent because every derived + // artifact has a deterministic id (see `*_id_for` above) and the + // `INSERT ... ON CONFLICT (id) DO NOTHING` clause collapses a + // re-fired transition to a no-op. + // + // The handler is responsible for calling `insert_prepared` before the + // `smart_http::receive_pack` call and `mark_applied` / `mark_cancelled` + // after. The `drain_applied` method is called once at startup, after + // migrations and before serving. Wiring those into the handler is + // tracked as the next slice of work; this commit adds the durable + // boundary and the DB-level idempotency the handler will lean on. + + /// Insert one `prepared` row per ref update in the push, returning the + /// rows as persisted. Called from the receive-pack handler BEFORE + /// `smart_http::receive_pack` runs. + /// + /// `request_id` is the per-handler UUID; the same value must be used + /// for every ref update in a single push, and it becomes the + /// deterministic seed for the push event, ref cert, and anchor job + /// ids. `pusher_did` is the verified DID from the + /// `AuthenticatedDid` extension (the canonical identity the + /// `require_signature` middleware injected). `signature_header` and + /// `signature_input` are the raw RFC 9421 header values, persisted + /// for audit; they were already verified at handler entry. + #[allow(dead_code, clippy::too_many_arguments)] // wired by the handler refactor in the next slice + pub async fn insert_pending_ref_transitions( + &self, + request_id: &str, + repo_id: &str, + node_did: &str, + pusher_did: &str, + ref_updates: &[crate::api::repos::RefUpdate], + signature_header: &str, + signature_input: &str, + content_digest: &str, + ) -> Result> { + let now = Utc::now().to_rfc3339(); + let mut out = Vec::with_capacity(ref_updates.len()); + for update in ref_updates { + let id = deterministic_id(&[ + "pending_ref_transition", + request_id, + repo_id, + &update.ref_name, + &update.old_sha, + &update.new_sha, + ]); + sqlx::query( + r#"INSERT INTO pending_ref_transitions + (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, + signature_header, signature_input, content_digest, state, created_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)"#, + ) + .bind(&id) + .bind(request_id) + .bind(repo_id) + .bind(&update.ref_name) + .bind(&update.old_sha) + .bind(&update.new_sha) + .bind(pusher_did) + .bind(node_did) + .bind(signature_header) + .bind(signature_input) + .bind(content_digest) + .bind(pending_state::PREPARED) + .bind(&now) + .execute(&self.pool) + .await?; + out.push(PendingRefTransition { + id, + request_id: request_id.to_string(), + repo_id: repo_id.to_string(), + ref_name: update.ref_name.clone(), + old_sha: update.old_sha.clone(), + new_sha: update.new_sha.clone(), + pusher_did: pusher_did.to_string(), + node_did: node_did.to_string(), + signature_header: signature_header.to_string(), + signature_input: signature_input.to_string(), + content_digest: content_digest.to_string(), + state: pending_state::PREPARED.to_string(), + created_at: now.clone(), + applied_at: None, + cancelled_at: None, + }); + } + Ok(out) + } + + /// Flip every `prepared` row attached to `request_id` to `applied`. + /// Called after `smart_http::receive_pack` returns Ok. A `prepared` + /// row that the handler never reaches this point for stays in + /// `prepared` and is dropped by the drain (the row is NEVER promoted + /// by anything other than this method), which is what closes the + /// reviewer's "a failed or cancelled receive-pack must not turn a + /// prepared intent into completed accounting or anchoring" invariant. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn mark_pending_ref_transitions_applied(&self, request_id: &str) -> Result { + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1, applied_at = $2 + WHERE request_id = $3 AND state = $4"#, + ) + .bind(pending_state::APPLIED) + .bind(&now) + .bind(request_id) + .bind(pending_state::PREPARED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Flip every `prepared` row attached to `request_id` to `cancelled`. + /// Called when the receive_pack call returns Err or the handler + /// future is dropped. The drain does not promote `cancelled` rows. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn mark_pending_ref_transitions_cancelled(&self, request_id: &str) -> Result { + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1, cancelled_at = $2 + WHERE request_id = $3 AND state = $4"#, + ) + .bind(pending_state::CANCELLED) + .bind(&now) + .bind(request_id) + .bind(pending_state::PREPARED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Return every `applied` row, oldest first. The startup drain calls + /// this once and processes each row by re-deriving the push event, + /// the per-ref cert, and the anchor handoff. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn list_pending_ref_transitions_applied( + &self, + limit: i64, + ) -> Result> { + let limit = limit.max(1); + let rows = sqlx::query( + r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, + signature_header, signature_input, content_digest, state, created_at, + applied_at, cancelled_at + FROM pending_ref_transitions + WHERE state = $1 + ORDER BY applied_at ASC NULLS LAST, id ASC + LIMIT $2"#, + ) + .bind(pending_state::APPLIED) + .bind(limit) + .fetch_all(&self.pool) + .await?; + Ok(rows + .into_iter() + .map(row_to_pending_ref_transition) + .collect()) + } + + /// Delete a row by id. Called by the recovery drain AFTER the push + /// event, the cert, and the anchor job have all landed. A subsequent + /// drain pass is a no-op for the same transition because the row is + /// gone and the deterministic artifact ids collide on `ON CONFLICT`. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn delete_pending_ref_transition(&self, id: &str) -> Result { + let res = sqlx::query("DELETE FROM pending_ref_transitions WHERE id = $1") + .bind(id) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Test-only: insert a row directly in the given state. Used to + /// simulate the crash window ("row is `applied` but the handler + /// never reached the push event / cert / anchor code") without + /// running the full handler. Mirrors the production insert but + /// takes the state as an argument so a test can stage a row that + /// the drain will pick up. + #[cfg(test)] + pub async fn insert_pending_ref_transition_for_test( + &self, + row: &PendingRefTransition, + ) -> Result<()> { + let applied_at = row.applied_at.clone().unwrap_or_default(); + let cancelled_at = row.cancelled_at.clone().unwrap_or_default(); + let applied_at_opt: Option<&str> = if applied_at.is_empty() { + None + } else { + Some(&applied_at) + }; + let cancelled_at_opt: Option<&str> = if cancelled_at.is_empty() { + None + } else { + Some(&cancelled_at) + }; + sqlx::query( + r#"INSERT INTO pending_ref_transitions + (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, + signature_header, signature_input, content_digest, state, created_at, + applied_at, cancelled_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(&row.id) + .bind(&row.request_id) + .bind(&row.repo_id) + .bind(&row.ref_name) + .bind(&row.old_sha) + .bind(&row.new_sha) + .bind(&row.pusher_did) + .bind(&row.node_did) + .bind(&row.signature_header) + .bind(&row.signature_input) + .bind(&row.content_digest) + .bind(&row.state) + .bind(&row.created_at) + .bind(applied_at_opt) + .bind(cancelled_at_opt) + .execute(&self.pool) + .await?; + Ok(()) + } + + /// Idempotent push event insert. Returns `true` if a NEW row was + /// created, `false` if the deterministic id collided with an + /// existing row (recovery re-fired the same transition). + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn record_push_with_id( + &self, + id: &str, + agent_did: &str, + repo_id: &str, + commit_hash: &str, + object_count: i64, + ) -> Result { + let res = sqlx::query( + r#"INSERT INTO push_events (id, agent_did, repo_id, commit_hash, object_count, pushed_at) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT (id) DO NOTHING"#, + ) + .bind(id) + .bind(agent_did) + .bind(repo_id) + .bind(commit_hash) + .bind(object_count) + .bind(Utc::now().to_rfc3339()) + .execute(&self.pool) + .await?; + Ok(res.rows_affected() == 1) + } + + /// Idempotent ref certificate insert. Returns `Some` if a NEW cert + /// was created, `None` if the unique `(repo_id, ref_name)` index + /// already had a row (the live path got there first, or a previous + /// recovery pass did). + /// + /// The primary key is the deterministic `id`; the unique index on + /// `(repo_id, ref_name)` is what makes the recovery exactly-once, + /// because a second insert for the same `(repo_id, ref_name)` + /// returns `None` rather than overwriting the existing cert. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn insert_ref_certificate_idempotent( + &self, + cert: &RefCertificate, + ) -> Result> { + let res = sqlx::query( + r#"INSERT INTO ref_certificates + (id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature, issued_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + ON CONFLICT (repo_id, ref_name) DO NOTHING + RETURNING id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature, issued_at"#, + ) + .bind(&cert.id) + .bind(&cert.repo_id) + .bind(&cert.ref_name) + .bind(&cert.old_sha) + .bind(&cert.new_sha) + .bind(&cert.pusher_did) + .bind(&cert.node_did) + .bind(&cert.signature) + .bind(&cert.issued_at) + .fetch_optional(&self.pool) + .await?; + Ok(res.map(row_to_cert)) + } + + /// Idempotent anchor job insert. Returns `true` if a NEW row was + /// created, `false` if the `(repo_id, ref_name, old_sha, new_sha)` + /// unique index already had a row. PR 2's transport will read these + /// rows; the recovery drain writes them with `ON CONFLICT DO NOTHING` + /// so re-running the drain cannot create a second upload request. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn insert_anchor_job_idempotent(&self, job: &AnchorJob) -> Result { + let res = sqlx::query( + r#"INSERT INTO anchor_jobs + (id, repo_id, ref_name, old_sha, new_sha, pusher_did, created_at, claimed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8) + ON CONFLICT (id) DO NOTHING"#, + ) + .bind(&job.id) + .bind(&job.repo_id) + .bind(&job.ref_name) + .bind(&job.old_sha) + .bind(&job.new_sha) + .bind(&job.pusher_did) + .bind(&job.created_at) + .bind(job.claimed_at.as_deref()) + .execute(&self.pool) + .await?; + Ok(res.rows_affected() == 1) + } + + /// Count anchor jobs for a transition, used by the test to assert + /// "at most one anchor upload" without depending on PR 2's transport. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn count_anchor_jobs( + &self, + repo_id: &str, + ref_name: &str, + old_sha: &str, + new_sha: &str, + ) -> Result { + let row = sqlx::query( + "SELECT COUNT(*) AS cnt FROM anchor_jobs + WHERE repo_id = $1 AND ref_name = $2 AND old_sha = $3 AND new_sha = $4", + ) + .bind(repo_id) + .bind(ref_name) + .bind(old_sha) + .bind(new_sha) + .fetch_one(&self.pool) + .await?; + Ok(row.get::("cnt")) + } + + /// Count push events for a transition, used by the test to assert + /// "exactly one push event" after recovery. + #[allow(dead_code)] // wired by the handler refactor in the next slice + pub async fn count_push_events( + &self, + repo_id: &str, + commit_hash: &str, + agent_did: &str, + ) -> Result { + let row = sqlx::query( + "SELECT COUNT(*) AS cnt FROM push_events + WHERE repo_id = $1 AND commit_hash = $2 AND agent_did = $3", + ) + .bind(repo_id) + .bind(commit_hash) + .bind(agent_did) + .fetch_one(&self.pool) + .await?; + Ok(row.get::("cnt")) + } + pub async fn list_ref_certificates( &self, repo_id: &str, @@ -3948,6 +4518,27 @@ fn row_to_cert(r: sqlx::postgres::PgRow) -> RefCertificate { } } +#[allow(dead_code)] // wired by the handler refactor in the next slice +fn row_to_pending_ref_transition(r: sqlx::postgres::PgRow) -> PendingRefTransition { + PendingRefTransition { + id: r.get("id"), + request_id: r.get("request_id"), + repo_id: r.get("repo_id"), + ref_name: r.get("ref_name"), + old_sha: r.get("old_sha"), + new_sha: r.get("new_sha"), + pusher_did: r.get("pusher_did"), + node_did: r.get("node_did"), + signature_header: r.get("signature_header"), + signature_input: r.get("signature_input"), + content_digest: r.get("content_digest"), + state: r.get("state"), + created_at: r.get("created_at"), + applied_at: r.get("applied_at"), + cancelled_at: r.get("cancelled_at"), + } +} + fn row_to_ref_update(r: sqlx::postgres::PgRow) -> ReceivedRefUpdate { ReceivedRefUpdate { id: r.get("id"), @@ -8552,3 +9143,383 @@ mod cid_candidate_order_tests { ); } } + +#[cfg(test)] +mod pending_ref_transition_tests { + //! #26 Split PR 1 — durable post-receive outbox at the DB layer. + //! + //! These tests exercise the producer / persistence / drain contracts + //! directly. The handler-level test (failure injection between + //! receive_pack and the bookkeeping) is a follow-up that lands with + //! the handler refactor in the next slice. Every test here uses + //! `Db::for_testing` + `run_migrations` to provision a clean schema, + //! so they are independent of any other test's seed state. + //! + //! Each test names the invariant it pins. Reverting the production + //! line under test turns the named assertion red. + + use super::{ + anchor_job_id_for, pending_state, push_event_id_for, ref_cert_id_for, AnchorJob, Db, + PendingRefTransition, + }; + use crate::api::repos::RefUpdate; + use chrono::Utc; + use sqlx::PgPool; + + async fn db(pool: PgPool) -> Db { + let db = Db::for_testing(pool); + db.run_migrations().await.unwrap(); + db + } + + fn ref_update(name: &str, old: &str, new: &str) -> RefUpdate { + RefUpdate { + ref_name: name.to_string(), + old_sha: old.to_string(), + new_sha: new.to_string(), + } + } + + /// The producer contract: every ref update in a push gets a `prepared` + /// row carrying the verified pusher, the signature header, and the + /// request id. `mark_applied` flips exactly those rows. + #[sqlx::test] + async fn insert_then_mark_applied_flips_state_for_every_ref(pool: PgPool) { + let db = db(pool).await; + let updates = vec![ + ref_update( + "refs/heads/main", + "a".repeat(40).as_str(), + "b".repeat(40).as_str(), + ), + ref_update( + "refs/heads/feature", + "c".repeat(40).as_str(), + "d".repeat(40).as_str(), + ), + ]; + let rows = db + .insert_pending_ref_transitions( + "req-1", + "repo-1", + "did:key:node", + "did:key:pusher", + &updates, + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + ) + .await + .unwrap(); + assert_eq!(rows.len(), 2, "one row per ref update"); + assert!(rows.iter().all(|r| r.state == pending_state::PREPARED)); + + let flipped = db + .mark_pending_ref_transitions_applied("req-1") + .await + .unwrap(); + assert_eq!(flipped, 2, "every prepared row for the request flips"); + + let drained = db.list_pending_ref_transitions_applied(100).await.unwrap(); + assert_eq!(drained.len(), 2); + assert!(drained.iter().all(|r| r.state == pending_state::APPLIED)); + assert!(drained.iter().all(|r| r.pusher_did == "did:key:pusher")); + assert!( + drained + .iter() + .all(|r| r.signature_header == "Signature: sig=..."), + "the original signature header must survive the round trip — \ + recovery re-derives the cert and the anchor under the original identity" + ); + } + + /// A second `mark_applied` for the same request is a no-op — the row is + /// already in `applied` and the state predicate prevents re-flipping. + /// This is what makes a recovery re-pass safe. + #[sqlx::test] + async fn mark_applied_is_idempotent_on_repeat(pool: PgPool) { + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-2", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ref_update( + "refs/heads/main", + "a".repeat(40).as_str(), + "b".repeat(40).as_str(), + )], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + ) + .await + .unwrap(); + + assert_eq!( + db.mark_pending_ref_transitions_applied("req-2") + .await + .unwrap(), + 1, + "first call flips the one row" + ); + assert_eq!( + db.mark_pending_ref_transitions_applied("req-2") + .await + .unwrap(), + 0, + "second call flips nothing — the row is already applied" + ); + } + + /// The reviewer's second proof: a `cancelled` row is never drained. + /// The drain's WHERE clause is on `state = 'applied'`, so a row that + /// never made it past receive_pack CANNOT become a push event, a + /// certificate, or an anchor handoff. + #[sqlx::test] + async fn cancelled_rows_are_not_returned_by_the_drain(pool: PgPool) { + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-3", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ref_update( + "refs/heads/main", + "a".repeat(40).as_str(), + "b".repeat(40).as_str(), + )], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + ) + .await + .unwrap(); + db.mark_pending_ref_transitions_cancelled("req-3") + .await + .unwrap(); + + let drained = db.list_pending_ref_transitions_applied(100).await.unwrap(); + assert!( + drained.is_empty(), + "a cancelled receive-pack must never reach the drain — the row's \ + state is `cancelled`, not `applied`, and the drain is keyed on `applied`" + ); + } + + /// Same proof, but for the pre-flip state. A `prepared` row (handler + /// crashed between `insert_prepared` and `mark_applied` / never + /// reached either post-receive branch) is also never drained. The + /// recovery cannot promote a `prepared` row by itself — only the + /// handler's post-Ok code does, by calling `mark_applied`. + #[sqlx::test] + async fn prepared_rows_are_not_returned_by_the_drain(pool: PgPool) { + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-4", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ref_update( + "refs/heads/main", + "a".repeat(40).as_str(), + "b".repeat(40).as_str(), + )], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + ) + .await + .unwrap(); + // No mark_applied / mark_cancelled call. The row stays `prepared`. + + let drained = db.list_pending_ref_transitions_applied(100).await.unwrap(); + assert!( + drained.is_empty(), + "a row the handler never reached the post-Ok branch for must not \ + be drained; only `mark_applied` flips a row, only the drain \ + picks up `applied` rows" + ); + } + + /// The reviewer's first proof (DB layer): a recovery re-pass on the + /// same `applied` row produces the same push event id, the same cert + /// id, and the same anchor job id, and the idempotent inserts all + /// collapse to no-ops. The drain deletes the row after the work + /// lands, so a third pass has nothing to do. + #[sqlx::test] + async fn drain_then_re_derive_is_idempotent(pool: PgPool) { + let db = db(pool).await; + let now = Utc::now().to_rfc3339(); + let row = PendingRefTransition { + id: super::deterministic_id(&[ + "pending_ref_transition", + "req-5", + "repo-1", + "refs/heads/main", + &"a".repeat(40), + &"b".repeat(40), + ]), + request_id: "req-5".to_string(), + repo_id: "repo-1".to_string(), + ref_name: "refs/heads/main".to_string(), + old_sha: "a".repeat(40), + new_sha: "b".repeat(40), + pusher_did: "did:key:pusher".to_string(), + node_did: "did:key:node".to_string(), + signature_header: "Signature: sig=...".to_string(), + signature_input: "Signature-Input: ...".to_string(), + content_digest: "Content-Digest: ...".to_string(), + state: pending_state::APPLIED.to_string(), + created_at: now.clone(), + applied_at: Some(now.clone()), + cancelled_at: None, + }; + db.insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + // First drain: picks up the row. Caller would now re-derive the + // artifacts; the row is then deleted. + let first = db.list_pending_ref_transitions_applied(100).await.unwrap(); + assert_eq!(first.len(), 1); + let push_id_1 = push_event_id_for(&row.request_id, &row.ref_name); + let cert_id_1 = ref_cert_id_for(&row.request_id, &row.ref_name); + let anchor_id_1 = + anchor_job_id_for(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha); + + // Second drain: row is still there (we did not delete). Re-derive + // the same ids; the inserts collapse. + let second = db.list_pending_ref_transitions_applied(100).await.unwrap(); + assert_eq!(second.len(), 1, "the row is still in `applied`"); + let push_id_2 = push_event_id_for(&row.request_id, &row.ref_name); + let cert_id_2 = ref_cert_id_for(&row.request_id, &row.ref_name); + let anchor_id_2 = + anchor_job_id_for(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha); + assert_eq!(push_id_1, push_id_2, "push id is deterministic"); + assert_eq!(cert_id_1, cert_id_2, "cert id is deterministic"); + assert_eq!(anchor_id_1, anchor_id_2, "anchor id is deterministic"); + + // Now exercise the idempotent inserts directly: a second + // `record_push_with_id` returns false, the cert insert returns + // None on the (repo_id, ref_name) unique, and the anchor insert + // returns false on the (repo_id, ref_name, old_sha, new_sha) + // unique. + assert!( + db.record_push_with_id(&push_id_1, &row.pusher_did, &row.repo_id, &row.new_sha, 0) + .await + .unwrap(), + "first push insert is created" + ); + assert!( + !db.record_push_with_id(&push_id_2, &row.pusher_did, &row.repo_id, &row.new_sha, 0) + .await + .unwrap(), + "second push insert with the same id collapses to a no-op" + ); + + // Anchor: one row, never two. + let job = AnchorJob { + id: anchor_id_1.clone(), + repo_id: row.repo_id.clone(), + ref_name: row.ref_name.clone(), + old_sha: row.old_sha.clone(), + new_sha: row.new_sha.clone(), + pusher_did: row.pusher_did.clone(), + created_at: now.clone(), + claimed_at: None, + }; + assert!(db.insert_anchor_job_idempotent(&job).await.unwrap()); + assert!( + !db.insert_anchor_job_idempotent(&job).await.unwrap(), + "a second anchor insert with the same id is a no-op" + ); + assert_eq!( + db.count_anchor_jobs(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha) + .await + .unwrap(), + 1, + "exactly one anchor job per transition, no matter how many recovery passes" + ); + assert_eq!( + db.count_push_events(&row.repo_id, &row.new_sha, &row.pusher_did) + .await + .unwrap(), + 1, + "exactly one push event per (repo, commit, pusher)" + ); + + // After the work lands, the drain deletes the row. A third pass + // sees nothing. + db.delete_pending_ref_transition(&row.id).await.unwrap(); + let third = db.list_pending_ref_transitions_applied(100).await.unwrap(); + assert!(third.is_empty(), "the row is gone after recovery"); + } + + /// `mark_cancelled` is also idempotent. The state predicate is + /// `state = 'prepared'`, so a second call flips nothing. + #[sqlx::test] + async fn mark_cancelled_is_idempotent_on_repeat(pool: PgPool) { + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-6", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ref_update( + "refs/heads/main", + "a".repeat(40).as_str(), + "b".repeat(40).as_str(), + )], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + ) + .await + .unwrap(); + assert_eq!( + db.mark_pending_ref_transitions_cancelled("req-6") + .await + .unwrap(), + 1 + ); + assert_eq!( + db.mark_pending_ref_transitions_cancelled("req-6") + .await + .unwrap(), + 0 + ); + } + + /// The `deterministic_id` helper uses an ASCII Unit Separator between + /// fields so that two distinct tuples can never collide by accidental + /// prefix overlap. `(a, bc)` and `(ab, c)` would otherwise hash the + /// same input. A regression on the separator shows up here. + #[test] + fn deterministic_id_avoids_prefix_overlap_collisions() { + let a = super::deterministic_id(&["a", "bc"]); + let b = super::deterministic_id(&["ab", "c"]); + assert_ne!(a, b, "the field separator must distinguish ab+bc from a+bc"); + } + + /// The push event id is stable across calls. The recovery drain + /// derives it the same way twice and gets the same value, which is + /// the entire reason for using a hash instead of a UUID. + #[test] + fn push_event_id_for_is_stable() { + assert_eq!( + push_event_id_for("req-x", "refs/heads/main"), + push_event_id_for("req-x", "refs/heads/main") + ); + assert_ne!( + push_event_id_for("req-x", "refs/heads/main"), + push_event_id_for("req-y", "refs/heads/main"), + "different request ids produce different push event ids" + ); + assert_ne!( + push_event_id_for("req-x", "refs/heads/main"), + push_event_id_for("req-x", "refs/heads/feature"), + "different refs produce different push event ids" + ); + } +} From 07109f4ec106a1dbd6211de79783ef89e3af7d94 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Fri, 28 Aug 2026 23:17:28 +0600 Subject: [PATCH 02/22] fix(node): wire durable outbox into the receive-pack handler (#26 split 1/4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the handler-level half of Split PR 1. The previous commit added the migration and the DB methods; this one threads them through crates/gitlawb-node/src/api/repos.rs:2007 (git_receive_pack), the cert issuer, and the startup drain. CHANGES IN THE HANDLER ====================== In git_receive_pack, AT THE LAST POSSIBLE MOMENT before the smart_http::receive_pack call, the handler now: 1. Generates a per-handler request_id (UUID). 2. Captures the raw Signature, Signature-Input, and Content-Digest headers from the request. 3. Calls db.insert_pending_ref_transitions(request_id, ...) which writes one row per ref update in state 'prepared'. The receive_pack call runs as before. After it returns: 4. On Ok: db.mark_pending_ref_transitions_applied(request_id) — the row is the ONLY thing that promotes a 'prepared' row to 'applied', and the drain reads only 'applied' rows. A process crash before this call leaves the row in 'prepared', which the drain never promotes. 5. On Err: db.mark_pending_ref_transitions_cancelled(request_id) — a failed receive_pack leaves the row in 'cancelled', which the drain never promotes. This is what closes the reviewer's two proofs: Proof 1 (crash window): if the process dies after mark_pending_ref_transitions_applied but before the bookkeeping writes, the row is in 'applied' and the next startup drain re-derives the push event, the per-ref certificate (carrying the ORIGINAL pusher DID, not a placeholder), and the anchor handoff. The drain uses the persisted authentic pusher DID and signature header, not a recovered placeholder. Proof 2 (failed receive-pack): the row is only ever flipped to 'applied' in the explicit Ok branch above. A 'prepared' or 'cancelled' row is invisible to the drain, so a failed or dropped receive_pack cannot turn a prepared intent into completed accounting or anchoring. BOOKKEEPING IS NOW DETERMINISTIC-ID =================================== The post-Ok bookkeeping at api/repos.rs:2448 now uses: - record_push_with_id with push_event_id_for(request_id, first_ref) — ON CONFLICT (id) DO NOTHING, so a recovery re-pass is a no-op. - issue_ref_certificate_idempotent with ref_cert_id_for(request_id, ref_name) — ON CONFLICT (repo_id, ref_name) DO NOTHING, returns None if a live-path cert already exists. - insert_anchor_job_idempotent with anchor_job_id_for(repo_id, ref_name, old_sha, new_sha) — the per-transition tuple key, so two pushes to the same ref produce one anchor upload per landed state. The legacy entry points (record_push, issue_ref_certificate, insert_ref_certificate) remain for callers that prefer a fresh UUID per cert; they are #[allow(dead_code)] for the PR 3 cert/CLI compat pass to decide whether to keep or remove. STARTUP DRAIN ============= crates/gitlawb-node/src/main.rs calls durable_outbox::drain_pending_ref_transitions(state, 1000) ONCE before serving, after migrations and after the existing peer / quarantine prunes. Non-fatal: a transient drain failure logs and leaves the rows for the next startup. durable_outbox::drain_pending_ref_transitions reads every 'applied' row, calls derive_one (which re-derives the three artifacts using the persisted authentic pusher DID and signature header), then deletes the row. A second drain pass is a no-op for both the artifacts (idempotent inserts) and the row (gone after the first pass). NEW END-TO-END TESTS ==================== crates/gitlawb-node/src/durable_outbox.rs adds three end-to-end tests in drain_tests, complementing the eight DB-layer tests in db::pending_ref_transition_tests: - drain_re_derives_all_three_artifacts_for_an_applied_row: the reviewer's first proof. Inserts a row in 'applied' state (the crash window), drains, asserts exactly one push event row, exactly one cert row carrying the original pusher DID (not a placeholder), and exactly one anchor job row. Asserts the deterministic cert id matches. Asserts a second drain pass is a no-op. - cancelled_row_produces_no_artifacts: the reviewer's second proof for the cancelled state. A row in 'cancelled' (receive_pack returned Err) is invisible to the drain. - prepared_row_produces_no_artifacts: the reviewer's second proof for the prepared state. A row in 'prepared' (handler crashed between insert_prepared and the post-Ok branch) is invisible to the drain. Each test names the invariant it pins and the production line it covers. Reverting that line turns the named assertion red. Compiles clean, 1099 tests pass with 0 regressions, clippy clean under -D warnings, fmt clean. Cross-PR overlap (declared in the PR description): - #134 (anchors auth): composes. The /arweave/anchors route already requires auth; this PR does not change the route. - #285 (advisory-lock session affinity): composes. The durable intent is written inside the same handler that holds the lock from #285; no changes to the lock layer. - #306 (Content-Digest on signed requests): composes. PR 1 persists the Content-Digest header that #306 makes mandatory. - #314 (small-order Ed25519): independent. PR 1's tests use strong keys. - #324 (libp2p keypair persistence): independent. PR 1 does not touch p2p identity. - #325 (gossip ref-update auth): independent. PR 1's signed envelope is the HTTP-side equivalent, not the gossip-side. - #382 (replication withheld-subtree trees): independent. PR 1 does not touch replication or pin selection. --- crates/gitlawb-node/src/api/repos.rs | 150 +++++++++- crates/gitlawb-node/src/cert.rs | 70 ++++- crates/gitlawb-node/src/db/mod.rs | 2 + crates/gitlawb-node/src/durable_outbox.rs | 348 ++++++++++++++++++++++ crates/gitlawb-node/src/main.rs | 16 + 5 files changed, 576 insertions(+), 10 deletions(-) create mode 100644 crates/gitlawb-node/src/durable_outbox.rs diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 896b80e13..9d3153aa2 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2245,6 +2245,68 @@ pub async fn git_receive_pack( let admission = smart_http::AdmissionGuard::new(_permit, _caller_permit) .with_hold(std::sync::Arc::clone(&guard)) .with_lease(lease.clone()); + + // #26 Split PR 1: durable intent for this push, written BEFORE + // the receive_pack call. Every ref update the pusher intends to + // land gets a `prepared` row carrying the verified pusher DID, + // the raw RFC 9421 signature header, signature-input, and + // content-digest that authorized the push, plus the request id. + // + // The state is flipped to `applied` (Ok) or `cancelled` (Err) + // AFTER receive_pack returns. The drain reads only `applied` + // rows, so a row that never gets the post-Ok flip stays in + // `prepared` (handler crash / dropped future) or `cancelled` + // (receive_pack Err) and is never promoted to a push event, a + // certificate, or an anchor. + // + // Inserted AT THE LAST POSSIBLE MOMENT, immediately before the + // receive_pack call, so a rejection above (owner enforcement, + // branch protection, etc.) does not produce a `prepared` row + // that nothing will ever flip. + let request_id = uuid::Uuid::new_v4().to_string(); + let signature_header = headers + .get("signature") + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + let signature_input = headers + .get("signature-input") + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + let content_digest = headers + .get("content-digest") + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + if let Err(e) = state + .db + .insert_pending_ref_transitions( + &request_id, + &record.id, + &state.node_did.to_string(), + auth.0.as_str(), + &ref_updates, + &signature_header, + &signature_input, + &content_digest, + ) + .await + { + // A durable-intent write failure here means we cannot + // guarantee recovery for the upcoming git apply. Refuse the + // push with 503 rather than risk a ref landing with no + // recovery record. + tracing::error!( + err = %e, + repo = %name, + "failed to persist durable post-receive intent; refusing push" + ); + return Err(AppError::Overloaded( + "durable intent write failed, retry shortly".into(), + )); + } + let receive_result = smart_http::receive_pack( &state.git_bin, &disk_path, @@ -2254,6 +2316,42 @@ pub async fn git_receive_pack( ) .await; + // #26 Split PR 1: state flip. The drain's WHERE clause keys on + // `state = 'applied'`, so this is the ONLY line that promotes a + // `prepared` row. A row that lands in this branch is a ref that + // Git already applied to disk; the recovery drain will re-derive + // the push event, the per-ref certificate, and the anchor + // handoff from it on the next startup. + if receive_result.is_ok() { + if let Err(e) = state + .db + .mark_pending_ref_transitions_applied(&request_id) + .await + { + tracing::error!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions applied; recovery will re-derive" + ); + // Don't fail the push — the ref is on disk and the drain + // will pick it up on the next startup regardless. + } + } else { + if let Err(e) = state + .db + .mark_pending_ref_transitions_cancelled(&request_id) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions cancelled" + ); + } + } + // #174 F2/U5: the post-receive replication tail runs in an independently owned // task. It parks on `git_encrypt_semaphore` (withheld / candidate / full-scan // resolution), so leaving it in the request future means a client/proxy disconnect @@ -2350,6 +2448,11 @@ pub async fn git_receive_pack( // Record push event for trust score and issue a signed ref certificate. // The route is behind `require_signature`, so the verified pusher identity is // always present; use it directly rather than re-parsing the headers. + // + // #26 Split PR 1: the push event id, the per-ref cert id, and the + // anchor job id are all derived from the same `request_id` captured + // above, so a recovery re-pass against the same transition + // produces the same primary keys and the idempotent inserts collapse. let did = auth.0.as_str(); { // Use the first new commit hash we parsed, fall back to timestamp @@ -2358,7 +2461,19 @@ pub async fn git_receive_pack( .map(|u| u.new_sha.clone()) .unwrap_or_else(|| Utc::now().timestamp().to_string()); - let _ = state.db.record_push(did, &record.id, &commit_hash, 0).await; + // The push event is keyed on the FIRST ref's name so a + // multi-ref push collapses to one push event row, not N. The + // deterministic id is the same one the recovery drain + // derives. + let first_ref_name = ref_updates + .first() + .map(|u| u.ref_name.clone()) + .unwrap_or_else(|| "refs/heads/main".to_string()); + let push_event_id = crate::db::push_event_id_for(&request_id, &first_ref_name); + let _ = state + .db + .record_push_with_id(&push_event_id, did, &record.id, &commit_hash, 0) + .await; if let Ok(push_count) = state.db.get_push_count(did).await { // 0.05 base (from registration) + 0.05 per push, capped at 1.0 // 1 push → 0.10, 5 pushes → 0.30, 19 pushes → 1.0 @@ -2370,23 +2485,52 @@ pub async fn git_receive_pack( // carrying that ref's real old→new transition. A multi-ref push must // not collapse to a single cert covering only the first ref. for update in &ref_updates { - match cert::issue_ref_certificate( + let cert_id = crate::db::ref_cert_id_for(&request_id, &update.ref_name); + match cert::issue_ref_certificate_idempotent( &state, &record.id, &update.ref_name, &update.old_sha, &update.new_sha, did, + &cert_id, ) .await { - Ok(c) => { + Ok(Some(c)) => { tracing::info!(cert_id = %c.id, repo = %record.name, ref_name = %update.ref_name, pusher = %did, "issued ref certificate") } + Ok(None) => { + tracing::debug!(ref_name = %update.ref_name, repo = %record.name, "ref certificate already exists for this ref, idempotent skip") + } Err(e) => { tracing::warn!(err = %e, ref_name = %update.ref_name, "failed to issue ref certificate") } } + + // Anchor handoff: insert an anchor_jobs row keyed on the + // per-transition tuple. PR 2 reads this row and uploads + // to the bundler. The deterministic id makes a recovery + // re-pass a no-op. + let anchor_id = crate::db::anchor_job_id_for( + &record.id, + &update.ref_name, + &update.old_sha, + &update.new_sha, + ); + let job = crate::db::AnchorJob { + id: anchor_id, + repo_id: record.id.clone(), + ref_name: update.ref_name.clone(), + old_sha: update.old_sha.clone(), + new_sha: update.new_sha.clone(), + pusher_did: did.to_string(), + created_at: Utc::now().to_rfc3339(), + claimed_at: None, + }; + if let Err(e) = state.db.insert_anchor_job_idempotent(&job).await { + tracing::warn!(err = %e, ref_name = %update.ref_name, "failed to enqueue anchor job") + } } } diff --git a/crates/gitlawb-node/src/cert.rs b/crates/gitlawb-node/src/cert.rs index 0ed50418e..6b05674c1 100644 --- a/crates/gitlawb-node/src/cert.rs +++ b/crates/gitlawb-node/src/cert.rs @@ -15,6 +15,15 @@ use crate::state::AppState; /// /// Builds a canonical JSON payload, signs it with the node's Ed25519 key, /// persists the certificate, and returns it. +/// +/// #26 Split PR 1: the live handler now uses +/// [`issue_ref_certificate_idempotent`] so the cert id is deterministic +/// and recovery re-derives the same primary key. This legacy entry +/// point remains for callers that prefer a fresh UUID per cert (it +/// keeps the older `insert_ref_certificate` upsert semantics); Split +/// PR 3 owns the cert/CLI compatibility decision of whether to keep +/// it or remove it. +#[allow(dead_code)] // kept for the PR 3 cert/CLI compat pass pub async fn issue_ref_certificate( state: &AppState, repo_id: &str, @@ -22,6 +31,56 @@ pub async fn issue_ref_certificate( old_sha: &str, new_sha: &str, pusher_did: &str, +) -> Result { + let cert = + build_ref_certificate(state, repo_id, ref_name, old_sha, new_sha, pusher_did, None).await?; + state.db.insert_ref_certificate(&cert).await +} + +/// #26 Split PR 1 — idempotent variant used by the recovery drain. +/// +/// `cert_id` is the deterministic id derived from +/// `(request_id, ref_name)` so a recovery re-pass against the same +/// transition produces the same primary key. The insert uses +/// `ON CONFLICT (repo_id, ref_name) DO NOTHING` (the existing +/// `insert_ref_certificate_idempotent` helper), so the function +/// returns `None` if a live-path cert already exists for the +/// `(repo_id, ref_name)` pair, and `Some(cert)` if it wrote a new +/// one. Either way, exactly one cert row exists for the transition. +pub async fn issue_ref_certificate_idempotent( + state: &AppState, + repo_id: &str, + ref_name: &str, + old_sha: &str, + new_sha: &str, + pusher_did: &str, + cert_id: &str, +) -> Result> { + let cert = build_ref_certificate( + state, + repo_id, + ref_name, + old_sha, + new_sha, + pusher_did, + Some(cert_id.to_string()), + ) + .await?; + state.db.insert_ref_certificate_idempotent(&cert).await +} + +/// Shared cert construction: build the JSON payload, sign it with the +/// node key, and assemble the `RefCertificate` row. `cert_id_override` +/// lets the recovery path plug in a deterministic id; the live path +/// passes `None` and gets a fresh UUID. +async fn build_ref_certificate( + state: &AppState, + repo_id: &str, + ref_name: &str, + old_sha: &str, + new_sha: &str, + pusher_did: &str, + cert_id_override: Option, ) -> Result { let node_did = state.node_did.to_string(); let issued_at = Utc::now().to_rfc3339(); @@ -40,8 +99,9 @@ pub async fn issue_ref_certificate( let signature = state.node_keypair.sign_b64(&payload_bytes); - let cert = RefCertificate { - id: Uuid::new_v4().to_string(), + let id = cert_id_override.unwrap_or_else(|| Uuid::new_v4().to_string()); + Ok(RefCertificate { + id, repo_id: repo_id.to_string(), ref_name: ref_name.to_string(), old_sha: old_sha.to_string(), @@ -50,9 +110,5 @@ pub async fn issue_ref_certificate( node_did, signature, issued_at, - }; - - // Persist and return the row as it exists in the database (on a - // conflict the existing row survives when it is newer). - state.db.insert_ref_certificate(&cert).await + }) } diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 88b8e5297..eff728e4f 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -1966,6 +1966,7 @@ impl Db { Ok(()) } + #[allow(dead_code)] // legacy live-path entry; PR 3 owns the deprecation decision pub async fn record_push( &self, agent_did: &str, @@ -2539,6 +2540,7 @@ impl Db { /// late-landing older cert cannot regress a ref's persisted state. Returns /// the full row as it now exists in the database (the original row on a /// rejected upsert; the passed row on insert). + #[allow(dead_code)] // legacy live-path entry; PR 3 owns the deprecation decision pub async fn insert_ref_certificate(&self, cert: &RefCertificate) -> Result { let row = sqlx::query( "INSERT INTO ref_certificates diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs new file mode 100644 index 000000000..46b3855af --- /dev/null +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -0,0 +1,348 @@ +//! #26 Split PR 1 — durable post-receive outbox: the recovery drain. +//! +//! This module owns the STARTUP drain for `pending_ref_transitions`. It +//! iterates every row in state `applied`, re-derives the push event, the +//! per-ref certificate, and the anchor handoff using the ORIGINAL pusher +//! DID and signature header that was persisted BEFORE the receive-pack +//! call landed the ref, and then deletes the row. +//! +//! The drain is invoked once at startup, after migrations and before +//! serving, in [`crate::main`]. It is also the function the failure- +//! injection end-to-end test calls to simulate a "node restart" after +//! the crash window the reviewer flagged. +//! +//! Idempotency is delegated to the DB layer. The push event and anchor +//! job use `ON CONFLICT (id) DO NOTHING` keyed on the deterministic +//! `(request_id, ref_name)` / `(repo_id, ref_name, old_sha, new_sha)` +//! id. The ref certificate uses +//! `insert_ref_certificate_idempotent`, which checks the unique +//! `(repo_id, ref_name)` index and returns `None` if a live-path cert +//! already exists. Re-running the drain against the same row is +//! therefore a no-op for the artifact writes; the row deletion at the +//! end is also idempotent because a missing `id` simply affects zero +//! rows. + +use crate::cert; +use crate::db::PendingRefTransition; +use crate::state::AppState; + +/// One drain pass. Returns the number of transitions re-derived. Called +/// from startup and from the failure-injection test. +/// +/// `limit` bounds the work per call. The startup caller passes a +/// generous cap (1000); tests pass a small one to assert behavior +/// without flooding the database. +pub async fn drain_pending_ref_transitions(state: AppState, limit: i64) -> anyhow::Result { + let rows = state.db.list_pending_ref_transitions_applied(limit).await?; + let mut count = 0; + for row in rows { + derive_one(&state, &row).await?; + state.db.delete_pending_ref_transition(&row.id).await?; + count += 1; + } + Ok(count) +} + +/// Re-derive the push event, the per-ref certificate, and the anchor +/// handoff for one `applied` row, using the persisted authentic pusher +/// identity. This is what closes the reviewer's invariant: the +/// recovered artifacts carry the original pusher DID, not a +/// placeholder. +/// +/// The push event id and ref certificate id are derived from +/// `(request_id, ref_name)`; the anchor job id from +/// `(repo_id, ref_name, old_sha, new_sha)`. All three inserts are +/// idempotent (see the module-level comment), so a second drain pass +/// against the same row is a no-op. +pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow::Result<()> { + // Push event: deterministic id, idempotent insert. + let push_id = crate::db::push_event_id_for(&row.request_id, &row.ref_name); + state + .db + .record_push_with_id(&push_id, &row.pusher_did, &row.repo_id, &row.new_sha, 0) + .await?; + + // Ref certificate: the cert is signed by the node, but the + // `pusher_did` field carries the ORIGINAL authenticated pusher. The + // idempotent insert returns None if a live-path cert already + // exists, in which case we leave it alone. + let cert_id = crate::db::ref_cert_id_for(&row.request_id, &row.ref_name); + let _ = cert::issue_ref_certificate_idempotent( + state, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + &row.pusher_did, + &cert_id, + ) + .await?; + + // Anchor handoff: the durable queue PR 2 reads from. Idempotent + // on the per-transition id; at most one row per landed state. + let anchor_id = + crate::db::anchor_job_id_for(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha); + let job = crate::db::AnchorJob { + id: anchor_id, + repo_id: row.repo_id.clone(), + ref_name: row.ref_name.clone(), + old_sha: row.old_sha.clone(), + new_sha: row.new_sha.clone(), + pusher_did: row.pusher_did.clone(), + created_at: chrono::Utc::now().to_rfc3339(), + claimed_at: None, + }; + state.db.insert_anchor_job_idempotent(&job).await?; + + Ok(()) +} + +#[cfg(test)] +mod drain_tests { + //! End-to-end failure-injection test the reviewer demanded: + //! + //! "Inject failure after Git applies the ref but before the first + //! transition/job write, restart the node, and show that the + //! original transition produces exactly one push event, one + //! certificate carrying the original pusher/proof, and at most + //! one anchor upload." + //! + //! The crash window is simulated by inserting a + //! `pending_ref_transitions` row directly in `applied` state + //! (bypassing the handler). The drain then re-derives the three + //! artifacts using the persisted authentic pusher DID and the + //! raw RFC 9421 signature header. Assertions check the invariants + //! the reviewer named: exactly one push event row, exactly one + //! cert row carrying the original pusher, exactly one anchor job + //! row. A second drain pass is a no-op. + //! + //! Each assertion names the invariant it pins. Reverting the + //! production line under test turns the named assertion red. + + use super::*; + use crate::db::pending_state; + use crate::db::Db; + use crate::db::PendingRefTransition; + use chrono::Utc; + + async fn _db(pool: sqlx::PgPool) -> Db { + let db = Db::for_testing(pool); + db.run_migrations().await.unwrap(); + db + } + + fn make_row(repo_id: &str, ref_name: &str, old: &str, new: &str) -> PendingRefTransition { + let now = Utc::now().to_rfc3339(); + PendingRefTransition { + id: crate::db::deterministic_id(&[ + "pending_ref_transition", + "req-1", + repo_id, + ref_name, + old, + new, + ]), + request_id: "req-1".to_string(), + repo_id: repo_id.to_string(), + ref_name: ref_name.to_string(), + old_sha: old.to_string(), + new_sha: new.to_string(), + pusher_did: "did:key:z6pusher".to_string(), + node_did: "did:key:z6node".to_string(), + signature_header: "Signature: sig=\"abc...\"".to_string(), + signature_input: "Signature-Input: sig=(\"@authority\");...".to_string(), + content_digest: "Content-Digest: sha-256=:...:".to_string(), + state: pending_state::APPLIED.to_string(), + created_at: now.clone(), + applied_at: Some(now), + cancelled_at: None, + } + } + + /// The reviewer's proof at the durable-outbox layer. Insert a row + /// in `applied` state (the crash window), drain, and assert + /// exactly one push event, one cert with the original pusher, + /// and one anchor job. + #[sqlx::test] + async fn drain_re_derives_all_three_artifacts_for_an_applied_row(pool: sqlx::PgPool) { + // Pre-create the repo so the FK-ish usage in tests doesn't blow up. + // The drain itself does not require a repo row to exist; the test + // only checks the derived artifacts. + let state = crate::test_support::test_state(pool).await; + + let repo_id = "repo-failure-injection"; + let ref_name = "refs/heads/main"; + let old = "a".repeat(40); + let new = "b".repeat(40); + let row = make_row(repo_id, ref_name, &old, &new); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 1, "exactly one transition re-derived"); + + // Push event: exactly one row, keyed on the deterministic id. + let _push_id = crate::db::push_event_id_for(&row.request_id, &row.ref_name); + let push_count = state + .db + .count_push_events(&row.repo_id, &row.new_sha, &row.pusher_did) + .await + .unwrap(); + assert_eq!( + push_count, 1, + "exactly one push event, keyed on the original pusher" + ); + + // Cert: exactly one row, carrying the original pusher DID. + let certs = state + .db + .list_ref_certificates(&row.repo_id, 10) + .await + .unwrap(); + assert_eq!(certs.len(), 1, "exactly one ref certificate"); + assert_eq!( + certs[0].pusher_did, row.pusher_did, + "cert carries the original pusher DID, not a placeholder" + ); + assert_eq!( + certs[0].id, + crate::db::ref_cert_id_for(&row.request_id, &row.ref_name), + "cert id is deterministic" + ); + assert_eq!(certs[0].new_sha, row.new_sha, "cert carries the new_sha"); + assert_eq!(certs[0].old_sha, row.old_sha, "cert carries the old_sha"); + + // Anchor job: exactly one row. + let anchor_count = state + .db + .count_anchor_jobs(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha) + .await + .unwrap(); + assert_eq!(anchor_count, 1, "exactly one anchor job per transition"); + + // The drain deleted the row. + let after = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert!( + after.is_empty(), + "drain deletes the row after the work lands" + ); + + // A second drain pass is a no-op. + let n2 = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n2, 0, "a second drain pass has nothing to do"); + } + + /// The reviewer's second proof, end-to-end. A `cancelled` row is + /// NEVER promoted by the drain. The drain only re-derives + /// artifacts for `applied` rows; a row that was `cancelled` + /// because receive_pack returned Err stays cancelled, and no + /// push event, cert, or anchor is created. + #[sqlx::test] + async fn cancelled_row_produces_no_artifacts(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let mut row = make_row( + "repo-cancel", + "refs/heads/main", + &"a".repeat(40), + &"b".repeat(40), + ); + row.state = pending_state::CANCELLED.to_string(); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 0, "the drain must not promote a cancelled row"); + + // No push event, no cert, no anchor. + let push_count = state + .db + .count_push_events(&row.repo_id, &row.new_sha, &row.pusher_did) + .await + .unwrap(); + assert_eq!(push_count, 0); + let certs = state + .db + .list_ref_certificates(&row.repo_id, 10) + .await + .unwrap(); + assert!(certs.is_empty()); + let anchor_count = state + .db + .count_anchor_jobs(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha) + .await + .unwrap(); + assert_eq!(anchor_count, 0); + + // The cancelled row is also left untouched. + let still = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert!(still.is_empty(), "drain reads only applied rows"); + } + + /// A `prepared` row that the handler never reached the post-Ok + /// branch for (e.g. process crash between insert_prepared and + /// mark_applied) is also never promoted. The drain reads only + /// `applied` rows, so a `prepared` row stays in `prepared` and + /// is invisible to the drain. + #[sqlx::test] + async fn prepared_row_produces_no_artifacts(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let mut row = make_row( + "repo-prep", + "refs/heads/main", + &"a".repeat(40), + &"b".repeat(40), + ); + row.state = pending_state::PREPARED.to_string(); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 0, "the drain must not promote a prepared row"); + + let push_count = state + .db + .count_push_events(&row.repo_id, &row.new_sha, &row.pusher_did) + .await + .unwrap(); + assert_eq!(push_count, 0); + let certs = state + .db + .list_ref_certificates(&row.repo_id, 10) + .await + .unwrap(); + assert!(certs.is_empty()); + let anchor_count = state + .db + .count_anchor_jobs(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha) + .await + .unwrap(); + assert_eq!(anchor_count, 0); + } +} diff --git a/crates/gitlawb-node/src/main.rs b/crates/gitlawb-node/src/main.rs index 66bfa0961..d4ae89b41 100644 --- a/crates/gitlawb-node/src/main.rs +++ b/crates/gitlawb-node/src/main.rs @@ -5,6 +5,7 @@ mod bootstrap; mod cert; mod config; mod db; +mod durable_outbox; mod encrypted_pin; mod error; mod git; @@ -676,6 +677,21 @@ async fn main() -> Result<()> { let grace = std::time::Duration::from_secs(config.shutdown_grace_secs); info!(grace_secs = config.shutdown_grace_secs, "axum server ready"); + // #26 Split PR 1: drain any `applied` rows left by a previous + // process that crashed after Git applied a ref but before the + // bookkeeping landed. Runs once, BEFORE the server accepts new + // pushes, so a recovery re-derivation does not race a fresh push. + // Non-fatal: a transient drain failure is logged and the rows + // remain `applied` for the next startup to pick up. + match durable_outbox::drain_pending_ref_transitions(state.clone(), 1000).await { + Ok(0) => {} + Ok(n) => info!(n, "drained pending ref transitions from prior run"), + Err(e) => warn!( + err = %e, + "pending ref transition drain failed at startup (non-fatal; will retry on next start)" + ), + } + // `into_make_service_with_connect_info` exposes the socket peer address as // `ConnectInfo` so the push limiter can key on the real client // when no trusted proxy header applies (see `rate_limit::client_key`). From 1fa9a1f8dd41a1cff40c694b3408a4944ec3b553 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Sat, 29 Aug 2026 20:49:40 +0600 Subject: [PATCH 03/22] fix(node): address four reviewer findings on #26 split 1/4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - P1-A: add startup reconcile step that promotes `prepared` rows to `applied` when the on-disk ref matches the row's `new_sha`. The recovery drain (which only reads `applied` rows) can now pick up a ref that landed when the live handler's `mark_pending_ref_transitions_applied` call errored or was interrupted. Strict SHA equality is the load-bearing check — a `prepared` row whose target did NOT actually land stays `prepared`. - P1-B: route the live handler's cert issuance through `cert::issue_ref_certificate` (the upsert) instead of `issue_ref_certificate_idempotent` (DO NOTHING). A re-push to the same ref now updates the cert's `old_sha` / `new_sha` / `pusher_did` / `issued_at` / `signature` to the new transition while preserving the deterministic `cert_id`. The recovery drain keeps the idempotent variant; both paths collapse to one row. - P2-A: refactor the drain into a `drain_pending_ref_transitions_with` testable seam that does per-row log-and-continue, and add `drain_pending_ref_transitions_all` that loops `DRAIN_PER_PASS_LIMIT=1000` rows for `DRAIN_MAX_PASSES=10` passes. A failing row no longer stalls the batch; a backlog above 1000 rows is fully processed across passes. - P2-B: add a `first_ref_name` column to `pending_ref_transitions` via migration v28. The live handler hoists a `first_ref_name` local and persists it on every row of the same `request_id`. The drain's `derive_one` keys the push event id on `row.first_ref_name` instead of `row.ref_name`, so live and recovery produce the same id and `ON CONFLICT (id) DO NOTHING` collapses a multi-ref push to one push event row (and one trust- score bump). Cert and anchor ids stay per-ref / per-transition. --- crates/gitlawb-node/src/api/repos.rs | 49 +- crates/gitlawb-node/src/cert.rs | 54 +- crates/gitlawb-node/src/db/mod.rs | 243 ++++- crates/gitlawb-node/src/durable_outbox.rs | 1062 ++++++++++++++++++++- crates/gitlawb-node/src/main.rs | 29 +- 5 files changed, 1398 insertions(+), 39 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 9d3153aa2..4395496a1 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2279,6 +2279,20 @@ pub async fn git_receive_pack( .and_then(|v| v.to_str().ok()) .unwrap_or("") .to_string(); + // The first ref name is hoisted here so it is the SAME value + // persisted on every `pending_ref_transitions` row of this request + // (request-scoped) AND the SAME value used to derive the push event + // id below. The recovery drain reads `first_ref_name` from each row + // and keys `push_event_id_for` on it, so the live and recovery + // paths produce the same id and the ON CONFLICT (id) DO NOTHING in + // `record_push_with_id` collapses a live push followed by a recovery + // pass into a single push event row. An empty `ref_updates` is + // defensive: receive-pack on a push with no refs would have already + // been rejected upstream, so this branch is unreachable in practice. + let first_ref_name = ref_updates + .first() + .map(|u| u.ref_name.clone()) + .unwrap_or_default(); if let Err(e) = state .db .insert_pending_ref_transitions( @@ -2290,6 +2304,7 @@ pub async fn git_receive_pack( &signature_header, &signature_input, &content_digest, + &first_ref_name, ) .await { @@ -2464,11 +2479,12 @@ pub async fn git_receive_pack( // The push event is keyed on the FIRST ref's name so a // multi-ref push collapses to one push event row, not N. The // deterministic id is the same one the recovery drain - // derives. - let first_ref_name = ref_updates - .first() - .map(|u| u.ref_name.clone()) - .unwrap_or_else(|| "refs/heads/main".to_string()); + // derives, because the drain reads `first_ref_name` from the + // outbox row (persisted above on every row of this request) and + // uses it in the same `push_event_id_for` call. The outer + // `first_ref_name` local was hoisted above the + // `insert_pending_ref_transitions` call so this id matches + // the persisted value exactly. let push_event_id = crate::db::push_event_id_for(&request_id, &first_ref_name); let _ = state .db @@ -2484,9 +2500,25 @@ pub async fn git_receive_pack( // Issue a signed certificate for every ref this push advanced, each // carrying that ref's real old→new transition. A multi-ref push must // not collapse to a single cert covering only the first ref. + // + // P1-B: this routes through `issue_ref_certificate` (the + // upsert, `ON CONFLICT (repo_id, ref_name) DO UPDATE`) so a + // re-push to the same ref updates the row's + // `old_sha` / `new_sha` / `pusher_did` / `issued_at` / + // `signature` to the new transition while preserving the + // original `id` (the `insert_ref_certificate` upsert's + // `EXCLUDED.issued_at > ref_certificates.issued_at` guard). + // The previous `issue_ref_certificate_idempotent` call + // (DO NOTHING) left the first cert's fields frozen on every + // later push. The deterministic `cert_id` makes a recovery + // re-pass safe: the recovery's `insert_ref_certificate_idempotent` + // (DO NOTHING) is a no-op when the live handler has already + // written a row, and the live handler's upsert preserves the + // original `id` so the deterministic id survives across the + // push / recovery / re-push cycle. for update in &ref_updates { let cert_id = crate::db::ref_cert_id_for(&request_id, &update.ref_name); - match cert::issue_ref_certificate_idempotent( + match cert::issue_ref_certificate( &state, &record.id, &update.ref_name, @@ -2497,12 +2529,9 @@ pub async fn git_receive_pack( ) .await { - Ok(Some(c)) => { + Ok(c) => { tracing::info!(cert_id = %c.id, repo = %record.name, ref_name = %update.ref_name, pusher = %did, "issued ref certificate") } - Ok(None) => { - tracing::debug!(ref_name = %update.ref_name, repo = %record.name, "ref certificate already exists for this ref, idempotent skip") - } Err(e) => { tracing::warn!(err = %e, ref_name = %update.ref_name, "failed to issue ref certificate") } diff --git a/crates/gitlawb-node/src/cert.rs b/crates/gitlawb-node/src/cert.rs index 6b05674c1..a68ae12e7 100644 --- a/crates/gitlawb-node/src/cert.rs +++ b/crates/gitlawb-node/src/cert.rs @@ -11,19 +11,38 @@ use uuid::Uuid; use crate::db::RefCertificate; use crate::state::AppState; -/// Issue a signed ref-update certificate for a successful push. +/// Issue a signed ref-update certificate for a successful push. The +/// live receive-pack handler calls this on every successful push. /// -/// Builds a canonical JSON payload, signs it with the node's Ed25519 key, -/// persists the certificate, and returns it. +/// `cert_id` is the deterministic id derived from `(request_id, +/// ref_name)` (see [`crate::db::ref_cert_id_for`]). It is required so +/// the recovery drain and the live handler produce the same primary +/// key: a live push followed by a recovery pass collapses to a +/// single cert row, and a re-push to the same `(repo, ref)` updates +/// the existing row's `old_sha` / `new_sha` / `pusher_did` / +/// `issued_at` / `signature` to the new transition while preserving +/// the original `id` (the `insert_ref_certificate` upsert is +/// keyed on `(repo_id, ref_name)` and only updates fields when the +/// new `issued_at` is strictly greater). /// -/// #26 Split PR 1: the live handler now uses -/// [`issue_ref_certificate_idempotent`] so the cert id is deterministic -/// and recovery re-derives the same primary key. This legacy entry -/// point remains for callers that prefer a fresh UUID per cert (it -/// keeps the older `insert_ref_certificate` upsert semantics); Split -/// PR 3 owns the cert/CLI compatibility decision of whether to keep -/// it or remove it. -#[allow(dead_code)] // kept for the PR 3 cert/CLI compat pass +/// #26 Split PR 1 P1-B: the live handler routes through this +/// function (the upsert), NOT through +/// [`issue_ref_certificate_idempotent`] (DO NOTHING). The +/// idempotent variant is reserved for the recovery drain. Both +/// paths use the same deterministic `cert_id` so a re-pass is +/// always safe: +/// +/// - Live handler → live upsert: re-push updates the row, preserves +/// the original `id`. The contract pinned by +/// `insert_ref_certificate_upserts_on_repo_ref` is restored. +/// - Live handler → recovery: live's `ON CONFLICT (id) DO UPDATE` +/// preserves the original `id`; the recovery's +/// `ON CONFLICT (repo_id, ref_name) DO NOTHING` is a no-op. +/// - Recovery → live handler: the recovery wrote a row with the +/// deterministic `id`; the live upsert (which preserves `id` and +/// only updates other fields when `issued_at` is strictly newer) +/// is a no-op for an equal-`issued_at` re-run and a refresh for +/// a strictly-newer one. pub async fn issue_ref_certificate( state: &AppState, repo_id: &str, @@ -31,9 +50,18 @@ pub async fn issue_ref_certificate( old_sha: &str, new_sha: &str, pusher_did: &str, + cert_id: &str, ) -> Result { - let cert = - build_ref_certificate(state, repo_id, ref_name, old_sha, new_sha, pusher_did, None).await?; + let cert = build_ref_certificate( + state, + repo_id, + ref_name, + old_sha, + new_sha, + pusher_did, + Some(cert_id.to_string()), + ) + .await?; state.db.insert_ref_certificate(&cert).await } diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index eff728e4f..ec80227cd 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -200,6 +200,20 @@ pub struct PendingRefTransition { pub created_at: String, pub applied_at: Option, pub cancelled_at: Option, + /// The first ref name in the live push's `ref_updates`, persisted on + /// every row so the recovery drain can reproduce the live path's + /// "one push event per push, keyed on the first ref" cardinality. A + /// multi-ref push writes N rows; the recovery drain must NOT emit N + /// push events (one per `ref_name`), which would inflate + /// `get_push_count` and the trust score. The cert and anchor ids + /// stay per-ref / per-transition — only the push event id is + /// request-scoped. + /// + /// Migration v28 added this column with `NOT NULL DEFAULT ''` and a + /// backfill `UPDATE` that copies `ref_name` into `first_ref_name` + /// for every historic row. The live handler now passes the request's + /// actual first ref name explicitly. + pub first_ref_name: String, } /// #26 Split PR 1 — anchor handoff row, owned by PR 1, consumed by PR 2. @@ -1329,6 +1343,31 @@ const MIGRATIONS: &[Migration] = &[ "CREATE INDEX IF NOT EXISTS idx_anchor_jobs_claimed_at ON anchor_jobs (claimed_at, id)", ], }, + Migration { + version: 28, + name: "pending_ref_transitions_add_first_ref_name", + stmts: &[ + // #26 Split PR 1 P2-B: the recovery drain must reproduce the + // live path's push event cardinality, which is "one push event + // per push, keyed on the first ref name". Without a persisted + // `first_ref_name` column, the drain would key each outbox + // row on its own `ref_name` and emit N push events for an + // N-ref push, over-counting `get_push_count` and inflating + // the trust score. + // + // The `NOT NULL DEFAULT ''` is required to add a NOT NULL + // column to a non-empty table in a single ALTER; a follow-up + // UPDATE backfills the value to `ref_name` for every + // historic row. Old rows in `applied` state that the drain + // processes will produce one push event per row (the + // pre-fix cardinality) — an accepted upgrade-window quirk + // with no historic state to regress. New rows written by + // the live handler always carry the request's actual + // `first_ref_name`. + "ALTER TABLE pending_ref_transitions ADD COLUMN IF NOT EXISTS first_ref_name TEXT NOT NULL DEFAULT ''", + "UPDATE pending_ref_transitions SET first_ref_name = ref_name WHERE first_ref_name = ''", + ], + }, ]; /// Max distinct source repos recorded per pinned object (F1, #173 jatmn round 8). @@ -2606,6 +2645,13 @@ impl Db { /// `require_signature` middleware injected). `signature_header` and /// `signature_input` are the raw RFC 9421 header values, persisted /// for audit; they were already verified at handler entry. + /// + /// `first_ref_name` is the FIRST ref name in the live push's + /// `ref_updates`. It is persisted on every row of the same + /// `request_id` so the recovery drain can reproduce the live + /// path's "one push event per push, keyed on the first ref" + /// cardinality. The caller computes it from `ref_updates` once and + /// passes the same value for every row. #[allow(dead_code, clippy::too_many_arguments)] // wired by the handler refactor in the next slice pub async fn insert_pending_ref_transitions( &self, @@ -2617,6 +2663,7 @@ impl Db { signature_header: &str, signature_input: &str, content_digest: &str, + first_ref_name: &str, ) -> Result> { let now = Utc::now().to_rfc3339(); let mut out = Vec::with_capacity(ref_updates.len()); @@ -2632,8 +2679,9 @@ impl Db { sqlx::query( r#"INSERT INTO pending_ref_transitions (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, - signature_header, signature_input, content_digest, state, created_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)"#, + signature_header, signature_input, content_digest, state, created_at, + first_ref_name) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)"#, ) .bind(&id) .bind(request_id) @@ -2648,6 +2696,7 @@ impl Db { .bind(content_digest) .bind(pending_state::PREPARED) .bind(&now) + .bind(first_ref_name) .execute(&self.pool) .await?; out.push(PendingRefTransition { @@ -2666,6 +2715,7 @@ impl Db { created_at: now.clone(), applied_at: None, cancelled_at: None, + first_ref_name: first_ref_name.to_string(), }); } Ok(out) @@ -2727,7 +2777,7 @@ impl Db { let rows = sqlx::query( r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - applied_at, cancelled_at + applied_at, cancelled_at, first_ref_name FROM pending_ref_transitions WHERE state = $1 ORDER BY applied_at ASC NULLS LAST, id ASC @@ -2743,6 +2793,69 @@ impl Db { .collect()) } + /// Return every `prepared` row, oldest first. The startup + /// `reconcile_prepared_from_disk` step enumerates these, checks + /// each row's `new_sha` against the on-disk ref via + /// `git::store::list_refs`, and promotes the rows whose target + /// actually landed to `applied`. Rows that did NOT land (ref + /// rejected by receive_pack, or a `mark_applied` error stranded + /// the row in `prepared` with the ref still on the old SHA) stay + /// in `prepared`. + #[allow(dead_code)] // wired by the startup reconcile + pub async fn list_pending_ref_transitions_prepared( + &self, + limit: i64, + ) -> Result> { + let limit = limit.max(1); + let rows = sqlx::query( + r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, + signature_header, signature_input, content_digest, state, created_at, + applied_at, cancelled_at, first_ref_name + FROM pending_ref_transitions + WHERE state = $1 + ORDER BY created_at ASC, id ASC + LIMIT $2"#, + ) + .bind(pending_state::PREPARED) + .bind(limit) + .fetch_all(&self.pool) + .await?; + Ok(rows + .into_iter() + .map(row_to_pending_ref_transition) + .collect()) + } + + /// Flip a set of `prepared` rows to `applied`. Called by the startup + /// reconcile step after the on-disk SHA matches each row's + /// `new_sha`. The `state = 'prepared'` guard is the second + /// barrier against re-promoting a row that was cancelled by + /// another path while the reconcile was in flight; only rows that + /// were still `prepared` at the moment the UPDATE runs are + /// flipped. + #[allow(dead_code)] // wired by the startup reconcile + pub async fn mark_pending_ref_transitions_applied_for_rows( + &self, + ids: &[String], + ) -> Result { + if ids.is_empty() { + return Ok(0); + } + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1, applied_at = $2 + WHERE id = ANY($3) AND state = $4"#, + ) + .bind(pending_state::APPLIED) + .bind(&now) + .bind(ids) + .bind(pending_state::PREPARED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + /// Delete a row by id. Called by the recovery drain AFTER the push /// event, the cert, and the anchor job have all landed. A subsequent /// drain pass is a no-op for the same transition because the row is @@ -2783,8 +2896,8 @@ impl Db { r#"INSERT INTO pending_ref_transitions (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - applied_at, cancelled_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + applied_at, cancelled_at, first_ref_name) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16)"#, ) .bind(&row.id) .bind(&row.request_id) @@ -2801,6 +2914,7 @@ impl Db { .bind(&row.created_at) .bind(applied_at_opt) .bind(cancelled_at_opt) + .bind(&row.first_ref_name) .execute(&self.pool) .await?; Ok(()) @@ -4538,6 +4652,7 @@ fn row_to_pending_ref_transition(r: sqlx::postgres::PgRow) -> PendingRefTransiti created_at: r.get("created_at"), applied_at: r.get("applied_at"), cancelled_at: r.get("cancelled_at"), + first_ref_name: r.get("first_ref_name"), } } @@ -7307,6 +7422,114 @@ mod ref_certificate_tests { ); } + /// P1-B: the live handler routes cert issuance through + /// `cert::issue_ref_certificate` (the upsert, NOT + /// `insert_ref_certificate_idempotent`'s DO NOTHING). This test + /// exercises the full `cert::issue_ref_certificate` call path + /// end-to-end through the `AppState`, asserting that: + /// + /// - a re-push to the same `(repo_id, ref_name)` updates + /// `old_sha` / `new_sha` / `pusher_did` / `issued_at` / + /// `signature` to the new transition's values, + /// - the deterministic `cert_id` (derived from + /// `ref_cert_id_for(request_id, ref_name)`) is preserved + /// across the re-push, and + /// - exactly one cert row exists for the ref after the + /// re-push. + /// + /// This pins the live-handler contract that the previous + /// `issue_ref_certificate_idempotent` call violated. The DB-level + /// `insert_ref_certificate_upserts_on_repo_ref` test pins the + /// underlying upsert SQL; this test pins the live-handler wrapper. + #[sqlx::test] + async fn issue_ref_certificate_upserts_on_repo_ref_via_live_path(pool: PgPool) { + use crate::cert; + use crate::db::ref_cert_id_for; + + let state = crate::test_support::test_state(pool.clone()).await; + let repo_id = uuid::Uuid::new_v4().to_string(); + state + .db + .create_repo(&RepoRecord { + id: repo_id.clone(), + name: "cert-upsert-live".into(), + owner_did: "did:key:zOWNER".into(), + description: None, + is_public: true, + default_branch: "main".into(), + created_at: Utc::now(), + updated_at: Utc::now(), + disk_path: "/tmp/cert-upsert-live".into(), + forked_from: None, + machine_id: None, + }) + .await + .unwrap(); + + // First push: 0000 -> 1111, pusher A. + let c1 = cert::issue_ref_certificate( + &state, + &repo_id, + "refs/heads/main", + "0000", + "1111", + "did:key:zFirstPusher", + &ref_cert_id_for("req-A", "refs/heads/main"), + ) + .await + .unwrap(); + + // Sleep 1ms so the second push's `issued_at` is strictly + // greater than the first. `build_ref_certificate` stamps + // `issued_at = Utc::now()`, and the upsert's per-column + // guard `EXCLUDED.issued_at > ref_certificates.issued_at` + // only updates on strictly-newer timestamps. + tokio::time::sleep(std::time::Duration::from_millis(2)).await; + + // Second push: aaaa -> bbbb, pusher B, SAME deterministic + // cert id (same `request_id` and `ref_name`). + let c2 = cert::issue_ref_certificate( + &state, + &repo_id, + "refs/heads/main", + "aaaa", + "bbbb", + "did:key:zSecondPusher", + &ref_cert_id_for("req-A", "refs/heads/main"), + ) + .await + .unwrap(); + + // The deterministic id is preserved across the re-push. + assert_eq!(c1.id, c2.id, "cert id is preserved across re-push"); + assert_eq!( + c1.id, + ref_cert_id_for("req-A", "refs/heads/main"), + "cert id is the deterministic (request_id, ref_name) hash" + ); + + // The upsert updated every other field to the second push. + assert_eq!(c1.new_sha, "1111", "first push's new_sha"); + assert_eq!(c2.new_sha, "bbbb", "re-push updates new_sha"); + assert_eq!(c1.pusher_did, "did:key:zFirstPusher"); + assert_eq!(c2.pusher_did, "did:key:zSecondPusher"); + assert_ne!( + c1.issued_at, c2.issued_at, + "issued_at advances on a re-push" + ); + assert_ne!(c1.signature, c2.signature, "signature is re-signed"); + + // Exactly one row in the table for the ref. + let certs = state.db.list_ref_certificates(&repo_id, 10).await.unwrap(); + assert_eq!(certs.len(), 1, "exactly one cert row per ref"); + assert_eq!(certs[0].id, c1.id, "the original id survives"); + assert_eq!(certs[0].new_sha, "bbbb", "row reflects the latest push"); + assert_eq!( + certs[0].pusher_did, "did:key:zSecondPusher", + "row reflects the latest pusher" + ); + } + #[sqlx::test] async fn list_ref_certificates_clamps_negative_limit(pool: PgPool) { let db = db(pool).await; @@ -9210,6 +9433,7 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", + "refs/heads/main", ) .await .unwrap(); @@ -9254,6 +9478,7 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", + "refs/heads/main", ) .await .unwrap(); @@ -9294,6 +9519,7 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", + "refs/heads/main", ) .await .unwrap(); @@ -9330,6 +9556,7 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", + "refs/heads/main", ) .await .unwrap(); @@ -9376,6 +9603,11 @@ mod pending_ref_transition_tests { created_at: now.clone(), applied_at: Some(now.clone()), cancelled_at: None, + // Single-ref test, so the request's first ref name is the + // same as the ref name. The new multi-ref test sets this + // explicitly to the request's actual first ref across all + // rows of the same `request_id`. + first_ref_name: "refs/heads/main".to_string(), }; db.insert_pending_ref_transition_for_test(&row) .await @@ -9476,6 +9708,7 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", + "refs/heads/main", ) .await .unwrap(); diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index 46b3855af..ffccf120f 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -25,24 +25,297 @@ use crate::cert; use crate::db::PendingRefTransition; use crate::state::AppState; +use chrono::{DateTime, Utc}; +use std::collections::HashMap; -/// One drain pass. Returns the number of transitions re-derived. Called -/// from startup and from the failure-injection test. +/// Promote `prepared` rows whose `new_sha` matches the on-disk ref to +/// `applied`, so the recovery drain (which only reads `state = +/// 'applied'`) picks them up on the next pass. The reconcile runs at +/// startup, BEFORE the drain. /// -/// `limit` bounds the work per call. The startup caller passes a -/// generous cap (1000); tests pass a small one to assert behavior -/// without flooding the database. +/// This is the second half of the P1-A fix. The first half is the +/// live handler's `mark_pending_ref_transitions_applied` call, which +/// can fail or be interrupted AFTER `receive_pack` returned `Ok`. A +/// `prepared` row whose target ref actually landed on disk has no +/// recovery path without this step: the drain's WHERE clause does not +/// see it, and a startup that boots and serves traffic would silently +/// lose the push event, the cert, and the anchor handoff for that +/// ref. +/// +/// Strict SHA equality is the load-bearing correctness check. A row +/// whose `new_sha` does NOT match the on-disk ref stays `prepared`; +/// the invariant "a failed receive-pack is never promoted to +/// completed accounting" is preserved by the equality check, not by +/// state alone. A `cancelled` row is also never promoted (the +/// `list_pending_ref_transitions_prepared` SELECT gates on +/// `state = 'prepared'`, and the UPDATE re-checks the state). +/// +/// The SHA check alone is not sufficient. A `prepared` row could +/// have a `new_sha` that currently matches the on-disk ref for a +/// reason OTHER than its own transition (e.g. a later push +/// re-introduced the same SHA on the same ref). To prevent the +/// recovery drain from writing artifacts for a transition the node +/// cannot prove actually happened, the reconcile ALSO requires the +/// row's `created_at` to be within [`MAX_RECONCILE_AGE`] of the +/// current time. Rows older than the window are left `prepared` for +/// human-attended recovery. This is the second correctness barrier, +/// and the reason a `prepared` row that happens to match a current +/// on-disk SHA does not silently turn into completed accounting. +pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow::Result { + let rows = state + .db + .list_pending_ref_transitions_prepared(limit) + .await?; + if rows.is_empty() { + return Ok(0); + } + + // Group rows by repo so we call `list_refs` once per repo, not + // once per row. + let mut by_repo: HashMap> = HashMap::new(); + for row in &rows { + by_repo.entry(row.repo_id.clone()).or_default().push(row); + } + + let mut to_promote: Vec = Vec::new(); + for (repo_id, repo_rows) in by_repo { + let repo = match state.db.get_repo_by_id(&repo_id).await? { + Some(r) => r, + None => { + // The repo row is gone. The outbox rows must have + // been orphaned by a hard delete; the reviewer's + // invariant does not bind here, so we leave them + // `prepared` and let a later startup (or a + // human-attended recovery) resolve them. Log so the + // operator can act. + tracing::warn!( + repo_id = %repo_id, + row_count = repo_rows.len(), + "reconcile: repo row missing; leaving prepared rows untouched" + ); + continue; + } + }; + let disk_path = std::path::Path::new(&repo.disk_path); + let refs = match crate::git::store::list_refs(disk_path) { + Ok(v) => v, + Err(e) => { + // A `list_refs` failure is not fatal: skip this + // repo's group, leave the rows `prepared` for a + // later startup. + tracing::warn!( + err = %e, + repo_id = %repo_id, + row_count = repo_rows.len(), + "reconcile: list_refs failed; leaving prepared rows untouched" + ); + continue; + } + }; + let disk_refs: HashMap = refs.into_iter().collect(); + + for row in repo_rows { + let matches = disk_refs + .get(&row.ref_name) + .map(|sha| sha == &row.new_sha) + .unwrap_or(false); + if !matches { + let on_disk = disk_refs + .get(&row.ref_name) + .cloned() + .unwrap_or_else(|| "".to_string()); + tracing::debug!( + request_id = %row.request_id, + repo_id = %row.repo_id, + ref_name = %row.ref_name, + row_new_sha = %row.new_sha, + on_disk_sha = %on_disk, + "reconcile: row's new_sha does not match on-disk ref; staying prepared" + ); + continue; + } + // SHA matched. Before promoting, confirm the row is + // recent enough to be the transition that produced the + // current on-disk SHA. A stale `prepared` row whose + // `new_sha` happens to equal the current ref value for + // some OTHER reason (e.g. a later push re-introduced + // the same SHA) would otherwise be promoted and the + // recovery drain would write artifacts for a transition + // we cannot prove happened. The `MAX_RECONCILE_AGE` + // window bounds the blast radius; older rows require + // human-attended recovery. + let row_age = DateTime::parse_from_rfc3339(&row.created_at) + .ok() + .map(|t| Utc::now().signed_duration_since(t.with_timezone(&Utc))) + .unwrap_or_else(|| { + // Unparseable `created_at` is a corruption + // signal. Treat the row as unpromotable so a + // human can look at it. + tracing::warn!( + row_id = %row.id, + request_id = %row.request_id, + created_at = %row.created_at, + "reconcile: unparseable created_at; staying prepared (human-attended recovery)" + ); + MAX_RECONCILE_AGE + chrono::Duration::seconds(1) + }); + if row_age > MAX_RECONCILE_AGE { + tracing::warn!( + row_id = %row.id, + request_id = %row.request_id, + repo_id = %row.repo_id, + ref_name = %row.ref_name, + row_new_sha = %row.new_sha, + row_age_secs = row_age.num_seconds(), + max_reconcile_age_secs = MAX_RECONCILE_AGE.num_seconds(), + "reconcile: row is older than the recovery window; staying prepared (human-attended recovery required)" + ); + continue; + } + to_promote.push(row.id.clone()); + } + } + + let flipped = state + .db + .mark_pending_ref_transitions_applied_for_rows(&to_promote) + .await?; + if flipped > 0 { + tracing::info!( + flipped, + "reconciled prepared -> applied via on-disk ref match" + ); + } + Ok(flipped as usize) +} + +/// Per-pass drain budget. Each call to `drain_pending_ref_transitions` +/// processes at most this many rows. +pub const DRAIN_PER_PASS_LIMIT: i64 = 1000; + +/// Maximum age (relative to `Utc::now()`) at which a `prepared` row +/// is auto-promoted by [`reconcile_prepared_from_disk`]. Rows older +/// than this stay `prepared` and require human-attended recovery. +/// +/// The window bounds the blast radius of a stale-row promotion: the +/// only way the on-disk SHA matches a `prepared` row's `new_sha` for +/// an OLD row is if some OTHER push re-introduced the same SHA on +/// the same ref after the original transition failed. With a bounded +/// window, that mis-match only matters for `created_at` within the +/// window — recent enough that an operator can correlate the row +/// with the live handler's logs. Older rows are deliberately left +/// `prepared` so a human can audit them rather than have the node +/// silently write a push event / cert / anchor for a transition the +/// node has no way to prove actually happened. +pub const MAX_RECONCILE_AGE: chrono::Duration = chrono::Duration::seconds(24 * 60 * 60); + +/// Maximum number of passes the startup drain will run before logging +/// a residual-backlog warning. With `DRAIN_PER_PASS_LIMIT = 1000` and +/// `DRAIN_MAX_PASSES = 10`, the startup drain will process up to +/// 10,000 rows in one boot. Rows beyond that remain `applied` and +/// are picked up on the next startup. +pub const DRAIN_MAX_PASSES: usize = 10; + +/// One drain pass. Returns the number of transitions fully re-derived +/// (artifacts written AND row deleted). Production callers use +/// [`drain_pending_ref_transitions_all`] to drain an unbounded backlog +/// across multiple passes; tests can call this directly with a small +/// `limit` to assert behavior on a single batch. +/// +/// P2-A: a `derive_one` failure on one row is logged but does NOT +/// abort the rest of the batch — the failing row stays `applied` +/// for a later startup to retry. A `delete_pending_ref_transition` +/// failure on a row whose `derive_one` succeeded is also logged and +/// the row remains `applied`; the next drain re-derives (idempotent +/// inserts make this safe) and tries the delete again. pub async fn drain_pending_ref_transitions(state: AppState, limit: i64) -> anyhow::Result { + drain_pending_ref_transitions_with(state, limit, |s, r| async move { derive_one(&s, &r).await }) + .await +} + +/// Testable seam for the drain loop. Production code calls +/// [`drain_pending_ref_transitions`], which delegates here with the +/// real [`derive_one`]. Tests inject a closure that fails for one +/// row and succeeds for another to assert that the loop does not +/// abort on a single error. +pub async fn drain_pending_ref_transitions_with( + state: AppState, + limit: i64, + derive_fn: F, +) -> anyhow::Result +where + F: Fn(AppState, PendingRefTransition) -> Fut, + Fut: std::future::Future>, +{ let rows = state.db.list_pending_ref_transitions_applied(limit).await?; let mut count = 0; for row in rows { - derive_one(&state, &row).await?; - state.db.delete_pending_ref_transition(&row.id).await?; - count += 1; + match derive_fn(state.clone(), row.clone()).await { + Ok(()) => { + if let Err(e) = state.db.delete_pending_ref_transition(&row.id).await { + tracing::warn!( + err = %e, + row_id = %row.id, + "drain: row derivation succeeded but delete failed; row will be re-derived next startup (idempotent inserts make this safe)" + ); + continue; + } + count += 1; + } + Err(e) => { + tracing::error!( + err = %e, + row_id = %row.id, + request_id = %row.request_id, + "drain: derive_fn failed; row left in `applied` for next startup" + ); + } + } } Ok(count) } +/// Drain an unbounded `applied` backlog across multiple passes. The +/// drain stops as soon as a pass returns fewer rows than +/// `per_pass_limit` (i.e. the backlog is exhausted). If +/// `max_passes` passes still leave a full pass of work, a warning is +/// logged and the function returns the count processed so far; the +/// residual rows remain `applied` for the next startup. +/// +/// The startup caller in [`crate::main`] uses +/// `DRAIN_PER_PASS_LIMIT` and `DRAIN_MAX_PASSES` from this module so +/// the test that asserts "backlog > 1000 is fully processed" can +/// reference the same constants. +pub async fn drain_pending_ref_transitions_all( + state: AppState, + per_pass_limit: i64, + max_passes: usize, +) -> anyhow::Result { + let mut total = 0; + for _ in 0..max_passes { + let n = drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; + total += n; + // `per_pass_limit` is `i64`; `n` is `usize` (the drain + // returns a row count). Cast for the comparison. + if (n as i64) < per_pass_limit { + return Ok(total); + } + } + // One more pass to detect residual backlog. If this pass is also + // full, log a warning and return what we have; the next startup + // will continue the work. + let residual = drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; + if (residual as i64) >= per_pass_limit { + tracing::warn!( + total = total + residual, + max_passes, + per_pass_limit, + "drain backlog exceeds startup budget; residual rows will be picked up on next restart" + ); + } + Ok(total + residual) +} + /// Re-derive the push event, the per-ref certificate, and the anchor /// handoff for one `applied` row, using the persisted authentic pusher /// identity. This is what closes the reviewer's invariant: the @@ -55,8 +328,15 @@ pub async fn drain_pending_ref_transitions(state: AppState, limit: i64) -> anyho /// idempotent (see the module-level comment), so a second drain pass /// against the same row is a no-op. pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow::Result<()> { - // Push event: deterministic id, idempotent insert. - let push_id = crate::db::push_event_id_for(&row.request_id, &row.ref_name); + // Push event: deterministic id, idempotent insert. The id is keyed + // on the REQUEST's first ref name (persisted on every row of this + // `request_id`) so the recovery push event id matches the live + // path's id and a live push followed by a recovery pass collapses + // to a single `push_events` row via `ON CONFLICT (id) DO NOTHING`. + // Per-ref certs and anchor jobs stay keyed on `row.ref_name` and + // `(repo, ref, old, new)` respectively — those are correctly + // transition-shaped. + let push_id = crate::db::push_event_id_for(&row.request_id, &row.first_ref_name); state .db .record_push_with_id(&push_id, &row.pusher_did, &row.repo_id, &row.new_sha, 0) @@ -156,6 +436,11 @@ mod drain_tests { created_at: now.clone(), applied_at: Some(now), cancelled_at: None, + // The existing tests are single-ref pushes, so the first + // ref name is the same as the ref name. The new multi-ref + // test sets this explicitly to the request's actual first + // ref name across all rows of the same `request_id`. + first_ref_name: ref_name.to_string(), } } @@ -345,4 +630,761 @@ mod drain_tests { .unwrap(); assert_eq!(anchor_count, 0); } + + // ----- P1-A reconcile tests ----- + // + // These tests cover the startup-time `reconcile_prepared_from_disk` + // step: a `prepared` row whose target ref actually landed on disk + // is promoted to `applied`; a row whose target did NOT land (or + // whose ref is missing) stays `prepared`; a `cancelled` row is + // never promoted; and a second call is a no-op. + // + // The on-disk state is a real bare git repo (so `list_refs` can + // read it) seeded with a synthetic commit via the plumbing + // commands `mktree` (empty tree) + `commit-tree` (root commit) + + // `update-ref` (point a ref at the commit). + + /// Build a real commit on a bare git repo's `ref_name`. Returns + /// the new commit SHA. Used by the reconcile tests to seed a + /// known SHA on disk so `list_refs` can read it back. + fn seed_ref_on_bare(bare_path: &std::path::Path, ref_name: &str) -> String { + use std::process::Command; + // Empty tree. + let tree = String::from_utf8( + Command::new("git") + .args(["mktree"]) + .current_dir(bare_path) + .stdin(std::process::Stdio::null()) + .output() + .expect("git mktree") + .stdout, + ) + .expect("mktree stdout utf8") + .trim() + .to_string(); + // Root commit on the empty tree. The env vars override any + // missing global config in CI. + let commit = String::from_utf8( + Command::new("git") + .args(["commit-tree", &tree, "-m", "test root"]) + .env("GIT_AUTHOR_NAME", "test") + .env("GIT_AUTHOR_EMAIL", "test@example.com") + .env("GIT_COMMITTER_NAME", "test") + .env("GIT_COMMITTER_EMAIL", "test@example.com") + .current_dir(bare_path) + .stdin(std::process::Stdio::null()) + .output() + .expect("git commit-tree") + .stdout, + ) + .expect("commit-tree stdout utf8") + .trim() + .to_string(); + // Point the ref at the commit. `update-ref` writes into the + // bare repo's refs/ tree. + Command::new("git") + .args(["update-ref", ref_name, &commit]) + .current_dir(bare_path) + .stdin(std::process::Stdio::null()) + .output() + .expect("git update-ref"); + commit + } + + /// Seed a `RepoRecord` row pointing at `disk_path` and return + /// the repo id. Mirrors what `repos::create_repo` does in + /// production, but without the rest of the create-repo + /// bookkeeping the test does not exercise. + async fn seed_repo_row(state: &crate::state::AppState, disk_path: &str) -> String { + use crate::db::RepoRecord; + use chrono::Utc; + let repo_id = uuid::Uuid::new_v4().to_string(); + state + .db + .create_repo(&RepoRecord { + id: repo_id.clone(), + name: "reconcile-test".into(), + owner_did: "did:key:z6owner".into(), + description: None, + is_public: true, + default_branch: "main".into(), + created_at: Utc::now(), + updated_at: Utc::now(), + disk_path: disk_path.to_string(), + forked_from: None, + machine_id: None, + }) + .await + .expect("create_repo"); + repo_id + } + + #[sqlx::test] + async fn reconcile_promotes_prepared_row_when_on_disk_sha_matches(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // Create a bare repo on disk with refs/heads/main = X. + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + // Persist a `prepared` row whose `new_sha` matches the on-disk + // SHA. This simulates the crash window the reviewer flagged: + // receive_pack returned Ok and the ref landed, but the + // handler never reached `mark_pending_ref_transitions_applied`. + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + // The drain must NOT see the row before reconcile (it only + // reads `applied`). + let pre_drain = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert!(pre_drain.is_empty(), "drain cannot see a prepared row"); + + // Reconcile: the row's new_sha matches the on-disk ref, so it + // should be promoted. + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 1, "exactly one row promoted to applied"); + + // The row is now in `applied` and the drain can see it. + let after_drain = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert_eq!(after_drain.len(), 1, "row is now visible to the drain"); + assert_eq!(after_drain[0].id, row.id, "the same row is promoted"); + assert_eq!(after_drain[0].state, pending_state::APPLIED); + assert!( + after_drain[0].applied_at.is_some(), + "applied_at is set on promotion" + ); + + // The prepared list is now empty. + let still_prepared = state + .db + .list_pending_ref_transitions_prepared(100) + .await + .unwrap(); + assert!( + still_prepared.is_empty(), + "no prepared rows remain after a successful reconcile" + ); + } + + #[sqlx::test] + async fn reconcile_leaves_prepared_row_when_on_disk_sha_differs(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // On-disk SHA is `aaaa...` (the live push landed at this SHA). + // The row's `new_sha` is `bbbb...` — the SHA the row CLAIMS + // the push went to, but the actual on-disk state disagrees. + // This models a row stranded by a `mark_applied` failure on a + // push whose target was rolled back, or any case where the + // recorded `new_sha` does not match reality. + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + assert_ne!(on_disk_sha, "b".repeat(40), "test sanity: SHAs differ"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row( + &repo_id, + "refs/heads/main", + &"0".repeat(40), + &"b".repeat(40), + ); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + // Reconcile: the SHA does not match, so NOTHING is promoted. + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 0, "no row promoted when SHAs do not match"); + + // The row is still `prepared` and the drain cannot see it. + let after_drain = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert!( + after_drain.is_empty(), + "drain must not see a row whose on-disk SHA does not match" + ); + let still_prepared = state + .db + .list_pending_ref_transitions_prepared(100) + .await + .unwrap(); + assert_eq!(still_prepared.len(), 1, "row stays prepared"); + assert_eq!(still_prepared[0].id, row.id); + assert!( + still_prepared[0].applied_at.is_none(), + "applied_at is NOT set on a non-promotion" + ); + } + + #[sqlx::test] + async fn reconcile_leaves_cancelled_row_untouched(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // On-disk ref matches the row's `new_sha` — but the row is + // `cancelled`, so the reconcile must NEVER promote it. The + // reviewer's invariant: a failed receive-pack is never + // promoted to completed accounting. + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::CANCELLED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 0, "cancelled rows are never promoted"); + + // The row is still cancelled and the drain still cannot see it. + let after_drain = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert!(after_drain.is_empty()); + let still_prepared = state + .db + .list_pending_ref_transitions_prepared(100) + .await + .unwrap(); + assert!( + still_prepared.is_empty(), + "cancelled rows do not show up in the prepared list either" + ); + } + + #[sqlx::test] + async fn reconcile_is_idempotent(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + // First call: 1 row promoted. + let n1 = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n1, 1); + // Second call: 0 rows — the row is no longer `prepared`, so + // the list query returns empty. + let n2 = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n2, 0, "a second reconcile is a no-op"); + + // Final state: applied, with applied_at set. + let applied = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert_eq!(applied.len(), 1); + assert_eq!(applied[0].state, pending_state::APPLIED); + assert!(applied[0].applied_at.is_some()); + } + + /// A `prepared` row whose `new_sha` happens to match the current + /// on-disk ref value for a reason OTHER than its own transition + /// (e.g. a later push re-introduced the same SHA on the same + /// ref) must NOT be promoted just because the SHAs match. The + /// `MAX_RECONCILE_AGE` window is the second correctness barrier: + /// rows older than the window stay `prepared` for + /// human-attended recovery. + /// + /// This test seeds a `prepared` row whose `new_sha` DOES match + /// the on-disk ref, but whose `created_at` is 25 hours in the + /// past (one hour past `MAX_RECONCILE_AGE = 24h`). The reconcile + /// must NOT promote it. + #[sqlx::test] + async fn reconcile_does_not_promote_stale_prepared_row(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // On-disk ref with a known SHA. + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + // Build a `prepared` row whose SHA matches the on-disk ref, + // but whose `created_at` is older than `MAX_RECONCILE_AGE`. + // This models a row that was stranded by an ancient + // `mark_applied` failure and then re-introduced the same + // SHA via a later push. + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + // 25 hours ago — outside the 24h window. + row.created_at = (chrono::Utc::now() - chrono::Duration::hours(25)).to_rfc3339(); + // `make_row` derives `id` from the deterministic hash using + // the `created_at` it generated at construction time. Now + // that we've overwritten `created_at`, the row's `id` no + // longer matches what `insert_pending_ref_transitions` + // would have produced in production, but the test only + // checks the reconcile's behavior, not the id's contents, + // so the stale id is harmless here. + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + // The SHA matches, but the row is older than the window: + // reconcile must NOT promote it. + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!( + n, 0, + "stale row stays prepared (SHA matched but age exceeded the recovery window)" + ); + + // The row is still `prepared` and the drain cannot see it. + let after_drain = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert!( + after_drain.is_empty(), + "drain must not see a row outside the recovery window" + ); + let still_prepared = state + .db + .list_pending_ref_transitions_prepared(100) + .await + .unwrap(); + assert_eq!(still_prepared.len(), 1, "row stays prepared"); + assert_eq!(still_prepared[0].id, row.id); + assert!( + still_prepared[0].applied_at.is_none(), + "applied_at is NOT set on a stale row" + ); + } + + /// A `prepared` row that is fresh (within `MAX_RECONCILE_AGE`) + /// and SHA-matches the on-disk ref MUST still be promoted. This + /// is the existing happy-path contract; the test pins it so a + /// future change to the age check does not silently break the + /// recovery path for legitimate stranded rows. + #[sqlx::test] + async fn reconcile_promotes_fresh_prepared_row_with_matching_sha(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + // `make_row` defaults `created_at` to `Utc::now()`, which is + // well within `MAX_RECONCILE_AGE`. The SHA matches. This + // row should be promoted. + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 1, "fresh row with matching SHA is promoted"); + } + + // ----- P2-A drain resilience tests ----- + // + // These tests cover the "drain must not abort on first failure" + // and "drain must not cap at 1000 rows per startup" findings. + // Backlog processing uses the production `drain_pending_ref_transitions_all` + // (the `DRAIN_PER_PASS_LIMIT` / `DRAIN_MAX_PASSES` constants from + // this module) so the test exercises the same wrapper the + // startup calls. Failure isolation uses + // `drain_pending_ref_transitions_with` to inject a closure that + // errors for one row and succeeds for the next. + + #[sqlx::test] + async fn drain_processes_backlog_larger_than_one_pass(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // Seed 1500 distinct `applied` rows. Each row needs a unique + // `request_id` so the deterministic `pending_ref_transition` + // PKs (which hash `request_id`) don't collide, and the + // push-event / cert / anchor PKs (which also hash + // `request_id`) don't collide either. + const N: usize = 1500; + for i in 0..N { + let mut row = make_row( + "repo-backlog", + "refs/heads/main", + &"0".repeat(40), + &format!("{:040x}", i as u64), + ); + // Override `request_id` to a unique value per row. The + // `id` is derived from this in `make_row`, so the unique + // `request_id` also gives a unique row PK. + row.request_id = format!("req-{i}"); + row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row.request_id, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + } + + // Drain with the production limits. Two passes of 1000 each + // cover all 1500 rows; the third pass would be empty and + // exits the loop early on the `n < per_pass_limit` check. + let total = drain_pending_ref_transitions_all( + state.clone(), + DRAIN_PER_PASS_LIMIT, + DRAIN_MAX_PASSES, + ) + .await + .unwrap(); + assert_eq!(total, N, "drain processed the full backlog"); + + // No `applied` rows remain. + let after = state + .db + .list_pending_ref_transitions_applied(10_000) + .await + .unwrap(); + assert!( + after.is_empty(), + "every applied row was processed and deleted" + ); + } + + #[sqlx::test] + async fn drain_continues_past_a_failing_row(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // Seed two `applied` rows, A first so the `ORDER BY applied_at + // ASC NULLS LAST, id ASC` query hits A before B. They have + // distinct `request_id`s so the deterministic PKs don't + // collide. + let mut row_a = make_row( + "repo-fail-then-pass", + "refs/heads/main", + &"0".repeat(40), + &"a".repeat(40), + ); + row_a.request_id = "req-A".to_string(); + row_a.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row_a.request_id, + &row_a.repo_id, + &row_a.ref_name, + &row_a.old_sha, + &row_a.new_sha, + ]); + let mut row_b = make_row( + "repo-fail-then-pass", + "refs/heads/main", + &"0".repeat(40), + &"b".repeat(40), + ); + row_b.request_id = "req-B".to_string(); + row_b.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row_b.request_id, + &row_b.repo_id, + &row_b.ref_name, + &row_b.old_sha, + &row_b.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&row_a) + .await + .unwrap(); + state + .db + .insert_pending_ref_transition_for_test(&row_b) + .await + .unwrap(); + + // Inject a closure that fails for row A and delegates to + // `derive_one` for everything else. Row B is processed + // normally; row A's failure is logged and the row stays + // `applied` for a future retry. The `Fn` bound on the seam + // forbids moving the id into the closure, so the closure + // clones the id from the outer `row_a_id` local on every + // iteration. + let row_a_id = row_a.id.clone(); + let n = drain_pending_ref_transitions_with(state.clone(), 100, |s, r| { + let target = row_a_id.clone(); + async move { + if r.id == target { + Err(anyhow::anyhow!("injected derive failure")) + } else { + derive_one(&s, &r).await + } + } + }) + .await + .unwrap(); + assert_eq!(n, 1, "only row B is fully processed and deleted"); + + // Row A is still in `applied` (NOT deleted, NOT re-derivable + // yet by a future pass that just calls `derive_one` — the + // inserted artifacts were never created). + let after = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert_eq!(after.len(), 1, "row A is still in `applied`"); + assert_eq!(after[0].id, row_a_id); + + // Row A's artifacts were not created (the closure errored + // before any insert ran). + let a_push = state + .db + .count_push_events(&row_a.repo_id, &row_a.new_sha, &row_a.pusher_did) + .await + .unwrap(); + assert_eq!(a_push, 0, "row A's push event was not created"); + let a_anchors = state + .db + .count_anchor_jobs( + &row_a.repo_id, + &row_a.ref_name, + &row_a.old_sha, + &row_a.new_sha, + ) + .await + .unwrap(); + assert_eq!(a_anchors, 0, "row A's anchor job was not created"); + let a_certs = state + .db + .list_ref_certificates(&row_a.repo_id, 10) + .await + .unwrap(); + // The cert table is per-(repo, ref) with one row. Row B's + // drain wrote a cert for this `(repo, ref)`. We need to + // check row A's specific cert by its deterministic id — + // the row A's `derive_one` never ran, so the row A cert id + // must not exist in the table. + let a_cert_id = crate::db::ref_cert_id_for(&row_a.request_id, &row_a.ref_name); + let a_cert = state.db.get_ref_certificate(&a_cert_id).await.unwrap(); + assert!( + a_cert.is_none(), + "row A's specific cert was not created (got {:?})", + a_cert.map(|c| c.id) + ); + // The single cert for this `(repo, ref)` is row B's. + assert_eq!(a_certs.len(), 1, "row B's cert exists in the table"); + let b_cert_id = crate::db::ref_cert_id_for(&row_b.request_id, &row_b.ref_name); + assert_eq!(a_certs[0].id, b_cert_id, "the only cert is row B's"); + + // Row B's other artifacts WERE created (the closure called + // the real `derive_one`, which writes the push event and + // anchor job for row B). + let b_push = state + .db + .count_push_events(&row_b.repo_id, &row_b.new_sha, &row_b.pusher_did) + .await + .unwrap(); + assert_eq!(b_push, 1, "row B's push event was created"); + let b_anchors = state + .db + .count_anchor_jobs( + &row_b.repo_id, + &row_b.ref_name, + &row_b.old_sha, + &row_b.new_sha, + ) + .await + .unwrap(); + assert_eq!(b_anchors, 1, "row B's anchor job was created"); + } + + // ----- P2-B multi-ref push event cardinality test ----- + // + // The live handler and the recovery drain must produce the same + // push event id for a multi-ref push. The id is keyed on + // `(request_id, first_ref_name)` where `first_ref_name` is the + // first ref in the live push. Every row of the same `request_id` + // carries the same `first_ref_name`, so the recovery drain + // produces N identical push event ids for an N-ref push, and + // `ON CONFLICT (id) DO NOTHING` collapses them to a single row. + // + // Certs stay per-ref (one per `(repo, ref)` transition); anchor + // jobs stay per-transition (one per `(repo, ref, old, new)` tuple). + // The push event is the only artifact that is request-scoped. + + #[sqlx::test] + async fn multi_ref_push_produces_exactly_one_event_across_live_and_recovery( + pool: sqlx::PgPool, + ) { + let state = crate::test_support::test_state(pool).await; + + // Three rows for the SAME `request_id`, distinct `ref_name`s. + // All three share `first_ref_name = "refs/heads/main"` (the + // first ref in the simulated push). The `new_sha` is the + // same across all three because this models a push that + // advanced a tip commit onto three refs at once (a common + // case for `git push --all` or for a single-commit push to + // multiple branches). + let shared_new_sha = "c".repeat(40); + let ref_names = [ + "refs/heads/main", + "refs/heads/feature-a", + "refs/heads/feature-b", + ]; + for (i, ref_name) in ref_names.iter().enumerate() { + let mut row = make_row("repo-multi", ref_name, &"0".repeat(40), &shared_new_sha); + row.request_id = "req-multi".to_string(); + row.first_ref_name = "refs/heads/main".to_string(); + row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row.request_id, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + ]); + // Vary `old_sha` per row so the anchor job PKs (which + // hash `old_sha`) don't collide. + row.old_sha = format!("{:040x}", (i + 1) as u64); + row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row.request_id, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + } + + // Drain all three rows. + let n = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 3, "all three rows re-derived"); + + // Exactly one push event row, keyed on the deterministic + // (request_id, first_ref_name) id. The three rows collapsed + // to a single event via `ON CONFLICT (id) DO NOTHING` in + // `record_push_with_id`. + let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); + assert_eq!( + push_count, 1, + "exactly one push event for a multi-ref push (trust-score predicate)" + ); + let events = state + .db + .count_push_events("repo-multi", &shared_new_sha, "did:key:z6pusher") + .await + .unwrap(); + assert_eq!(events, 1, "exactly one push_events row"); + + // The deterministic id is the one the live path would have + // written. + let expected_id = crate::db::push_event_id_for("req-multi", "refs/heads/main"); + // We don't have a direct "select by id" for push_events; the + // count of 1 already proves the cardinality. Assert the id + // is stable for completeness. + assert_eq!( + expected_id, + crate::db::push_event_id_for("req-multi", "refs/heads/main"), + "push_event_id_for is deterministic" + ); + + // Certs: one per ref (the cert contract is per-ref, NOT + // collapsed by first_ref_name). Three rows → three certs. + let certs = state + .db + .list_ref_certificates("repo-multi", 10) + .await + .unwrap(); + assert_eq!( + certs.len(), + 3, + "one cert per ref transition (not collapsed by first_ref_name)" + ); + + // Anchor jobs: one per `(repo, ref, old, new)` transition. + // Three rows → three anchor jobs. + for (i, ref_name) in ref_names.iter().enumerate() { + let n = state + .db + .count_anchor_jobs( + "repo-multi", + ref_name, + &format!("{:040x}", (i + 1) as u64), + &shared_new_sha, + ) + .await + .unwrap(); + assert_eq!(n, 1, "one anchor job per transition"); + } + } } diff --git a/crates/gitlawb-node/src/main.rs b/crates/gitlawb-node/src/main.rs index d4ae89b41..827dc1ad7 100644 --- a/crates/gitlawb-node/src/main.rs +++ b/crates/gitlawb-node/src/main.rs @@ -683,7 +683,34 @@ async fn main() -> Result<()> { // pushes, so a recovery re-derivation does not race a fresh push. // Non-fatal: a transient drain failure is logged and the rows // remain `applied` for the next startup to pick up. - match durable_outbox::drain_pending_ref_transitions(state.clone(), 1000).await { + // + // P1-A: the reconcile step runs FIRST and promotes any `prepared` + // row whose target SHA actually landed on disk. This is the path + // that recovers a ref when the post-receive + // `mark_pending_ref_transitions_applied` call errored or was + // interrupted after `receive_pack` returned Ok. Without this + // step, the drain (gated on `state = 'applied'`) would never see + // those rows. + match durable_outbox::reconcile_prepared_from_disk( + state.clone(), + durable_outbox::DRAIN_PER_PASS_LIMIT, + ) + .await + { + Ok(0) => {} + Ok(n) => info!(n, "reconciled prepared -> applied via on-disk ref match"), + Err(e) => warn!( + err = %e, + "pending ref transition reconcile failed at startup (non-fatal; will retry on next start)" + ), + } + match durable_outbox::drain_pending_ref_transitions_all( + state.clone(), + durable_outbox::DRAIN_PER_PASS_LIMIT, + durable_outbox::DRAIN_MAX_PASSES, + ) + .await + { Ok(0) => {} Ok(n) => info!(n, "drained pending ref transitions from prior run"), Err(e) => warn!( From 2638063160b5d77aef28c32ee3ab1ab5dc7d1e8e Mon Sep 17 00:00:00 2001 From: Gravirei Date: Sun, 30 Aug 2026 14:07:21 +0600 Subject: [PATCH 04/22] fix(node): address four reviewer findings on #26 split 1/4 (round 2) --- crates/gitlawb-node/src/cert.rs | 22 +- crates/gitlawb-node/src/db/mod.rs | 135 ++++++- crates/gitlawb-node/src/durable_outbox.rs | 433 +++++++++++++++++++--- 3 files changed, 536 insertions(+), 54 deletions(-) diff --git a/crates/gitlawb-node/src/cert.rs b/crates/gitlawb-node/src/cert.rs index a68ae12e7..5324d618b 100644 --- a/crates/gitlawb-node/src/cert.rs +++ b/crates/gitlawb-node/src/cert.rs @@ -27,17 +27,19 @@ use crate::state::AppState; /// /// #26 Split PR 1 P1-B: the live handler routes through this /// function (the upsert), NOT through -/// [`issue_ref_certificate_idempotent`] (DO NOTHING). The -/// idempotent variant is reserved for the recovery drain. Both -/// paths use the same deterministic `cert_id` so a re-pass is +/// [`issue_ref_certificate_idempotent`] (DO NOTHING). After the +/// reviewer-1 round-2 fix, the recovery drain also routes through +/// this function (P1: refresh a stale cert), so both paths use the +/// same deterministic `cert_id` and the same upsert. A re-pass is /// always safe: /// /// - Live handler → live upsert: re-push updates the row, preserves /// the original `id`. The contract pinned by /// `insert_ref_certificate_upserts_on_repo_ref` is restored. /// - Live handler → recovery: live's `ON CONFLICT (id) DO UPDATE` -/// preserves the original `id`; the recovery's -/// `ON CONFLICT (repo_id, ref_name) DO NOTHING` is a no-op. +/// preserves the original `id`; the recovery's same upsert is +/// a no-op for an equal-`issued_at` re-run and a refresh for a +/// strictly-newer one. /// - Recovery → live handler: the recovery wrote a row with the /// deterministic `id`; the live upsert (which preserves `id` and /// only updates other fields when `issued_at` is strictly newer) @@ -65,7 +67,7 @@ pub async fn issue_ref_certificate( state.db.insert_ref_certificate(&cert).await } -/// #26 Split PR 1 — idempotent variant used by the recovery drain. +/// #26 Split PR 1 — idempotent variant. /// /// `cert_id` is the deterministic id derived from /// `(request_id, ref_name)` so a recovery re-pass against the same @@ -74,7 +76,13 @@ pub async fn issue_ref_certificate( /// `insert_ref_certificate_idempotent` helper), so the function /// returns `None` if a live-path cert already exists for the /// `(repo_id, ref_name)` pair, and `Some(cert)` if it wrote a new -/// one. Either way, exactly one cert row exists for the transition. +/// one. +/// +/// Retained for any future caller that wants DO-NOTHING semantics +/// (e.g. an explicit "never overwrite" handler); the live and +/// recovery paths both use [`issue_ref_certificate`] (the upsert) +/// after the P1 fix in #26 Split 1 round 2. +#[allow(dead_code)] pub async fn issue_ref_certificate_idempotent( state: &AppState, repo_id: &str, diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index ec80227cd..bf7545966 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -2666,6 +2666,18 @@ impl Db { first_ref_name: &str, ) -> Result> { let now = Utc::now().to_rfc3339(); + // P2 (reviewer-2 round 2): wrap the multi-row insert in a + // transaction. A mid-loop failure used to return the error + // and leave the rows already inserted as `prepared`, which + // the receive-pack handler then refused to call. The + // stranded `prepared` rows were eventually reaped by the + // startup reconcile, but the partial-success state was + // observable in the DB and could mask a partial push + // intent. The transaction rolls the prior inserts back + // when any single row fails, so the caller either sees a + // complete `prepared` set for the request or sees none of + // them and the handler can safely return 503. + let mut tx = self.pool.begin().await?; let mut out = Vec::with_capacity(ref_updates.len()); for update in ref_updates { let id = deterministic_id(&[ @@ -2697,7 +2709,7 @@ impl Db { .bind(pending_state::PREPARED) .bind(&now) .bind(first_ref_name) - .execute(&self.pool) + .execute(&mut *tx) .await?; out.push(PendingRefTransition { id, @@ -2718,6 +2730,7 @@ impl Db { first_ref_name: first_ref_name.to_string(), }); } + tx.commit().await?; Ok(out) } @@ -9384,8 +9397,8 @@ mod pending_ref_transition_tests { //! line under test turns the named assertion red. use super::{ - anchor_job_id_for, pending_state, push_event_id_for, ref_cert_id_for, AnchorJob, Db, - PendingRefTransition, + anchor_job_id_for, deterministic_id, pending_state, push_event_id_for, ref_cert_id_for, + AnchorJob, Db, PendingRefTransition, RepoRecord, }; use crate::api::repos::RefUpdate; use chrono::Utc; @@ -9726,6 +9739,122 @@ mod pending_ref_transition_tests { ); } + /// P2 (reviewer-2 round 2): the multi-row `insert_pending_ref_transitions` + /// must be atomic. A mid-loop failure (here simulated by pre-seeding a + /// row whose PK collides with the second ref's deterministic id) must + /// roll the first row back; otherwise the handler can return 503 after + /// some `prepared` rows are already on disk, leaving the request in + /// an inconsistent state for the startup reconcile to clean up. + #[sqlx::test] + async fn insert_pending_ref_transitions_rolls_back_on_mid_loop_failure(pool: sqlx::PgPool) { + let db = db(pool).await; + // Seed a repo so the FK (if any) is satisfied. + db.create_repo(&RepoRecord { + id: "repo-atomic".to_string(), + name: "atomic".to_string(), + owner_did: "did:key:zAtomic".to_string(), + description: None, + is_public: true, + default_branch: "main".to_string(), + created_at: Utc::now(), + updated_at: Utc::now(), + disk_path: "/tmp/atomic".to_string(), + forked_from: None, + machine_id: None, + }) + .await + .unwrap(); + + // Pre-seed a row that collides with the SECOND ref update's + // deterministic id, so the loop's second INSERT fails on PK. + let second_ref = "refs/heads/feature-a"; + let second_old = "2".repeat(40); + let second_new = "3".repeat(40); + let collision_id = deterministic_id(&[ + "pending_ref_transition", + "req-atomic", + "repo-atomic", + second_ref, + &second_old, + &second_new, + ]); + // Direct insert bypassing the helper to land a `prepared` row + // with the colliding id. + sqlx::query( + r#"INSERT INTO pending_ref_transitions + (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, + signature_header, signature_input, content_digest, state, created_at, + first_ref_name) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)"#, + ) + .bind(&collision_id) + .bind("req-pre-seed") + .bind("repo-atomic") + .bind(second_ref) + .bind(&second_old) + .bind(&second_new) + .bind("did:key:zPre") + .bind("did:key:zNode") + .bind("sig-pre") + .bind("sig-input-pre") + .bind("digest-pre") + .bind(pending_state::PREPARED) + .bind(Utc::now().to_rfc3339()) + .bind("refs/heads/main") + .execute(&db.pool) + .await + .unwrap(); + + // Now call the production helper. The first ref (main) inserts + // fine; the second ref collides and the loop returns Err. + let res = db + .insert_pending_ref_transitions( + "req-atomic", + "repo-atomic", + "did:key:zNode", + "did:key:zPusher", + &[ + ref_update("refs/heads/main", &"1".repeat(40), &"2".repeat(40)), + ref_update(second_ref, &second_old, &second_new), + ], + "sig", + "sig-input", + "digest", + "refs/heads/main", + ) + .await; + assert!( + res.is_err(), + "the colliding insert must return Err (pre-condition for the rollback check)" + ); + + // The atomicity half: NO `req-atomic` row may exist. Without + // the transaction the first row would have been persisted + // before the second failed, and the startup reconcile would + // later see a stranded `prepared` row pointing at a push + // that never ran. + let stranded = sqlx::query_scalar::<_, i64>( + "SELECT COUNT(*) FROM pending_ref_transitions WHERE request_id = $1", + ) + .bind("req-atomic") + .fetch_one(&db.pool) + .await + .unwrap(); + assert_eq!( + stranded, 0, + "the transaction must roll back the first row when the second fails" + ); + // The pre-seeded row is unaffected. + let pres = sqlx::query_scalar::<_, i64>( + "SELECT COUNT(*) FROM pending_ref_transitions WHERE id = $1", + ) + .bind(&collision_id) + .fetch_one(&db.pool) + .await + .unwrap(); + assert_eq!(pres, 1, "the pre-seeded row is untouched"); + } + /// The `deterministic_id` helper uses an ASCII Unit Separator between /// fields so that two distinct tuples can never collide by accidental /// prefix overlap. `(a, bc)` and `(ab, c)` would otherwise hash the diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index ffccf120f..ca5dd7c03 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -211,9 +211,14 @@ pub const MAX_RECONCILE_AGE: chrono::Duration = chrono::Duration::seconds(24 * 6 /// Maximum number of passes the startup drain will run before logging /// a residual-backlog warning. With `DRAIN_PER_PASS_LIMIT = 1000` and -/// `DRAIN_MAX_PASSES = 10`, the startup drain will process up to -/// 10,000 rows in one boot. Rows beyond that remain `applied` and -/// are picked up on the next startup. +/// `DRAIN_MAX_PASSES = 10`, the startup drain runs `max_passes` regular +/// passes (10 × 1000 = 10,000 rows) plus ONE residual pass that +/// detects overrun and surfaces the residual-backlog warning at +/// `drain_pending_ref_transitions_all`'s tail. Total rows per boot +/// before the warning fires: 11,000. Rows beyond that remain +/// `applied` and are picked up on the next startup. P2-doc +/// (reviewer-2 round 2): the previous comment said "up to 10,000" but +/// the residual pass is the +1. pub const DRAIN_MAX_PASSES: usize = 10; /// One drain pass. Returns the number of transitions fully re-derived @@ -228,7 +233,18 @@ pub const DRAIN_MAX_PASSES: usize = 10; /// failure on a row whose `derive_one` succeeded is also logged and /// the row remains `applied`; the next drain re-derives (idempotent /// inserts make this safe) and tries the delete again. -pub async fn drain_pending_ref_transitions(state: AppState, limit: i64) -> anyhow::Result { +/// +/// P2-D (reviewer-2 round 2): the function returns +/// `(processed, examined)` rather than just `processed`. The caller +/// keys the drain's `n < per_pass_limit` exit condition on +/// `examined` so a pass where every row fails (or every +/// `derive_one` succeeds but every delete fails) still tells the +/// outer loop "more rows remain" — the previous `processed` count +/// was 0 and the loop exited on the first fully-failing pass. +pub async fn drain_pending_ref_transitions( + state: AppState, + limit: i64, +) -> anyhow::Result<(usize, usize)> { drain_pending_ref_transitions_with(state, limit, |s, r| async move { derive_one(&s, &r).await }) .await } @@ -242,13 +258,14 @@ pub async fn drain_pending_ref_transitions_with( state: AppState, limit: i64, derive_fn: F, -) -> anyhow::Result +) -> anyhow::Result<(usize, usize)> where F: Fn(AppState, PendingRefTransition) -> Fut, Fut: std::future::Future>, { let rows = state.db.list_pending_ref_transitions_applied(limit).await?; - let mut count = 0; + let mut processed = 0; + let examined = rows.len(); for row in rows { match derive_fn(state.clone(), row.clone()).await { Ok(()) => { @@ -260,7 +277,7 @@ where ); continue; } - count += 1; + processed += 1; } Err(e) => { tracing::error!( @@ -272,7 +289,7 @@ where } } } - Ok(count) + Ok((processed, examined)) } /// Drain an unbounded `applied` backlog across multiple passes. The @@ -293,27 +310,32 @@ pub async fn drain_pending_ref_transitions_all( ) -> anyhow::Result { let mut total = 0; for _ in 0..max_passes { - let n = drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; - total += n; - // `per_pass_limit` is `i64`; `n` is `usize` (the drain - // returns a row count). Cast for the comparison. - if (n as i64) < per_pass_limit { + let (processed, examined) = + drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; + total += processed; + // Key the exit on `examined`, not `processed`. A fully-failing + // batch returns processed=0 but examined=per_pass_limit; the + // loop must keep draining the backlog, not stop on the first + // 0-successes pass (P2-D, reviewer-2 round 2). + if (examined as i64) < per_pass_limit { return Ok(total); } } // One more pass to detect residual backlog. If this pass is also // full, log a warning and return what we have; the next startup // will continue the work. - let residual = drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; - if (residual as i64) >= per_pass_limit { + let (residual_processed, residual_examined) = + drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; + total += residual_processed; + if (residual_examined as i64) >= per_pass_limit { tracing::warn!( - total = total + residual, + total, max_passes, per_pass_limit, "drain backlog exceeds startup budget; residual rows will be picked up on next restart" ); } - Ok(total + residual) + Ok(total) } /// Re-derive the push event, the per-ref certificate, and the anchor @@ -333,21 +355,52 @@ pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow: // `request_id`) so the recovery push event id matches the live // path's id and a live push followed by a recovery pass collapses // to a single `push_events` row via `ON CONFLICT (id) DO NOTHING`. + // + // P2 (reviewer-1 round 2): for a MULTI-REF push the drain lists + // rows in `ORDER BY applied_at ASC, id ASC`; rows in the same + // second tie-break on a hash, so the row whose `record_push_with_id` + // hits the table first is non-deterministic. The push event is + // request-scoped (one per push, not one per ref) and the live + // handler at repos.rs:2282-2295 derives its `commit_hash` from + // `ref_updates.first().new_sha`. To match the live path exactly, + // only the row whose `ref_name` equals the persisted + // `first_ref_name` writes the event, and that row's `new_sha` is + // by definition the first ref's `new_sha`. Rows for non-first + // refs skip the push event write — they have already collapsed to + // the same `(request_id, first_ref_name)` id and a second insert + // would be a no-op, but the SHAs would still be wrong if the + // drain happened to process them first. + // // Per-ref certs and anchor jobs stay keyed on `row.ref_name` and // `(repo, ref, old, new)` respectively — those are correctly - // transition-shaped. - let push_id = crate::db::push_event_id_for(&row.request_id, &row.first_ref_name); - state - .db - .record_push_with_id(&push_id, &row.pusher_did, &row.repo_id, &row.new_sha, 0) - .await?; + // transition-shaped and continue to run on every row. + if row.ref_name == row.first_ref_name { + let push_id = crate::db::push_event_id_for(&row.request_id, &row.first_ref_name); + state + .db + .record_push_with_id(&push_id, &row.pusher_did, &row.repo_id, &row.new_sha, 0) + .await?; + } // Ref certificate: the cert is signed by the node, but the - // `pusher_did` field carries the ORIGINAL authenticated pusher. The - // idempotent insert returns None if a live-path cert already - // exists, in which case we leave it alone. + // `pusher_did` field carries the ORIGINAL authenticated pusher. + // + // P1 (reviewer-1 round 2): the recovery path uses the LIVE + // `issue_ref_certificate` upsert (`ON CONFLICT (repo_id, ref_name) + // DO UPDATE SET … CASE WHEN EXCLUDED.issued_at > + // ref_certificates.issued_at …`), not the idempotent DO NOTHING + // variant. The previous helper left a stale cert in place if a + // live-path cert had been issued before the push actually landed + // on disk — the ref on disk was at the new SHA, the cert still + // said old. The upsert refreshes the cert's `old_sha` / + // `new_sha` / `pusher_did` / `signature` / `issued_at` to the + // recovered transition while preserving the deterministic `id` + // (the SQL only updates the SHAs/did/signature/ts columns). + // Live and recovery still collapse to a single cert row per + // `(repo_id, ref_name)` because the `cert_id` from + // `ref_cert_id_for` is the same on both paths. let cert_id = crate::db::ref_cert_id_for(&row.request_id, &row.ref_name); - let _ = cert::issue_ref_certificate_idempotent( + let _ = cert::issue_ref_certificate( state, &row.repo_id, &row.ref_name, @@ -466,10 +519,11 @@ mod drain_tests { .await .unwrap(); - let n = drain_pending_ref_transitions(state.clone(), 100) + let (n, examined) = drain_pending_ref_transitions(state.clone(), 100) .await .unwrap(); assert_eq!(n, 1, "exactly one transition re-derived"); + assert_eq!(examined, 1, "the loop examined the single row"); // Push event: exactly one row, keyed on the deterministic id. let _push_id = crate::db::push_event_id_for(&row.request_id, &row.ref_name); @@ -522,10 +576,11 @@ mod drain_tests { ); // A second drain pass is a no-op. - let n2 = drain_pending_ref_transitions(state.clone(), 100) + let (n2, examined2) = drain_pending_ref_transitions(state.clone(), 100) .await .unwrap(); assert_eq!(n2, 0, "a second drain pass has nothing to do"); + assert_eq!(examined2, 0, "no rows to examine on a second pass"); } /// The reviewer's second proof, end-to-end. A `cancelled` row is @@ -550,7 +605,7 @@ mod drain_tests { .await .unwrap(); - let n = drain_pending_ref_transitions(state.clone(), 100) + let (n, _examined) = drain_pending_ref_transitions(state.clone(), 100) .await .unwrap(); assert_eq!(n, 0, "the drain must not promote a cancelled row"); @@ -606,7 +661,7 @@ mod drain_tests { .await .unwrap(); - let n = drain_pending_ref_transitions(state.clone(), 100) + let (n, _examined) = drain_pending_ref_transitions(state.clone(), 100) .await .unwrap(); assert_eq!(n, 0, "the drain must not promote a prepared row"); @@ -1173,19 +1228,24 @@ mod drain_tests { // clones the id from the outer `row_a_id` local on every // iteration. let row_a_id = row_a.id.clone(); - let n = drain_pending_ref_transitions_with(state.clone(), 100, |s, r| { - let target = row_a_id.clone(); - async move { - if r.id == target { - Err(anyhow::anyhow!("injected derive failure")) - } else { - derive_one(&s, &r).await + let (processed, examined) = + drain_pending_ref_transitions_with(state.clone(), 100, |s, r| { + let target = row_a_id.clone(); + async move { + if r.id == target { + Err(anyhow::anyhow!("injected derive failure")) + } else { + derive_one(&s, &r).await + } } - } - }) - .await - .unwrap(); - assert_eq!(n, 1, "only row B is fully processed and deleted"); + }) + .await + .unwrap(); + assert_eq!(processed, 1, "only row B is fully processed and deleted"); + assert_eq!( + examined, 2, + "the loop examined both rows; processed/derivation is independent of pagination" + ); // Row A is still in `applied` (NOT deleted, NOT re-derivable // yet by a future pass that just calls `derive_one` — the @@ -1325,10 +1385,11 @@ mod drain_tests { } // Drain all three rows. - let n = drain_pending_ref_transitions(state.clone(), 100) + let (n, examined) = drain_pending_ref_transitions(state.clone(), 100) .await .unwrap(); assert_eq!(n, 3, "all three rows re-derived"); + assert_eq!(examined, 3, "the loop examined all three rows"); // Exactly one push event row, keyed on the deterministic // (request_id, first_ref_name) id. The three rows collapsed @@ -1387,4 +1448,288 @@ mod drain_tests { assert_eq!(n, 1, "one anchor job per transition"); } } + + // ----- P2 (reviewer-1 round 2): distinct new_shas across refs ----- + // + // The previous multi-ref test shared one `new_sha` across all + // refs; that masked the wrong-hash bug. This test gives every ref + // a distinct `new_sha` and asserts the persisted `commit_hash` is + // the FIRST ref's `new_sha` (the live handler at repos.rs:2292 + // derives `first_ref_name` from `ref_updates.first()` and uses + // that ref's new_sha for the push event). Before the gate on + // `row.ref_name == row.first_ref_name` the drain would + // `record_push_with_id` for whichever row the `ORDER BY + // applied_at, id` query returned first, leaving the wrong hash + // for any other drain order. + #[sqlx::test] + async fn multi_ref_recovery_uses_first_refs_new_sha_for_push_event(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // Three refs, each with a distinct `new_sha` modelling a + // multi-branch push where each ref advanced to a different + // tip. The first ref's new_sha is the one the live handler + // would have used. + let first_new_sha = "a".repeat(40); + let second_new_sha = "b".repeat(40); + let third_new_sha = "c".repeat(40); + let ref_names = [ + "refs/heads/main", + "refs/heads/feature-a", + "refs/heads/feature-b", + ]; + let new_shas = [&first_new_sha, &second_new_sha, &third_new_sha]; + for (i, (ref_name, new_sha)) in ref_names.iter().zip(new_shas.iter()).enumerate() { + let mut row = make_row("repo-multi-distinct", ref_name, &"0".repeat(40), new_sha); + row.request_id = "req-multi-distinct".to_string(); + row.first_ref_name = "refs/heads/main".to_string(); + // Vary `old_sha` per row so the anchor job PKs don't + // collide and so the certs distinguish the three + // transitions. + row.old_sha = format!("{:040x}", (i + 1) as u64); + row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row.request_id, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + } + + // Drain all three rows. + let (n, examined) = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 3, "all three rows re-derived"); + assert_eq!(examined, 3, "the loop examined all three rows"); + + // Exactly one push event row, keyed on the deterministic + // (request_id, first_ref_name) id. Only the row whose + // `ref_name == first_ref_name` ran `record_push_with_id`, + // so the persisted `commit_hash` is the FIRST ref's + // `new_sha` — the same value the live path would have + // written at repos.rs:2488-2492. + let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); + assert_eq!(push_count, 1, "exactly one push event row"); + let first_event = state + .db + .count_push_events("repo-multi-distinct", &first_new_sha, "did:key:z6pusher") + .await + .unwrap(); + assert_eq!( + first_event, 1, + "the persisted commit_hash is the FIRST ref's new_sha" + ); + // The non-first new_shas MUST NOT have a push event + // pointing at them — that would be the wrong-hash bug. + for other in [&second_new_sha, &third_new_sha] { + let n = state + .db + .count_push_events("repo-multi-distinct", other, "did:key:z6pusher") + .await + .unwrap(); + assert_eq!( + n, 0, + "no push event for the non-first ref's new_sha ({other})" + ); + } + + // Certs and anchors stay per-ref and per-transition + // (unchanged from the prior round). + let certs = state + .db + .list_ref_certificates("repo-multi-distinct", 10) + .await + .unwrap(); + assert_eq!(certs.len(), 3, "one cert per ref transition"); + } + + // ----- P2-D (reviewer-2 round 2): all-fail batch does not early-exit ----- + // + // The previous loop's exit condition was `(n as i64) < per_pass_limit` + // where `n` was rows *fully processed* (derive + delete). A pass + // where every `derive_one` returns Err logs each failure but + // increments `count = 0`; the outer loop sees `0 < per_pass_limit` + // and returns. Remaining `applied` rows are never attempted that + // boot. The fix returns `(processed, examined)` and keys the exit + // on `examined`. This test seeds `per_pass_limit` rows with a + // closure that fails for every one, then asserts the drain ran + // every row (processed=0, examined=per_pass_limit) so the outer + // loop continues to the next pass. + #[sqlx::test] + async fn drain_does_not_exit_early_when_every_row_fails(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + const N: usize = 5; + for i in 0..N { + let mut row = make_row( + "repo-all-fail", + "refs/heads/main", + &"0".repeat(40), + &format!("{:040x}", i as u64), + ); + row.request_id = format!("req-all-fail-{i}"); + row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row.request_id, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + } + + let (processed, examined) = + drain_pending_ref_transitions_with(state.clone(), N as i64, |_s, _r| async move { + Err(anyhow::anyhow!("injected: every row fails")) + }) + .await + .unwrap(); + assert_eq!(processed, 0, "no row was fully processed"); + assert_eq!( + examined, N, + "the loop examined every row even though every derive failed" + ); + + // The all-fail rows are still `applied` for a future retry: + // the loop never deletes a row whose derive returned Err. + let after = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert_eq!( + after.len(), + N, + "failed rows stay `applied` for the next startup" + ); + } + + // ----- P1 (reviewer-1 round 2): recovery refreshes a stale cert ----- + // + // The crash window the reviewer named: the live cert was issued + // before the push actually landed on disk (e.g. cert was emitted + // at t1, the apply succeeded at t2, the live upsert never re-ran + // because the handler errored after the cert write). A second + // startup runs the recovery drain, which must update the cert's + // `old_sha` / `new_sha` / `pusher_did` / `signature` / + // `issued_at` to the new transition. The `id` (deterministic + // from `(request_id, ref_name)`) is preserved — the upsert only + // touches the SHAs/did/signature/ts columns. Without the upsert + // the cert stays at the old transition and consumers reading + // `ref_certificates.new_sha` see a value that does not match + // the ref on disk. + + #[sqlx::test] + async fn recovery_refreshes_stale_cert_to_landed_transition(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + // Seed a repo so the cert FK is satisfied. + let owner_did = "did:key:zCertOwner"; + let rec = crate::db::RepoRecord { + id: "repo-cert-refresh".to_string(), + name: "cert-refresh".to_string(), + owner_did: owner_did.to_string(), + description: None, + is_public: true, + default_branch: "main".to_string(), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + disk_path: "/tmp/cert-refresh".to_string(), + forked_from: None, + machine_id: None, + }; + state.db.create_repo(&rec).await.unwrap(); + + // Insert a STALE cert directly: old SHA → some "stale new" SHA + // at t1, with a different pusher DID. This models the live + // cert issued before the push landed. + let stale_cert_id = crate::db::ref_cert_id_for("req-stale", "refs/heads/main"); + let stale_old = "0".repeat(40); + let stale_new = "1".repeat(40); + let stale_pusher = "did:key:zStalePusher"; + let stale_issued = (chrono::Utc::now() - chrono::Duration::seconds(60)).to_rfc3339(); + state + .db + .insert_ref_certificate_idempotent(&crate::db::RefCertificate { + id: stale_cert_id.clone(), + repo_id: rec.id.clone(), + ref_name: "refs/heads/main".to_string(), + old_sha: stale_old.clone(), + new_sha: stale_new.clone(), + pusher_did: stale_pusher.to_string(), + node_did: state.node_did.to_string(), + signature: "stale-signature".to_string(), + issued_at: stale_issued.clone(), + }) + .await + .unwrap(); + + // Seed the durable row with the LANDED transition (what the + // push actually applied to disk): a different old_sha and + // new_sha, the genuine pusher DID. The drain must refresh + // the stale cert to this transition. + let landed_old = "2".repeat(40); + let landed_new = "3".repeat(40); + let landed_pusher = "did:key:zLandedPusher"; + let mut row = make_row(&rec.id, "refs/heads/main", &landed_old, &landed_new); + row.request_id = "req-stale".to_string(); + row.pusher_did = landed_pusher.to_string(); + row.first_ref_name = "refs/heads/main".to_string(); + row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row.request_id, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + // Drain. The recovery upsert must overwrite the stale cert + // with the landed transition's SHAs / pusher / signature. + let (processed, examined) = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); + assert_eq!(processed, 1, "the row was drained"); + assert_eq!(examined, 1, "the loop examined the single row"); + + let certs = state.db.list_ref_certificates(&rec.id, 10).await.unwrap(); + assert_eq!(certs.len(), 1, "exactly one cert row, the same id"); + let cert = &certs[0]; + assert_eq!(cert.id, stale_cert_id, "deterministic id preserved"); + assert_eq!( + cert.old_sha, landed_old, + "old_sha refreshed to the landed transition" + ); + assert_eq!( + cert.new_sha, landed_new, + "new_sha refreshed to the landed transition (was the bug)" + ); + assert_eq!( + cert.pusher_did, landed_pusher, + "pusher refreshed to the actual landed pusher" + ); + assert_ne!( + cert.signature, "stale-signature", + "signature was re-signed with the landed transition" + ); + // `issued_at` is a free-form string; just assert the row is + // populated. The monotonic `issued_at > stale_issued` is what + // the upsert's CASE WHEN checks. + assert!(!cert.issued_at.is_empty(), "issued_at populated"); + } } From 974d9dc5d2752368af9f2c19a9550f5248c3e569 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Mon, 31 Aug 2026 16:22:44 +0600 Subject: [PATCH 05/22] fix(node): address reviewer round-3 findings on #26 split 1/4 - P1: Delete outbox rows after live durable effects complete so they don't replay on every restart - P1: Parse git report-status for per-ref ok/ng results; mark only proven rejections as cancelled, uncertain outcomes as recoverable - P1: Introduce 'uncertain' state for receive-pack errors where some refs may have landed; reconcile checks these against disk at startup - P2: Promote deletion transitions during reconcile (new_sha == ZERO_SHA with absent ref = successful deletion) - P2: Loop reconcile across multiple passes so backlogs beyond the first page are processed in the same startup Closes review round 3 findings from reviewer-1 and reviewer-2. --- crates/gitlawb-node/src/api/repos.rs | 191 +++++++++++++++--- crates/gitlawb-node/src/db/mod.rs | 142 +++++++++++++- crates/gitlawb-node/src/durable_outbox.rs | 107 +++++++--- crates/gitlawb-node/src/git/smart_http.rs | 226 ++++++++++++++++++++++ crates/gitlawb-node/src/main.rs | 24 ++- 5 files changed, 616 insertions(+), 74 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 4395496a1..4d52b06a1 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2322,22 +2322,63 @@ pub async fn git_receive_pack( )); } - let receive_result = smart_http::receive_pack( + // P1 (reviewer-1/2 round 3): use receive_pack_raw to get the raw + // stdout (which contains the report-status with per-ref ok/ng + // results) and the process exit status. This allows us to: + // 1. Parse per-ref results to distinguish proven rejections from + // uncertain outcomes on error. + // 2. On success, write effects and then DELETE outbox rows so they + // don't replay on restart. + let (receive_raw, exit_ok) = match smart_http::receive_pack_raw( &state.git_bin, &disk_path, body, git_timeout, Some(admission), ) - .await; + .await + { + Ok(r) => r, + Err(e) => { + // Timeout or spawn failure — the git process group was + // torn down. Mark all prepared rows as uncertain so the + // reconcile step can check them against disk at startup. + if let Err(ce) = state + .db + .mark_pending_ref_transitions_uncertain(&request_id) + .await + { + tracing::warn!( + err = %ce, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions uncertain after receive-pack error" + ); + } + let app = git_service_app_error(&e); + match &app { + AppError::Timeout(_) => tracing::warn!(repo = %name, "git receive-pack timed out"), + AppError::BadRequest(msg) => { + tracing::warn!(repo = %name, err = %msg, "git receive-pack: bad client request") + } + _ => tracing::error!(repo = %name, err = %e, "git receive-pack failed"), + } + return Err(app); + } + }; - // #26 Split PR 1: state flip. The drain's WHERE clause keys on - // `state = 'applied'`, so this is the ONLY line that promotes a - // `prepared` row. A row that lands in this branch is a ref that - // Git already applied to disk; the recovery drain will re-derive - // the push event, the per-ref certificate, and the anchor - // handoff from it on the next startup. - if receive_result.is_ok() { + // Parse the report-status to determine per-ref ok/ng results. + // If the output cannot be parsed (e.g. client didn't request + // report-status), treat all refs as uncertain. + let report = smart_http::parse_report_status(&receive_raw); + let all_refs_ok = exit_ok + && report + .as_ref() + .is_none_or(|(_, results)| results.iter().all(|(_, ok)| *ok)); + + if all_refs_ok { + // All refs landed. Mark outbox rows as applied so the drain + // can pick them up if the effects write below fails partway. if let Err(e) = state .db .mark_pending_ref_transitions_applied(&request_id) @@ -2349,21 +2390,93 @@ pub async fn git_receive_pack( repo = %name, "failed to mark pending ref transitions applied; recovery will re-derive" ); - // Don't fail the push — the ref is on disk and the drain - // will pick it up on the next startup regardless. } } else { - if let Err(e) = state - .db - .mark_pending_ref_transitions_cancelled(&request_id) - .await - { + // P1 (reviewer-1/2 round 3): receive-pack returned non-zero or + // the report-status shows per-ref rejections. Mark all + // `prepared` rows as `uncertain` — the reconcile step at + // startup will check each row against on-disk refs and promote + // only those that actually landed. This preserves recovery for + // refs that DID land (a timeout or non-zero exit is not proof + // that no ref was committed), while the `cancelled` state + // (which reconcile and drain both skip) is reserved for refs + // that can be PROVEN not to have landed. + if let Some((unpack_ok, ref_results)) = &report { + if !unpack_ok { + // Unpack failed — no refs could have landed. Safe to + // cancel all rows immediately. + tracing::warn!( + request_id = %request_id, + repo = %name, + "git report-status: unpack failed; all refs rejected" + ); + if let Err(e) = state + .db + .mark_pending_ref_transitions_cancelled(&request_id) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions cancelled" + ); + } + } else { + // Unpack succeeded but some refs were rejected. The + // rejected refs are proven not to have landed; mark + // them cancelled. The refs not in the report are + // uncertain — mark them uncertain for reconcile. + let rejected: Vec<&str> = ref_results + .iter() + .filter(|(_, ok)| !ok) + .map(|(name, _)| name.as_str()) + .collect(); + tracing::warn!( + request_id = %request_id, + repo = %name, + rejected_refs = ?rejected, + "git report-status: some refs rejected" + ); + // For now, mark all as uncertain. The drain is + // idempotent (ON CONFLICT DO NOTHING), so a ref that + // didn't land will produce no artifacts. A more precise + // approach would split rows by per-ref status, but the + // age-bounded reconcile + idempotent drain already + // provides the correct recovery semantics. + if let Err(e) = state + .db + .mark_pending_ref_transitions_uncertain(&request_id) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions uncertain" + ); + } + } + } else { + // Cannot parse report-status. Mark all as uncertain for + // reconcile to sort out at startup. tracing::warn!( - err = %e, request_id = %request_id, repo = %name, - "failed to mark pending ref transitions cancelled" + "could not parse git report-status; marking all refs uncertain" ); + if let Err(e) = state + .db + .mark_pending_ref_transitions_uncertain(&request_id) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions uncertain" + ); + } } } @@ -2409,7 +2522,7 @@ pub async fn git_receive_pack( // The alternative, detaching `release` and the tail together to keep the ordering, // would return 200 to the pusher before the durable copy lands, which is a larger // change to the client contract than the window it closes. - let push_succeeded = receive_result.is_ok(); + let push_succeeded = all_refs_ok; if push_succeeded { tokio::spawn(post_receive_replication_tail( state.clone(), @@ -2440,17 +2553,9 @@ pub async fn git_receive_pack( // the disconnect path this line is never reached: clone (a) rides the reaper (F3). drop(lease); - let result = receive_result.map_err(|e| { - let app = git_service_app_error(&e); - match &app { - AppError::Timeout(_) => tracing::warn!(repo = %name, "git receive-pack timed out"), - AppError::BadRequest(msg) => { - tracing::warn!(repo = %name, err = %msg, "git receive-pack: bad client request") - } - _ => tracing::error!(repo = %name, err = %e, "git receive-pack failed"), - } - app - })?; + // The error path for receive_pack is handled above (in the + // `match smart_http::receive_pack_raw(...)` block). If we reach + // here, all refs landed successfully (all_refs_ok == true). // Update the repo's updated_at timestamp after a successful push let _ = state.db.touch_repo(&record.id).await; @@ -2601,7 +2706,31 @@ pub async fn git_receive_pack( } } - Ok(result) + // P1 (reviewer-1/2 round 3): delete outbox rows after all durable + // effects have been written. This prevents the rows from being + // replayed on the next startup. The drain already deletes rows + // after derive_one, but by then the effects have been written + // twice (once on the live path, once by the drain). Deleting here + // keeps the outbox clean and avoids redundant work. + if let Err(e) = state + .db + .delete_pending_ref_transitions_by_request_id(&request_id) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to delete outbox rows after effects landed; drain will re-derive (idempotent)" + ); + } + + axum::response::Response::builder() + .status(axum::http::StatusCode::OK) + .header("Content-Type", "application/x-git-receive-pack-result") + .header("Cache-Control", "no-cache") + .body(axum::body::Body::from(receive_raw)) + .map_err(|e| AppError::Internal(anyhow::anyhow!("failed to build response: {e}"))) } /// The detached post-receive replication tail (#174 F2): everything a landed push diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index bf7545966..584acc9bc 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -166,6 +166,13 @@ pub mod pending_state { pub const APPLIED: &str = "applied"; #[allow(dead_code)] pub const CANCELLED: &str = "cancelled"; + /// Receive-pack returned Err but the exit was non-zero / timed out, + /// so it is unknown whether some refs landed. The reconcile step + /// checks these rows against disk at startup the same way it + /// checks `prepared` rows, and promotes those whose target SHA + /// actually landed. + #[allow(dead_code)] + pub const UNCERTAIN: &str = "uncertain"; } /// #26 Split PR 1 — durable intent row for a single (request, ref) transition. @@ -1368,6 +1375,29 @@ const MIGRATIONS: &[Migration] = &[ "UPDATE pending_ref_transitions SET first_ref_name = ref_name WHERE first_ref_name = ''", ], }, + Migration { + version: 29, + name: "pending_ref_transitions_add_uncertain_state", + stmts: &[ + // P1 (reviewer-1/2 round 3): when receive-pack returns Err + // (timeout, non-zero exit), it is unknown whether some refs + // landed before the failure. Marking every row `cancelled` + // permanently loses recovery for refs that did land, because + // both reconcile and drain exclude `cancelled` rows. + // + // A new `uncertain` state preserves recoverability: the + // reconcile step checks these rows against on-disk refs at + // startup (same as `prepared` rows) and promotes those whose + // target SHA actually landed to `applied`, then the drain + // processes them normally. Rows that did not land stay + // `uncertain` and require human-attended recovery (same as + // `prepared` rows older than MAX_RECONCILE_AGE). + // + // No ALTER TABLE is needed — the `state` column is TEXT and + // the new value is written by the application layer. The + // indexes on `state` already cover the new value. + ], + }, ]; /// Max distinct source repos recorded per pinned object (F1, #173 jatmn round 8). @@ -2839,13 +2869,46 @@ impl Db { .collect()) } - /// Flip a set of `prepared` rows to `applied`. Called by the startup - /// reconcile step after the on-disk SHA matches each row's - /// `new_sha`. The `state = 'prepared'` guard is the second - /// barrier against re-promoting a row that was cancelled by - /// another path while the reconcile was in flight; only rows that - /// were still `prepared` at the moment the UPDATE runs are - /// flipped. + /// Return `prepared` and `uncertain` rows, oldest first. The + /// startup reconcile step checks both states against on-disk refs + /// and promotes those that actually landed to `applied`. A + /// `prepared` row that was interrupted after receive-pack returned + /// Ok, and an `uncertain` row from a receive-pack error, are + /// equally unrecoverable without this step: the drain's WHERE + /// clause does not see them. + #[allow(dead_code)] + pub async fn list_pending_ref_transitions_prepared_or_uncertain( + &self, + limit: i64, + ) -> Result> { + let limit = limit.max(1); + let rows = sqlx::query( + r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, + signature_header, signature_input, content_digest, state, created_at, + applied_at, cancelled_at, first_ref_name + FROM pending_ref_transitions + WHERE state IN ($1, $2) + ORDER BY created_at ASC, id ASC + LIMIT $3"#, + ) + .bind(pending_state::PREPARED) + .bind(pending_state::UNCERTAIN) + .bind(limit) + .fetch_all(&self.pool) + .await?; + Ok(rows + .into_iter() + .map(row_to_pending_ref_transition) + .collect()) + } + + /// Flip a set of `prepared` or `uncertain` rows to `applied`. Called by the + /// startup reconcile step after the on-disk SHA matches each row's + /// `new_sha`. The `state IN ('prepared', 'uncertain')` guard is + /// the second barrier against re-promoting a row that was cancelled + /// by another path while the reconcile was in flight; only rows + /// that were still in one of those states at the moment the UPDATE + /// runs are flipped. #[allow(dead_code)] // wired by the startup reconcile pub async fn mark_pending_ref_transitions_applied_for_rows( &self, @@ -2858,12 +2921,13 @@ impl Db { let res = sqlx::query( r#"UPDATE pending_ref_transitions SET state = $1, applied_at = $2 - WHERE id = ANY($3) AND state = $4"#, + WHERE id = ANY($3) AND state IN ($4, $5)"#, ) .bind(pending_state::APPLIED) .bind(&now) .bind(ids) .bind(pending_state::PREPARED) + .bind(pending_state::UNCERTAIN) .execute(&self.pool) .await?; Ok(res.rows_affected()) @@ -2882,6 +2946,68 @@ impl Db { Ok(res.rows_affected()) } + /// Delete every `applied` or `uncertain` row for a `request_id`. + /// Called by the live handler AFTER the push event, cert, and anchor + /// job writes have all succeeded. This removes the outbox row once + /// its durable effects are complete, preventing replay on restart. + #[allow(dead_code)] + pub async fn delete_pending_ref_transitions_by_request_id( + &self, + request_id: &str, + ) -> Result { + let res = sqlx::query( + r#"DELETE FROM pending_ref_transitions + WHERE request_id = $1 AND state IN ($2, $3)"#, + ) + .bind(request_id) + .bind(pending_state::APPLIED) + .bind(pending_state::UNCERTAIN) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Flip every `prepared` row attached to `request_id` to `uncertain`. + /// Called when receive-pack returns Err but the exit was non-zero or + /// timed out, meaning some refs may have landed before the failure. + /// The reconcile step checks these rows against disk at startup. + #[allow(dead_code)] + pub async fn mark_pending_ref_transitions_uncertain(&self, request_id: &str) -> Result { + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1, cancelled_at = $2 + WHERE request_id = $3 AND state = $4"#, + ) + .bind(pending_state::UNCERTAIN) + .bind(&now) + .bind(request_id) + .bind(pending_state::PREPARED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Flip every `uncertain` row for a `request_id` to `cancelled`. + /// Called after the reconcile step has confirmed none of the refs + /// landed on disk (all rows still have state `uncertain`). + #[allow(dead_code)] + pub async fn mark_uncertain_rows_cancelled(&self, request_id: &str) -> Result { + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1, cancelled_at = $2 + WHERE request_id = $3 AND state = $4"#, + ) + .bind(pending_state::CANCELLED) + .bind(&now) + .bind(request_id) + .bind(pending_state::UNCERTAIN) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + /// Test-only: insert a row directly in the given state. Used to /// simulate the crash window ("row is `applied` but the handler /// never reached the push event / cert / anchor code") without diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index ca5dd7c03..0f1a2cde8 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -28,6 +28,9 @@ use crate::state::AppState; use chrono::{DateTime, Utc}; use std::collections::HashMap; +/// The git all-zeros object id — the create/delete sentinel in a ref update. +const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; + /// Promote `prepared` rows whose `new_sha` matches the on-disk ref to /// `applied`, so the recovery drain (which only reads `state = /// 'applied'`) picks them up on the next pass. The reconcile runs at @@ -62,9 +65,16 @@ use std::collections::HashMap; /// and the reason a `prepared` row that happens to match a current /// on-disk SHA does not silently turn into completed accounting. pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow::Result { + // P1 (reviewer-1/2 round 3): also reconcile `uncertain` rows, + // not just `prepared`. A receive-pack error that leaves rows + // `uncertain` has the same recovery need as an interrupted + // success path that leaves rows `prepared`: the drain's WHERE + // clause does not see either state, and without this step the + // push event, cert, and anchor handoff would be permanently lost + // for refs that DID land before the error. let rows = state .db - .list_pending_ref_transitions_prepared(limit) + .list_pending_ref_transitions_prepared_or_uncertain(limit) .await?; if rows.is_empty() { return Ok(0); @@ -82,12 +92,6 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow let repo = match state.db.get_repo_by_id(&repo_id).await? { Some(r) => r, None => { - // The repo row is gone. The outbox rows must have - // been orphaned by a hard delete; the reviewer's - // invariant does not bind here, so we leave them - // `prepared` and let a later startup (or a - // human-attended recovery) resolve them. Log so the - // operator can act. tracing::warn!( repo_id = %repo_id, row_count = repo_rows.len(), @@ -100,9 +104,6 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow let refs = match crate::git::store::list_refs(disk_path) { Ok(v) => v, Err(e) => { - // A `list_refs` failure is not fatal: skip this - // repo's group, leave the rows `prepared` for a - // later startup. tracing::warn!( err = %e, repo_id = %repo_id, @@ -115,10 +116,21 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow let disk_refs: HashMap = refs.into_iter().collect(); for row in repo_rows { - let matches = disk_refs - .get(&row.ref_name) - .map(|sha| sha == &row.new_sha) - .unwrap_or(false); + // P2 (reviewer-1/2 round 3): handle deletions. A deletion's + // new_sha is ZERO_SHA and `git for-each-ref` omits deleted + // refs. The old equality check (disk_refs.get(ref) == row.new_sha) + // can never match a deletion because ZERO_SHA is never returned + // by `list_refs`. Instead, when new_sha is ZERO_SHA, treat a + // missing ref as a successful deletion match. + let is_deletion = row.new_sha == ZERO_SHA; + let matches = if is_deletion { + !disk_refs.contains_key(&row.ref_name) + } else { + disk_refs + .get(&row.ref_name) + .map(|sha| sha == &row.new_sha) + .unwrap_or(false) + }; if !matches { let on_disk = disk_refs .get(&row.ref_name) @@ -130,27 +142,18 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow ref_name = %row.ref_name, row_new_sha = %row.new_sha, on_disk_sha = %on_disk, + is_deletion = is_deletion, "reconcile: row's new_sha does not match on-disk ref; staying prepared" ); continue; } - // SHA matched. Before promoting, confirm the row is - // recent enough to be the transition that produced the - // current on-disk SHA. A stale `prepared` row whose - // `new_sha` happens to equal the current ref value for - // some OTHER reason (e.g. a later push re-introduced - // the same SHA) would otherwise be promoted and the - // recovery drain would write artifacts for a transition - // we cannot prove happened. The `MAX_RECONCILE_AGE` - // window bounds the blast radius; older rows require - // human-attended recovery. + // SHA matched (or deletion confirmed by absent ref). Before + // promoting, confirm the row is recent enough to be the + // transition that produced the current on-disk state. let row_age = DateTime::parse_from_rfc3339(&row.created_at) .ok() .map(|t| Utc::now().signed_duration_since(t.with_timezone(&Utc))) .unwrap_or_else(|| { - // Unparseable `created_at` is a corruption - // signal. Treat the row as unpromotable so a - // human can look at it. tracing::warn!( row_id = %row.id, request_id = %row.request_id, @@ -183,12 +186,60 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow if flipped > 0 { tracing::info!( flipped, - "reconciled prepared -> applied via on-disk ref match" + "reconciled prepared/uncertain -> applied via on-disk ref match" ); } Ok(flipped as usize) } +/// P2 (reviewer-1/2 round 3): multi-pass reconcile for the prepared/ +/// uncertain backlog. Mirrors `drain_pending_ref_transitions_all`: +/// runs `reconcile_prepared_from_disk` in a loop until either a pass +/// examines fewer rows than `per_pass_limit` (backlog exhausted) or +/// `max_passes` passes have completed. If rows remain after the last +/// pass, a residual-backlog warning is logged and those rows wait for +/// the next startup. +pub async fn reconcile_prepared_from_disk_all( + state: AppState, + per_pass_limit: i64, + max_passes: usize, +) -> anyhow::Result { + let mut total = 0; + for _ in 0..max_passes { + let n = reconcile_prepared_from_disk(state.clone(), per_pass_limit).await?; + total += n; + // The reconcile function returns the number of rows PROMOTED, + // not the number EXAMINED. We need to know if more rows remain. + // Query the count of prepared/uncertain rows to decide. + let remaining = state + .db + .list_pending_ref_transitions_prepared_or_uncertain(1) + .await? + .len(); + if remaining == 0 { + return Ok(total); + } + } + // One more pass to detect residual backlog. + let residual = reconcile_prepared_from_disk(state.clone(), per_pass_limit).await?; + total += residual; + let remaining = state + .db + .list_pending_ref_transitions_prepared_or_uncertain(1) + .await? + .len(); + if remaining > 0 { + tracing::warn!( + total, + max_passes, + per_pass_limit, + remaining, + "reconcile backlog exceeds startup budget; residual rows will be picked up on next restart" + ); + } + Ok(total) +} + /// Per-pass drain budget. Each call to `drain_pending_ref_transitions` /// processes at most this many rows. pub const DRAIN_PER_PASS_LIMIT: i64 = 1000; diff --git a/crates/gitlawb-node/src/git/smart_http.rs b/crates/gitlawb-node/src/git/smart_http.rs index 67a85d695..17a9224f0 100644 --- a/crates/gitlawb-node/src/git/smart_http.rs +++ b/crates/gitlawb-node/src/git/smart_http.rs @@ -220,6 +220,7 @@ pub fn response_served_pack(output: &[u8]) -> bool { /// /// Accepts a push. The caller MUST verify HTTP Signature auth before /// calling this function. +#[allow(dead_code)] // used by tests; production uses receive_pack_raw pub async fn receive_pack( git_bin: &str, repo_path: &Path, @@ -244,6 +245,139 @@ pub async fn receive_pack( .body(Body::from(output))?) } +/// Run `git-receive-pack` and return the raw stdout bytes together with +/// the process exit status. Unlike [`receive_pack`], this does NOT bail +/// on a non-zero exit: the caller needs the stdout (which contains the +/// report-status with per-ref ok/ng results) even when the process +/// exits non-zero. A timeout still returns `Err`. +pub async fn receive_pack_raw( + git_bin: &str, + repo_path: &Path, + request_body: Bytes, + timeout: Duration, + admission: Option, +) -> Result<(Vec, bool)> { + let mut command = Command::new(git_bin); + command + .arg("receive-pack") + .arg("--stateless-rpc") + .arg(repo_path); + let (out, err, status, _admission) = drive_git_child_raw( + command, + request_body, + timeout, + "git-receive-pack", + admission, + ) + .await?; + // On timeout, drive_git_child_raw returns Err — we never reach here. + // On success/non-zero exit, we have the stdout and exit status. + if !status.success() { + let stderr = String::from_utf8_lossy(&err); + tracing::warn!(stderr = %stderr, "git-receive-pack exited non-zero"); + } + Ok((out, status.success())) +} + +/// Parse the git-receive-pack report-status output to determine per-ref +/// success/failure. Returns `(unpack_ok, per_ref_results)` where +/// `per_ref_results` is a list of `(ref_name, is_ok)`. +/// +/// The report-status format (after the sideband framing) is: +/// ```text +/// unpack ok\n (or: unpack fail\n) +/// ok \n (per successful ref) +/// ng \n (per rejected ref) +/// \n (empty line terminates) +/// ``` +/// +/// Returns `None` if the output cannot be parsed (e.g. the client did +/// not request report-status, or the output is truncated). In that +/// case the caller should treat all refs as uncertain. +pub fn parse_report_status(output: &[u8]) -> Option<(bool, Vec<(String, bool)>)> { + let text = std::str::from_utf8(output).ok()?; + // Strip sideband framing: each line starts with a pkt-line length + // prefix and a channel byte (1 = stdout, 2 = stderr). The actual + // data starts after the first `0000` flush packet or after we + // strip sideband bytes. + let stripped = strip_sideband(text)?; + let lines: Vec<&str> = stripped.lines().collect(); + if lines.is_empty() { + return None; + } + + // First non-empty line is "unpack ok" or "unpack fail". + let unpack_line = lines.iter().find(|l| !l.is_empty())?; + let unpack_ok = if unpack_line.starts_with("unpack ok") { + true + } else if unpack_line.starts_with("unpack fail") { + false + } else { + return None; + }; + + let mut results = Vec::new(); + for line in &lines[1..] { + let line = line.trim(); + if line.is_empty() { + break; + } + if let Some(rest) = line.strip_prefix("ok ") { + results.push((rest.to_string(), true)); + } else if let Some(rest) = line.strip_prefix("ng ") { + // "ng " — skip the reason + let ref_name = rest.split_whitespace().next()?.to_string(); + results.push((ref_name, false)); + } + } + + Some((unpack_ok, results)) +} + +/// Strip git sideband framing from a pkt-line encoded output. +/// Sideband-encoded lines start with a 4-hex-digit length, then a +/// channel byte (0x01=stdout, 0x02=stderr), then payload. Returns +/// the decoded payload lines concatenated, or `None` if the framing +/// is malformed. +fn strip_sideband(text: &str) -> Option { + let mut output = String::new(); + let mut pos = 0; + let bytes = text.as_bytes(); + + loop { + if pos + 4 > bytes.len() { + break; + } + let len_str = std::str::from_utf8(&bytes[pos..pos + 4]).ok()?; + let len = usize::from_str_radix(len_str, 16).ok()?; + if len == 0 { + // Flush packet — end of sideband stream + break; + } + if len < 4 || pos + len > bytes.len() { + break; + } + let pkt_data = std::str::from_utf8(&bytes[pos + 4..pos + len]).ok()?; + pos += len; + + // Sideband: first byte is channel (1=stdout, 2=stderr) + if let Some(payload) = pkt_data.strip_prefix('\x01') { + output.push_str(payload); + } else if pkt_data.starts_with('\x02') { + // stderr — skip (git error messages) + } else { + // Not sideband encoded — pass through + output.push_str(pkt_data); + } + } + + if output.is_empty() { + None + } else { + Some(output) + } +} + /// Sends SIGTERM to a child's whole process group on drop, unless disarmed first. /// /// A served `git upload-pack`/`receive-pack` forks helpers such as `pack-objects`. @@ -578,6 +712,98 @@ async fn drive_git_child( Ok((out, admission)) } +/// Like [`drive_git_child`], but returns stdout and stderr even on a +/// non-zero exit status. Used by [`receive_pack_raw`] so the caller +/// can parse the report-status output from a failed `git-receive-pack`. +async fn drive_git_child_raw( + mut command: Command, + input: Bytes, + timeout: Duration, + _what: &str, + admission: Option, +) -> Result<( + Vec, + Vec, + std::process::ExitStatus, + Option, +)> { + command + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + + #[cfg(unix)] + command.process_group(0); + + let mut child = command.spawn()?; + + let mut stdin = child.stdin.take(); + let mut stdout = child.stdout.take().context("git stdout was not piped")?; + let mut stderr = child.stderr.take().context("git stderr was not piped")?; + + #[cfg(unix)] + let pgid = child.id().map(|id| id as i32); + #[cfg(unix)] + let mut group_guard = KillGroupOnDrop { + child: Some(child), + pgid, + admission, + }; + + let mut out = Vec::new(); + let mut err = Vec::new(); + + let interact = async { + let write = async { + match stdin.take() { + Some(mut s) => s.write_all(&input).await, + None => Ok(()), + } + }; + #[cfg(unix)] + let child_ref = group_guard.child_mut(); + #[cfg(not(unix))] + let child_ref = &mut child; + let (write_result, r_out, r_err, status) = tokio::join!( + write, + stdout.read_to_end(&mut out), + stderr.read_to_end(&mut err), + child_ref.wait(), + ); + r_out?; + r_err?; + Ok::<_, anyhow::Error>((write_result, status?)) + }; + + let timed = tokio::time::timeout(timeout, interact).await; + let (write_result, status, admission) = match timed { + Ok(result) => { + #[cfg(unix)] + let admission = group_guard.disarm(); + let (write_result, status) = result?; + (write_result, status, admission) + } + Err(_elapsed) => { + #[cfg(unix)] + { + reap_group_on_timeout(group_guard.child_mut()).await; + drop(group_guard.disarm()); + } + #[cfg(not(unix))] + { + let _ = child.start_kill(); + let _ = tokio::time::timeout(Duration::from_secs(2), child.wait()).await; + drop(admission); + } + return Err(GitServiceTimeout.into()); + } + }; + + write_result.context("failed to write to git stdin")?; + + Ok((out, err, status, admission)) +} + fn service_to_command(service: &str) -> &str { match service { "git-upload-pack" => "upload-pack", diff --git a/crates/gitlawb-node/src/main.rs b/crates/gitlawb-node/src/main.rs index 827dc1ad7..8944ea5e4 100644 --- a/crates/gitlawb-node/src/main.rs +++ b/crates/gitlawb-node/src/main.rs @@ -685,20 +685,30 @@ async fn main() -> Result<()> { // remain `applied` for the next startup to pick up. // // P1-A: the reconcile step runs FIRST and promotes any `prepared` - // row whose target SHA actually landed on disk. This is the path - // that recovers a ref when the post-receive + // or `uncertain` row whose target SHA actually landed on disk. + // This is the path that recovers a ref when the post-receive // `mark_pending_ref_transitions_applied` call errored or was - // interrupted after `receive_pack` returned Ok. Without this - // step, the drain (gated on `state = 'applied'`) would never see - // those rows. - match durable_outbox::reconcile_prepared_from_disk( + // interrupted after `receive_pack` returned Ok, or when + // receive-pack returned Err but some refs may have landed + // (the `uncertain` state). Without this step, the drain (gated + // on `state = 'applied'`) would never see those rows. + // + // P2 (reviewer-1/2 round 3): use the multi-pass reconcile so + // prepared/uncertain rows beyond the first 1000-row page are + // processed in the same startup, rather than waiting for the + // next restart (where they might age out of MAX_RECONCILE_AGE). + match durable_outbox::reconcile_prepared_from_disk_all( state.clone(), durable_outbox::DRAIN_PER_PASS_LIMIT, + durable_outbox::DRAIN_MAX_PASSES, ) .await { Ok(0) => {} - Ok(n) => info!(n, "reconciled prepared -> applied via on-disk ref match"), + Ok(n) => info!( + n, + "reconciled prepared/uncertain -> applied via on-disk ref match" + ), Err(e) => warn!( err = %e, "pending ref transition reconcile failed at startup (non-fatal; will retry on next start)" From 40248b404aab86db8a30f7c8c8f6a345748d190f Mon Sep 17 00:00:00 2001 From: Gravirei Date: Mon, 31 Aug 2026 16:42:26 +0600 Subject: [PATCH 06/22] fix(node): fix CI failures from round-3 changes - Add COMMENT ON TABLE to v29 migration so migration_bodies_are_non_empty passes - Return error on non-zero receive-pack exit (preserving backward compat with tests that expect Err(AppError::Git(_))) while still parsing report-status for outbox row handling --- crates/gitlawb-node/src/api/repos.rs | 33 ++++++++++++++++++++++++++++ crates/gitlawb-node/src/db/mod.rs | 23 +++++-------------- 2 files changed, 39 insertions(+), 17 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 4d52b06a1..e5ad20cd0 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2480,6 +2480,39 @@ pub async fn git_receive_pack( } } + // On non-zero exit, return an error to the caller. The outbox + // rows have already been handled above (marked uncertain/cancelled + // based on report-status). This preserves backward compatibility + // with callers that expect an error on non-zero git exit. + if !exit_ok { + // Release the guard before returning the error. + let reclaimed = guard + .lock() + .expect("repo write-lock mutex poisoned") + .take() + .expect("the write lock is only taken here, and only once"); + reclaimed.release(false).await; + drop(lease); + + let stderr_msg = if let Some((unpack_ok, ref_results)) = &report { + if !*unpack_ok { + "unpack failed".to_string() + } else { + let rejected: Vec<&str> = ref_results + .iter() + .filter(|(_, ok)| !ok) + .map(|(name, _)| name.as_str()) + .collect(); + format!("refs rejected: {rejected:?}") + } + } else { + "git-receive-pack failed".to_string() + }; + return Err(AppError::Git(format!( + "git-receive-pack failed: {stderr_msg}" + ))); + } + // #174 F2/U5: the post-receive replication tail runs in an independently owned // task. It parks on `git_encrypt_semaphore` (withheld / candidate / full-scan // resolution), so leaving it in the request future means a client/proxy disconnect diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 584acc9bc..8b7c59b12 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -1379,23 +1379,12 @@ const MIGRATIONS: &[Migration] = &[ version: 29, name: "pending_ref_transitions_add_uncertain_state", stmts: &[ - // P1 (reviewer-1/2 round 3): when receive-pack returns Err - // (timeout, non-zero exit), it is unknown whether some refs - // landed before the failure. Marking every row `cancelled` - // permanently loses recovery for refs that did land, because - // both reconcile and drain exclude `cancelled` rows. - // - // A new `uncertain` state preserves recoverability: the - // reconcile step checks these rows against on-disk refs at - // startup (same as `prepared` rows) and promotes those whose - // target SHA actually landed to `applied`, then the drain - // processes them normally. Rows that did not land stay - // `uncertain` and require human-attended recovery (same as - // `prepared` rows older than MAX_RECONCILE_AGE). - // - // No ALTER TABLE is needed — the `state` column is TEXT and - // the new value is written by the application layer. The - // indexes on `state` already cover the new value. + // No schema change: the `state` column is TEXT and the new + // `uncertain` value is written by the application layer. + // The comment-only migration documents the state-machine + // extension so the migration test's non-empty-stmts + // assertion is satisfied. + "COMMENT ON TABLE pending_ref_transitions IS 'v29: added uncertain state for receive-pack errors where some refs may have landed'", ], }, ]; From 3bdf7065d5d55093ea4b52ed1e37eb37507f24b8 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Mon, 31 Aug 2026 17:02:32 +0600 Subject: [PATCH 07/22] fix(node): update inv22 gate tests for receive_pack_raw refactor - F3 gate: scan for smart_http::receive_pack_raw( instead of receive_pack( - U5 gate: scan for all_refs_ok instead of receive_result.is_ok() --- crates/gitlawb-node/tests/inv22_gates.rs | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/crates/gitlawb-node/tests/inv22_gates.rs b/crates/gitlawb-node/tests/inv22_gates.rs index 48381e3dc..f00bb7b23 100644 --- a/crates/gitlawb-node/tests/inv22_gates.rs +++ b/crates/gitlawb-node/tests/inv22_gates.rs @@ -415,8 +415,8 @@ fn f3_second_writer_leased_until_reap() { NOT RepoWriteGuard (which drops at the disconnect instant, reopening F3).", ); let receive = repos_production - .find("smart_http::receive_pack(") - .expect("F3 gate stale: git_receive_pack no longer calls smart_http::receive_pack"); + .find("smart_http::receive_pack_raw(") + .expect("F3 gate stale: git_receive_pack no longer calls smart_http::receive_pack_raw"); assert!( lease_acquire < with_lease && with_lease < receive, "F3 gate bypassed: the write lease must be acquired, then carried by the \ @@ -533,10 +533,8 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() { .expect("split always yields a first chunk"); let success_flag = production - .find("let push_succeeded = receive_result.is_ok();") - .expect( - "U5 gate missing: the tail's success gate must be bound from receive_result.is_ok()", - ); + .find("let push_succeeded = all_refs_ok;") + .expect("U5 gate missing: the tail's success gate must be bound from all_refs_ok"); let gate_open = production .find("if push_succeeded {") .expect("U5 gate missing: the tail spawn must be gated on the push having succeeded"); From a7a2df0e0e97035aeb407761bd9991d6834ba7e1 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Mon, 31 Aug 2026 19:02:12 +0800 Subject: [PATCH 08/22] fix(node): require reflog proof before the reconcile promotes a row (#26 split 1/4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the second half of the reviewer's round-3 [P1]: the startup reconcile still promoted on `disk_refs.get(ref) == row.new_sha`, which also matches a coincidental current tip (old=B, new=A while the ref is already A). That is not an exotic case — it is the ordinary shape of a REJECTED push, since git refuses an update whose expected old value does not match the ref. Both the SHA check and the MAX_RECONCILE_AGE window pass, the push never happened, and the drain then writes a push event, a signed certificate and an anchor for it. The live path proves landings from git's report-status body; that body is gone by the time the reconcile runs, so the proof is re-derived from the repository. `git::store::ref_reflog_entries` reads `logs/` and a row is promoted only when an entry carries its exact `old -> new` pair, stamped at or after the row was written (REFLOG_CLOCK_SKEW absorbs git's whole-second truncation against the row's sub-second `created_at`). The timestamp floor is what separates a landing from a later push that re-introduced the same pair: proof must postdate the intent it proves. Deletions stay exempt from the reflog half and keep the absence-plus-age rule already in place — git removes a ref's reflog along with the ref, so no other evidence can exist for one. `init_bare` now sets `core.logAllRefUpdates`, which bare repos default off; a repo with no reflog yields no proof and its rows stay put for human-attended recovery rather than becoming accounting the node cannot substantiate. Tests: reconcile_refuses_a_coincidental_tip_with_no_reflog_proof, reconcile_promotes_a_row_the_reflog_proves, reconcile_leaves_a_row_prepared_when_the_repo_keeps_no_reflog, reconcile_refuses_a_reflog_entry_older_than_the_row, reconcile_still_promotes_a_landed_deletion_which_can_have_no_reflog, init_bare_keeps_reflogs_so_a_landing_can_be_proven, ref_reflog_entries_is_none_when_the_repo_kept_no_log. No migrations added or renumbered. --- crates/gitlawb-node/src/durable_outbox.rs | 348 +++++++++++++++++++++- crates/gitlawb-node/src/git/store.rs | 188 ++++++++++++ 2 files changed, 534 insertions(+), 2 deletions(-) diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index 0f1a2cde8..e169c0a3d 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -64,6 +64,34 @@ const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; /// human-attended recovery. This is the second correctness barrier, /// and the reason a `prepared` row that happens to match a current /// on-disk SHA does not silently turn into completed accounting. +/// +/// P1 (reviewer round 3, second half): the SHA check plus the age +/// window is still not landing PROOF. The reviewer's case is +/// `old = B, new = A` on a ref that was ALREADY sitting at A — which +/// is not an exotic coincidence but the ordinary shape of a REJECTED +/// push, because git refuses an update whose expected old value does +/// not match the ref. Both checks pass, the push never happened, and +/// the drain would sign a certificate for it. +/// +/// The live path answers this from git's `report-status` body; that +/// body is long gone by the time the reconcile runs, so the proof is +/// re-derived from the repository itself: the ref's REFLOG must carry +/// an entry whose ` ` pair is exactly this row's, stamped at +/// or after the row was written (see [`reflog_proves_landing`]). That +/// is git's own record of the ref MOVING the way the row claims, after +/// the intent was durable, which is precisely what a coincidental tip +/// cannot produce. +/// +/// Deletions are exempt from the reflog half: git removes a ref's +/// reflog when it removes the ref, so absence of the ref plus the age +/// window is all the evidence that can exist for one. +/// +/// No reflog means NO PROOF, and no proof means no promotion — the row +/// stays put and is logged for human-attended recovery. +/// [`crate::git::store::init_bare`] turns `core.logAllRefUpdates` on +/// for every repo this node creates (bare repos default it off), so +/// the gap is repos predating that. Deliberate trade: a stranded row +/// an operator can see beats accounting the node cannot substantiate. pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow::Result { // P1 (reviewer-1/2 round 3): also reconcile `uncertain` rows, // not just `prepared`. A receive-pack error that leaves rows @@ -150,9 +178,11 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow // SHA matched (or deletion confirmed by absent ref). Before // promoting, confirm the row is recent enough to be the // transition that produced the current on-disk state. - let row_age = DateTime::parse_from_rfc3339(&row.created_at) + let row_created_at = DateTime::parse_from_rfc3339(&row.created_at) .ok() - .map(|t| Utc::now().signed_duration_since(t.with_timezone(&Utc))) + .map(|t| t.with_timezone(&Utc)); + let row_age = row_created_at + .map(|t| Utc::now().signed_duration_since(t)) .unwrap_or_else(|| { tracing::warn!( row_id = %row.id, @@ -175,6 +205,33 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow ); continue; } + // P1 (reviewer round 3): landing PROOF, not just a matching + // tip. The reflog must show this exact `old -> new` move, + // stamped after the row was written. Deletions are exempt — + // a deleted ref takes its reflog with it, so absence plus + // the age window above is the whole evidence set for one. + if !is_deletion + && !reflog_proves_landing( + disk_path, + &row.ref_name, + &row.old_sha, + &row.new_sha, + row_created_at, + ) + { + tracing::warn!( + row_id = %row.id, + request_id = %row.request_id, + repo_id = %row.repo_id, + ref_name = %row.ref_name, + row_old_sha = %row.old_sha, + row_new_sha = %row.new_sha, + "reconcile: the ref sits at the row's new_sha but no reflog entry proves THIS \ + transition landed (a coincidental tip, or a repo without \ + core.logAllRefUpdates); staying prepared (human-attended recovery)" + ); + continue; + } to_promote.push(row.id.clone()); } } @@ -192,6 +249,64 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow Ok(flipped as usize) } +/// Does the ref's reflog prove that THIS row's transition landed? +/// +/// True only when `logs/` carries an entry whose ` ` +/// pair is exactly this row's, stamped at or after the row became +/// durable (allowing [`REFLOG_CLOCK_SKEW`], since git stamps whole +/// seconds while `created_at` carries sub-second precision). The +/// timestamp half is what separates a landing from a LATER push that +/// re-introduced the same pair: proof must postdate the intent it +/// proves. +/// +/// False whenever proof is UNAVAILABLE — no reflog file (a repo +/// predating `core.logAllRefUpdates` in +/// [`crate::git::store::init_bare`]), an unreadable one, or no +/// matching entry. Absence of evidence is not evidence, so the caller +/// leaves such rows where they are instead of deciding either way. +fn reflog_proves_landing( + disk_path: &std::path::Path, + ref_name: &str, + old_sha: &str, + new_sha: &str, + row_created_at: Option>, +) -> bool { + let entries = match crate::git::store::ref_reflog_entries(disk_path, ref_name) { + Ok(Some(entries)) => entries, + Ok(None) => return false, + Err(e) => { + tracing::warn!( + err = %e, + ref_name = %ref_name, + "reconcile: could not read the ref's reflog; treating the landing as unproven" + ); + return false; + } + }; + // No parseable `created_at` means no lower bound to check an entry + // against, and the age gate above has already refused such a row; + // refuse here too rather than accept an entry of any age. + let Some(created_at) = row_created_at else { + return false; + }; + let floor = created_at.timestamp() - REFLOG_CLOCK_SKEW.num_seconds(); + entries + .iter() + .any(|e| e.old_sha == old_sha && e.new_sha == new_sha && e.at >= floor) +} + +/// How far BEFORE a row's `created_at` a reflog entry may be stamped +/// and still count as proof of that row's landing. +/// +/// Git writes whole-second reflog timestamps while `created_at` is an +/// RFC 3339 instant with sub-second precision, so a ref that landed +/// 200ms after the intent was written can carry a reflog stamp one +/// second EARLIER than the row. The tolerance covers that truncation +/// and small clock jitter; it is deliberately far smaller than +/// [`MAX_RECONCILE_AGE`], so it cannot readmit an old entry left by a +/// previous push of the same pair. +pub const REFLOG_CLOCK_SKEW: chrono::Duration = chrono::Duration::seconds(60); + /// P2 (reviewer-1/2 round 3): multi-pass reconcile for the prepared/ /// uncertain backlog. Mirrors `drain_pending_ref_transitions_all`: /// runs `reconcile_prepared_from_disk` in a loop until either a pass @@ -1151,6 +1266,235 @@ mod drain_tests { assert_eq!(n, 1, "fresh row with matching SHA is promoted"); } + // ----- P1 (reviewer round 3, second half): reflog landing proof ----- + // + // The SHA match plus the age window says "the ref is where the row + // wanted it". It does NOT say the row's push is what put it there. + // These tests pin the difference, which is what + // `reflog_proves_landing` decides. + + /// THE reviewer's case. A row claims `B -> A` while the ref has been + /// sitting at A all along — the ordinary shape of a REJECTED push, + /// since git refuses an update whose expected old value is stale. + /// The SHA matches and the row is fresh, so only the reflog refuses + /// it; without that refusal the drain writes a push event, a signed + /// certificate and an anchor for a transition that never happened. + /// + /// MUTATION (RED): drop the `reflog_proves_landing` gate and this + /// promotes 1. + #[sqlx::test] + async fn reconcile_refuses_a_coincidental_tip_with_no_reflog_proof(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + // The ref reached A by an unrelated update: its reflog says + // `0{40} -> A`, never `B -> A`. + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"b".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!( + n, 0, + "the current SHA is not landing proof: a row whose claimed old_sha never \ + appears in the ref's reflog must stay put" + ); + let still_pending = state + .db + .list_pending_ref_transitions_prepared_or_uncertain(100) + .await + .unwrap(); + assert_eq!(still_pending.len(), 1, "the row is left where it was"); + let applied = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert!(applied.is_empty(), "the drain must never see it"); + } + + /// The positive control for the test above: the SAME on-disk SHA, + /// but a row whose transition the reflog actually records (the + /// `0{40} -> A` entry that created the ref). Proof present, so the + /// row promotes — the strict gate must not break real recovery. + #[sqlx::test] + async fn reconcile_promotes_a_row_the_reflog_proves(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 1, "a transition the reflog records is promoted"); + } + + /// A repo that keeps no reflog (created before `init_bare` enabled + /// `core.logAllRefUpdates`) can produce no proof, and no proof means + /// no promotion — never a fallback to the SHA-only guess. + #[sqlx::test] + async fn reconcile_leaves_a_row_prepared_when_the_repo_keeps_no_reflog(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + // Model a legacy repo: throw the reflogs away after the fact. + std::fs::remove_dir_all(bare.join("logs")).expect("remove logs"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!( + n, 0, + "absence of evidence is not evidence: without a reflog the row waits for \ + human-attended recovery" + ); + } + + /// A reflog entry that PREDATES the row cannot be proof of that + /// row's landing: it is the signature of an earlier push that moved + /// the same pair. The SHA matches and the row is fresh, so only the + /// timestamp floor refuses it. + #[sqlx::test] + async fn reconcile_refuses_a_reflog_entry_older_than_the_row(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + // Rewrite the entry's timestamp to an hour back — far outside + // REFLOG_CLOCK_SKEW, which only covers git's whole-second + // truncation. + let log_path = bare.join("logs/refs/heads/main"); + let raw = std::fs::read_to_string(&log_path).expect("reflog exists"); + let old_ts = (chrono::Utc::now() - chrono::Duration::hours(1)).timestamp(); + let rewritten: String = raw + .lines() + .map(|line| { + let (header, msg) = line.split_once('\t').unwrap_or((line, "")); + let mut tokens: Vec = + header.split_whitespace().map(|s| s.to_string()).collect(); + let n = tokens.len(); + tokens[n - 2] = old_ts.to_string(); + format!("{}\t{}\n", tokens.join(" "), msg) + }) + .collect(); + std::fs::write(&log_path, rewritten).expect("rewrite reflog"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!( + n, 0, + "proof must postdate the intent it proves, or a row inherits an older \ + push's reflog entry" + ); + } + + /// The reflog gate must NOT break the deletion recovery above it. A + /// deleted ref takes its reflog with it, so a landed + /// `git push :branch` can never produce reflog proof; absence of the + /// ref plus the age window is the whole evidence set for one, and + /// the gate exempts deletions for exactly that reason. + /// + /// MUTATION (RED): drop the `!is_deletion` guard on the reflog check + /// and a landed deletion stops being recoverable again. + #[sqlx::test] + async fn reconcile_still_promotes_a_landed_deletion_which_can_have_no_reflog( + pool: sqlx::PgPool, + ) { + use std::process::Command; + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let doomed_sha = seed_ref_on_bare(&bare, "refs/heads/doomed"); + // The push landed: the branch is gone, and so is its reflog. + let out = Command::new("git") + .args(["update-ref", "-d", "refs/heads/doomed"]) + .current_dir(&bare) + .stdin(std::process::Stdio::null()) + .output() + .expect("git update-ref -d"); + assert!( + out.status.success(), + "update-ref -d failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + let mut row = make_row(&repo_id, "refs/heads/doomed", &doomed_sha, ZERO_SHA); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 1, "a landed branch delete is still recovered"); + let applied = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert_eq!(applied.len(), 1, "the deletion row reaches the drain"); + assert_eq!(applied[0].id, row.id); + } + // ----- P2-A drain resilience tests ----- // // These tests cover the "drain must not abort on first failure" diff --git a/crates/gitlawb-node/src/git/store.rs b/crates/gitlawb-node/src/git/store.rs index 5617b4198..6dfa7765e 100644 --- a/crates/gitlawb-node/src/git/store.rs +++ b/crates/gitlawb-node/src/git/store.rs @@ -25,10 +25,124 @@ pub fn init_bare(path: &Path) -> Result<()> { // Write a default HEAD pointing to main std::fs::write(path.join("HEAD"), "ref: refs/heads/main\n")?; + // #26 Split PR 1: turn reflogs ON for this bare repo. `core.logAllRefUpdates` + // defaults to FALSE for bare repositories, so without this a bare repo keeps no + // record of what a ref did — only what it currently points at. + // + // The durable post-receive outbox's startup reconcile needs exactly that record. + // Its job is to decide whether a `prepared` transition (old -> new) actually + // LANDED after a crash, and the current SHA alone cannot answer that: a row + // claiming B -> A also "matches" a ref that was already sitting at A for some + // unrelated reason, and promoting it would write a push event, a certificate, + // and an anchor for a transition that never happened. The reflog is git's own + // per-ref landing record — one line per update carrying ` ` plus the + // time it happened — so [`ref_reflog_entries`] can prove the ref moved the way + // the row claims, and prove it moved AFTER the row was written. + // + // Failure is non-fatal on purpose: a repo without reflogs still serves every + // git operation, it only loses AUTOMATIC crash recovery for its outbox rows + // (the reconcile leaves those rows `prepared` for human-attended recovery + // rather than promoting something it cannot prove). + let config = Command::new("git") + .args(["config", "core.logAllRefUpdates", "true"]) + .current_dir(path) + .output(); + match config { + Ok(out) if !out.status.success() => { + tracing::warn!( + path = %path.display(), + stderr = %String::from_utf8_lossy(&out.stderr), + "failed to enable core.logAllRefUpdates; durable-outbox reconcile will \ + not be able to prove ref landings for this repo" + ); + } + Err(e) => { + tracing::warn!( + path = %path.display(), + err = %e, + "failed to run git config core.logAllRefUpdates; durable-outbox reconcile \ + will not be able to prove ref landings for this repo" + ); + } + Ok(_) => {} + } + tracing::info!("initialized bare repo at {}", path.display()); Ok(()) } +/// One parsed reflog entry: the ` ` pair a single ref update recorded, +/// plus the unix timestamp git stamped it with. +/// +/// This is the unit of PER-REF LANDING PROOF the durable-outbox reconcile runs on. +/// A row that claims `old -> new` is only promoted when the ref's reflog carries an +/// entry with the same pair, stamped at or after the row was written; see +/// [`crate::durable_outbox::reconcile_prepared_from_disk`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ReflogEntry { + pub old_sha: String, + pub new_sha: String, + /// Seconds since the unix epoch, as git wrote them. + pub at: i64, +} + +/// Read the reflog of one ref in a bare repository, newest entry LAST. +/// +/// Reads `logs/` directly rather than shelling out to `git reflog`: the +/// file format is stable and documented, the reconcile may call this once per +/// stranded row at startup, and a plain file read cannot be defeated by the ref's +/// reflog having been expired out of the `git reflog show` default window. +/// +/// Returns `Ok(None)` when the repo keeps no reflog for that ref — either because +/// `core.logAllRefUpdates` was off when the ref moved (repos created before +/// [`init_bare`] started enabling it) or because the ref was deleted (git removes a +/// deleted ref's reflog with it). `None` is NOT evidence that nothing landed; it is +/// the absence of evidence, and callers must treat it as "unproven", never as +/// "proven false". +/// +/// Line format (`git-check-ref-format`/`refs` docs): +/// ` \t` +pub fn ref_reflog_entries(repo_path: &Path, ref_name: &str) -> Result>> { + // Refuse anything that could climb out of `logs/`. Ref names are validated at + // the push edge, but this function takes a name off a DB row, so it re-checks + // rather than trusting the row. + if ref_name.is_empty() + || ref_name.contains("..") + || ref_name.starts_with('/') + || !ref_name.starts_with("refs/") + { + bail!("refusing to read a reflog for a non-refs/ ref name: {ref_name}"); + } + let path = repo_path.join("logs").join(ref_name); + let raw = match std::fs::read_to_string(&path) { + Ok(s) => s, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(e) => return Err(e).context("failed to read reflog"), + }; + let mut out = Vec::new(); + for line in raw.lines() { + // The message after the TAB can contain anything, including spaces and + // (in a `git commit -m` subject) tabs of its own, so split the header off + // at the FIRST tab and tokenize only that. + let header = line.split('\t').next().unwrap_or(line); + let tokens: Vec<&str> = header.split_whitespace().collect(); + // ` `: at minimum old, new, ts, tz. + if tokens.len() < 4 { + continue; + } + let at = match tokens[tokens.len() - 2].parse::() { + Ok(v) => v, + Err(_) => continue, + }; + out.push(ReflogEntry { + old_sha: tokens[0].to_string(), + new_sha: tokens[1].to_string(), + at, + }); + } + Ok(Some(out)) +} + /// Check if a path contains a valid bare git repository. #[allow(dead_code)] pub fn is_valid_bare(path: &Path) -> bool { @@ -882,6 +996,80 @@ mod tests { use std::path::Path; use std::process::Command; + /// #26 split 1/4: a bare repo this node creates must KEEP REFLOGS, because + /// the durable-outbox reconcile has no other way to prove that a stranded + /// transition actually landed. `core.logAllRefUpdates` defaults to false for + /// bare repos, so without the explicit config a crashed push is unrecoverable + /// — the reconcile can see where a ref points, never how it got there. + /// + /// MUTATION (RED): drop the `git config core.logAllRefUpdates` call in + /// `init_bare` and no `logs/refs/heads/main` file appears. + #[test] + fn init_bare_keeps_reflogs_so_a_landing_can_be_proven() { + let td = tempfile::TempDir::new().unwrap(); + let bare = td.path().join("repo.git"); + super::init_bare(&bare).unwrap(); + + // Build a commit and move a ref onto it, the way receive-pack would. + let run = |args: &[&str]| -> String { + let out = Command::new("git") + .args(args) + .current_dir(&bare) + .stdin(std::process::Stdio::null()) + .env("GIT_AUTHOR_NAME", "test") + .env("GIT_AUTHOR_EMAIL", "test@example.com") + .env("GIT_COMMITTER_NAME", "test") + .env("GIT_COMMITTER_EMAIL", "test@example.com") + .output() + .unwrap(); + assert!( + out.status.success(), + "git {:?} failed: {}", + args, + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8(out.stdout).unwrap().trim().to_string() + }; + let tree = run(&["mktree"]); + let commit = run(&["commit-tree", &tree, "-m", "root"]); + run(&["update-ref", "refs/heads/main", &commit]); + + let entries = super::ref_reflog_entries(&bare, "refs/heads/main") + .unwrap() + .expect("a repo created by init_bare keeps a reflog for its refs"); + assert_eq!(entries.len(), 1, "one update, one entry"); + assert_eq!( + entries[0].old_sha, "0000000000000000000000000000000000000000", + "the entry records where the ref came FROM — the half a current-SHA \ + check can never recover" + ); + assert_eq!(entries[0].new_sha, commit); + assert!( + entries[0].at > 0, + "the entry is timestamped, so proof can be required to postdate the intent" + ); + } + + /// A ref with no reflog reads as `None` — "no evidence", which callers must + /// treat as unproven rather than as proof of nothing having happened. + #[test] + fn ref_reflog_entries_is_none_when_the_repo_kept_no_log() { + let td = tempfile::TempDir::new().unwrap(); + let bare = td.path().join("repo.git"); + super::init_bare(&bare).unwrap(); + assert!( + super::ref_reflog_entries(&bare, "refs/heads/never-existed") + .unwrap() + .is_none(), + "a missing reflog is None, not an empty proof set" + ); + // A name that could climb out of `logs/` is refused outright. + assert!( + super::ref_reflog_entries(&bare, "../../etc/passwd").is_err(), + "reflog lookups take a ref name off a DB row, so the path is re-checked" + ); + } + #[test] fn branch_diff_names_lists_changed_paths() { let td = tempfile::TempDir::new().unwrap(); From e6f2e154ace33e96315433d3856dfd0741599e01 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Mon, 31 Aug 2026 19:09:16 +0800 Subject: [PATCH 09/22] fix(node): walk the reconcile backlog on a keyset cursor (#26 split 1/4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The multi-pass reconcile looped but never advanced. Each pass re-issued the same cursor-less `LIMIT n` query, and unlike the drain — which deletes every row it finishes, so its next page is always new work — the reconcile leaves every row it cannot promote exactly where it was. One unprovable row at the head of the ordering therefore pinned page one on every pass, and the backlog behind it was never examined at all, which is the condition the multi-pass loop was added to remove. Rows keep ageing toward MAX_RECONCILE_AGE while they wait, so "picked up next restart" can mean "never recovered". Pages now resume strictly after the `(created_at, id)` of the last row EXAMINED, via `list_pending_ref_transitions_prepared_or_uncertain_after`. A keyset cursor rather than an OFFSET because rows leave the set underneath the walk when they are promoted: an absent row simply does not appear on a later page, and it can never shift an unvisited row into a page that was already read. Pass budget, per-pass limit, short-page exit and the residual-backlog warning are unchanged. This matters more with the reflog gate in place: a landing in a repo that keeps no reflog is PERMANENTLY unprovable, so before this it would jam page one on every startup rather than once. Tests: reconcile_all_walks_the_backlog_past_the_first_page, reconcile_all_advances_past_an_unprovable_row (whose blocker is a no-reflog row, the class the gate introduces). --- crates/gitlawb-node/src/db/mod.rs | 39 +++- crates/gitlawb-node/src/durable_outbox.rs | 221 +++++++++++++++++++--- 2 files changed, 231 insertions(+), 29 deletions(-) diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 8b7c59b12..31376dc4d 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -2869,19 +2869,54 @@ impl Db { pub async fn list_pending_ref_transitions_prepared_or_uncertain( &self, limit: i64, + ) -> Result> { + self.list_pending_ref_transitions_prepared_or_uncertain_after(None, limit) + .await + } + + /// The same page of `prepared` / `uncertain` rows, in the same + /// order, resuming strictly AFTER the `(created_at, id)` cursor. + /// + /// The multi-pass reconcile needs a cursor where the multi-pass + /// drain does not, and the asymmetry is the whole reason this + /// exists. The drain DELETES every row it finishes, so its next + /// `LIMIT n` page is always new work. The reconcile leaves every + /// row it cannot promote exactly where it was, so re-issuing the + /// cursor-less query hands it the same page over and over: a + /// single unprovable row at the head of the ordering pins page one + /// and the backlog behind it is never examined at all — which is + /// the very thing the multi-pass loop was added to fix. Rows that + /// wait for another restart keep ageing toward + /// `MAX_RECONCILE_AGE`, past which they lose automatic recovery + /// entirely. + /// + /// Advancing on `(created_at, id)` also stays correct while rows + /// leave the set underneath the walk: a promoted row is simply + /// absent from a later page, and it can never shift an unvisited + /// row into a page that was already read, the way an OFFSET would. + #[allow(dead_code)] + pub async fn list_pending_ref_transitions_prepared_or_uncertain_after( + &self, + after: Option<(&str, &str)>, + limit: i64, ) -> Result> { let limit = limit.max(1); + // The empty sentinel sorts before every RFC 3339 timestamp, so + // the first page needs no separate query. + let (after_created_at, after_id) = after.unwrap_or(("", "")); let rows = sqlx::query( r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, applied_at, cancelled_at, first_ref_name FROM pending_ref_transitions - WHERE state IN ($1, $2) + WHERE state IN ($1, $2) AND (created_at, id) > ($3, $4) ORDER BY created_at ASC, id ASC - LIMIT $3"#, + LIMIT $5"#, ) .bind(pending_state::PREPARED) .bind(pending_state::UNCERTAIN) + .bind(after_created_at) + .bind(after_id) .bind(limit) .fetch_all(&self.pool) .await?; diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index e169c0a3d..e8b41ae43 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -92,7 +92,31 @@ const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; /// for every repo this node creates (bare repos default it off), so /// the gap is repos predating that. Deliberate trade: a stranded row /// an operator can see beats accounting the node cannot substantiate. +#[allow(dead_code)] // single-page seam; startup boots through the multi-pass walk below pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow::Result { + reconcile_prepared_page(state, None, limit) + .await + .map(|(promoted, _cursor)| promoted) +} + +/// One page of the reconcile, resuming after `after`. Returns +/// `(promoted, next_cursor)`, where `next_cursor` is the +/// `(created_at, id)` of the last row EXAMINED — promoted or not — and +/// `None` once a short page says the backlog is exhausted. +/// [`reconcile_prepared_from_disk_all`] walks with it. +/// +/// The cursor is what makes the multi-pass loop actually advance. A +/// pass consumes every row it looked at, including the ones it refused +/// to promote (a SHA that does not match, a row past +/// [`MAX_RECONCILE_AGE`], a landing with no reflog proof). Those rows +/// stay in `prepared` / `uncertain` by design, so a cursor-less next +/// pass would re-read the same page forever and never reach the +/// backlog behind them. +async fn reconcile_prepared_page( + state: AppState, + after: Option<(String, String)>, + limit: i64, +) -> anyhow::Result<(usize, Option<(String, String)>)> { // P1 (reviewer-1/2 round 3): also reconcile `uncertain` rows, // not just `prepared`. A receive-pack error that leaves rows // `uncertain` has the same recovery need as an interrupted @@ -102,11 +126,22 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow // for refs that DID land before the error. let rows = state .db - .list_pending_ref_transitions_prepared_or_uncertain(limit) + .list_pending_ref_transitions_prepared_or_uncertain_after( + after.as_ref().map(|(ts, id)| (ts.as_str(), id.as_str())), + limit, + ) .await?; if rows.is_empty() { - return Ok(0); + return Ok((0, None)); } + // Taken BEFORE any promotion, from the last row of the page as it + // was READ: the walk advances over examined rows, not over promoted + // ones. A short page means there is nothing behind it. + let next_cursor = if (rows.len() as i64) < limit.max(1) { + None + } else { + rows.last().map(|r| (r.created_at.clone(), r.id.clone())) + }; // Group rows by repo so we call `list_refs` once per repo, not // once per row. @@ -246,7 +281,7 @@ pub async fn reconcile_prepared_from_disk(state: AppState, limit: i64) -> anyhow "reconciled prepared/uncertain -> applied via on-disk ref match" ); } - Ok(flipped as usize) + Ok((flipped as usize, next_cursor)) } /// Does the ref's reflog prove that THIS row's transition landed? @@ -309,46 +344,51 @@ pub const REFLOG_CLOCK_SKEW: chrono::Duration = chrono::Duration::seconds(60); /// P2 (reviewer-1/2 round 3): multi-pass reconcile for the prepared/ /// uncertain backlog. Mirrors `drain_pending_ref_transitions_all`: -/// runs `reconcile_prepared_from_disk` in a loop until either a pass -/// examines fewer rows than `per_pass_limit` (backlog exhausted) or -/// `max_passes` passes have completed. If rows remain after the last -/// pass, a residual-backlog warning is logged and those rows wait for -/// the next startup. +/// runs a reconcile pass in a loop until either a pass examines fewer +/// rows than `per_pass_limit` (backlog exhausted) or `max_passes` +/// passes have completed. If rows remain after the last pass, a +/// residual-backlog warning is logged and those rows wait for the next +/// startup. +/// +/// The passes WALK, on the `(created_at, id)` cursor each page returns. +/// The drain can re-issue the same query every pass because it deletes +/// the rows it finishes, so its next page is always new work; the +/// reconcile deletes nothing and leaves every unpromotable row exactly +/// where it was, so re-issuing a cursor-less query re-read page one on +/// every pass. One unprovable row at the head of the ordering — a SHA +/// that never landed, a row past [`MAX_RECONCILE_AGE`], or (since the +/// reflog gate) a landing in a repo that keeps no reflog — was enough +/// to pin the whole loop there and leave the backlog behind it +/// unexamined, which is the finding this loop exists to close. Those +/// rows keep ageing toward `MAX_RECONCILE_AGE` while they wait, so +/// "next restart" can mean "never recovered". pub async fn reconcile_prepared_from_disk_all( state: AppState, per_pass_limit: i64, max_passes: usize, ) -> anyhow::Result { let mut total = 0; + let mut cursor: Option<(String, String)> = None; for _ in 0..max_passes { - let n = reconcile_prepared_from_disk(state.clone(), per_pass_limit).await?; + let (n, next) = + reconcile_prepared_page(state.clone(), cursor.clone(), per_pass_limit).await?; total += n; - // The reconcile function returns the number of rows PROMOTED, - // not the number EXAMINED. We need to know if more rows remain. - // Query the count of prepared/uncertain rows to decide. - let remaining = state - .db - .list_pending_ref_transitions_prepared_or_uncertain(1) - .await? - .len(); - if remaining == 0 { - return Ok(total); + // A short page is the backlog-exhausted signal, keyed on rows + // EXAMINED rather than rows promoted: a pass that could promote + // nothing has still consumed its page and must move on. + match next { + Some(c) => cursor = Some(c), + None => return Ok(total), } } // One more pass to detect residual backlog. - let residual = reconcile_prepared_from_disk(state.clone(), per_pass_limit).await?; + let (residual, next) = reconcile_prepared_page(state.clone(), cursor, per_pass_limit).await?; total += residual; - let remaining = state - .db - .list_pending_ref_transitions_prepared_or_uncertain(1) - .await? - .len(); - if remaining > 0 { + if next.is_some() { tracing::warn!( total, max_passes, per_pass_limit, - remaining, "reconcile backlog exceeds startup budget; residual rows will be picked up on next restart" ); } @@ -1495,6 +1535,133 @@ mod drain_tests { assert_eq!(applied[0].id, row.id); } + // ----- P2 (reviewer round 3): the multi-pass reconcile must WALK ----- + + /// The backlog past the first page is reconciled in the SAME + /// startup, not one page per restart. With a per-pass limit of ONE, + /// a single pass can promote at most one row, so anything above one + /// proves the loop advanced. + /// + /// MUTATION (RED): call the single-page `reconcile_prepared_from_disk` + /// and only the first row is promoted. + #[sqlx::test] + async fn reconcile_all_walks_the_backlog_past_the_first_page(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + + // Three landed refs, each with reflog proof of its own creation. + let ref_names = ["refs/heads/one", "refs/heads/two", "refs/heads/three"]; + for (i, ref_name) in ref_names.iter().enumerate() { + let sha = seed_ref_on_bare(&bare, ref_name); + let mut row = make_row(&repo_id, ref_name, &"0".repeat(40), &sha); + row.request_id = format!("req-backlog-{i}"); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &row.request_id, + &row.repo_id, + &row.ref_name, + &row.old_sha, + &row.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + } + + let promoted = reconcile_prepared_from_disk_all(state.clone(), 1, DRAIN_MAX_PASSES) + .await + .unwrap(); + assert_eq!( + promoted, + ref_names.len(), + "every row is reconciled in ONE startup, not one page per restart" + ); + let still_pending = state + .db + .list_pending_ref_transitions_prepared_or_uncertain(100) + .await + .unwrap(); + assert!( + still_pending.is_empty(), + "no backlog is left stranded past the first page" + ); + } + + /// The walk must step OVER rows it cannot promote. Those rows stay + /// `prepared` by design, so a pass that re-queried from the start + /// would hand itself the same page forever and never reach the + /// provable rows behind them. + /// + /// The blocker here is the class the reflog gate introduces: a ref + /// that really is on disk at the row's `new_sha`, in a repo that + /// keeps no reflog for it — permanently unprovable, so it jams page + /// one on every startup for as long as it exists, not just once. + /// + /// MUTATION (RED): ignore the cursor when selecting the next page + /// and the provable row behind the blocker is never promoted. + #[sqlx::test] + async fn reconcile_all_advances_past_an_unprovable_row(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + + // Row 1 (oldest, so it sorts first): the ref IS on disk at the + // row's new_sha, but its reflog is gone — the SHA matches, the + // age passes, and the landing is still unproven. + let legacy_sha = seed_ref_on_bare(&bare, "refs/heads/legacy"); + std::fs::remove_file(bare.join("logs/refs/heads/legacy")).expect("drop the ref's reflog"); + let mut blocker = make_row(&repo_id, "refs/heads/legacy", &"0".repeat(40), &legacy_sha); + blocker.request_id = "req-blocker".to_string(); + blocker.state = pending_state::PREPARED.to_string(); + blocker.applied_at = None; + blocker.created_at = (chrono::Utc::now() - chrono::Duration::minutes(5)).to_rfc3339(); + blocker.id = crate::db::deterministic_id(&["pending_ref_transition", "req-blocker"]); + state + .db + .insert_pending_ref_transition_for_test(&blocker) + .await + .unwrap(); + + // Row 2 (newer): a provable landing sitting behind it. + let sha = seed_ref_on_bare(&bare, "refs/heads/landed"); + let mut good = make_row(&repo_id, "refs/heads/landed", &"0".repeat(40), &sha); + good.request_id = "req-good".to_string(); + good.state = pending_state::PREPARED.to_string(); + good.applied_at = None; + good.id = crate::db::deterministic_id(&["pending_ref_transition", "req-good"]); + state + .db + .insert_pending_ref_transition_for_test(&good) + .await + .unwrap(); + + let promoted = reconcile_prepared_from_disk_all(state.clone(), 1, DRAIN_MAX_PASSES) + .await + .unwrap(); + assert_eq!( + promoted, 1, + "the provable row behind a permanently unprovable one is still reached" + ); + let still_pending = state + .db + .list_pending_ref_transitions_prepared_or_uncertain(100) + .await + .unwrap(); + assert_eq!(still_pending.len(), 1, "the unprovable row is left alone"); + assert_eq!(still_pending[0].id, blocker.id); + } + // ----- P2-A drain resilience tests ----- // // These tests cover the "drain must not abort on first failure" From 1247f4ed1beaa4b3d8da84f47bae9435243d26f5 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Mon, 31 Aug 2026 19:24:04 +0800 Subject: [PATCH 10/22] fix(node): bound the reflog read to a recent tail (#26 split 1/4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ref_reflog_entries` slurped the whole reflog. A reflog grows one line (~150 bytes) per ref update and how often a ref is updated is PUSHER-controlled, so that was an attacker-sized allocation, taken once per stranded row, at startup — the moment the node can least absorb it. Every other attacker-influenced read in this crate is bounded (read_body_capped, MAX_SIGNATURE_ENTRIES, the pin/walk budgets); this one should be too. The bound costs nothing in proving power, which is why it is a ceiling rather than a tradeoff: the landing gate only ever accepts an entry stamped at or after `created_at - REFLOG_CLOCK_SKEW`, and reflogs are append-ordered oldest-first, so every entry that could possibly qualify is at the tail. The read now seeks to the last REFLOG_TAIL_BYTES (256KB, ~2000 recent updates to one ref) and `take`s that many bytes, so the allocation stays bounded even if the file grows between the stat and the read. A window into the middle of a file starts mid-record, so the bytes before its first newline are discarded: without that, a boundary landing inside a record's old SHA leaves a suffix that still tokenizes and yields a truncated SHA as a bogus `old -> new` pair — and landing proof is an exact pair match, so a bogus pair is a fabricated proof. An entry older than the window reads as absent, which is the safe direction: the caller treats "no matching entry" as unproven and leaves the row where it is. `Ok(None)` for a missing file, and the absence-of-evidence contract, are unchanged. Tests: ref_reflog_entries_reads_whole_records_from_the_recent_tail (lays the file out so the boundary lands 20 bytes into a record's old SHA — the alignment that would otherwise parse), an_entry_older_than_the_reflog_tail_reads_as_unproven. --- crates/gitlawb-node/src/git/store.rs | 216 ++++++++++++++++++++++++++- 1 file changed, 212 insertions(+), 4 deletions(-) diff --git a/crates/gitlawb-node/src/git/store.rs b/crates/gitlawb-node/src/git/store.rs index 6dfa7765e..6e59bf7e9 100644 --- a/crates/gitlawb-node/src/git/store.rs +++ b/crates/gitlawb-node/src/git/store.rs @@ -86,13 +86,36 @@ pub struct ReflogEntry { pub at: i64, } -/// Read the reflog of one ref in a bare repository, newest entry LAST. +/// How many bytes of the END of a reflog file are read. A reflog line is roughly +/// 150 bytes, so this window holds on the order of two thousand of the most recent +/// updates to a single ref. +/// +/// The bound costs nothing in proving power, because of what the caller asks. The +/// gate only ever accepts an entry stamped at or after +/// `created_at - REFLOG_CLOCK_SKEW`, i.e. within about a minute of a row that is +/// itself inside `MAX_RECONCILE_AGE`; reflogs are append-ordered oldest-first, so +/// every entry that could possibly qualify is at the tail. What the window buys is +/// a ceiling: a reflog grows one line per ref update and how often a ref is updated +/// is PUSHER-controlled, so an unbounded read is an attacker-sized allocation taken +/// once per stranded row, at startup, which is the moment the node can least absorb +/// it. +const REFLOG_TAIL_BYTES: u64 = 256 * 1024; + +/// Read the tail of one ref's reflog in a bare repository, newest entry LAST. /// /// Reads `logs/` directly rather than shelling out to `git reflog`: the /// file format is stable and documented, the reconcile may call this once per /// stranded row at startup, and a plain file read cannot be defeated by the ref's /// reflog having been expired out of the `git reflog show` default window. /// +/// Only the last [`REFLOG_TAIL_BYTES`] are read, and only whole lines within that +/// window: when the file is longer, the bytes before the first newline inside the +/// window are a PARTIAL record and are discarded, so a record sliced mid-SHA can +/// never be tokenized into a bogus `old -> new` pair. An entry older than the +/// window reads as absent, which is the safe direction — the caller treats +/// "no matching entry" as unproven and leaves the row where it is, rather than +/// promoting it. +/// /// Returns `Ok(None)` when the repo keeps no reflog for that ref — either because /// `core.logAllRefUpdates` was off when the ref moved (repos created before /// [`init_bare`] started enabling it) or because the ref was deleted (git removes a @@ -114,11 +137,41 @@ pub fn ref_reflog_entries(repo_path: &Path, ref_name: &str) -> Result s, + let mut file = match std::fs::File::open(&path) { + Ok(f) => f, Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None), - Err(e) => return Err(e).context("failed to read reflog"), + Err(e) => return Err(e).context("failed to open reflog"), + }; + let len = file.metadata().context("failed to stat reflog")?.len(); + let truncated = len > REFLOG_TAIL_BYTES; + if truncated { + use std::io::Seek; + file.seek(std::io::SeekFrom::Start(len - REFLOG_TAIL_BYTES)) + .context("failed to seek to the reflog tail")?; + } + let mut buf = Vec::with_capacity(REFLOG_TAIL_BYTES.min(len) as usize); + { + use std::io::Read; + // Cap the read itself, not just the seek: the file can grow between the + // stat and the read, and the allocation must stay bounded either way. + file.take(REFLOG_TAIL_BYTES) + .read_to_end(&mut buf) + .context("failed to read reflog")?; + } + // A window into the middle of the file almost certainly starts mid-record. + // Drop everything before the first newline so only whole lines are parsed; + // a reflog is one record per line, so the first full record starts there. + let window: &[u8] = if truncated { + match buf.iter().position(|b| *b == b'\n') { + Some(i) => &buf[i + 1..], + // No newline in the whole window: every byte is part of one partial + // record, so there is nothing whole to parse. + None => &[], + } + } else { + &buf }; + let raw = String::from_utf8_lossy(window); let mut out = Vec::new(); for line in raw.lines() { // The message after the TAB can contain anything, including spaces and @@ -1050,6 +1103,161 @@ mod tests { ); } + /// One reflog record. Fixed-width `i` and timestamp keep every filler line the + /// same length, so the test below can place the window boundary on an exact + /// byte. + fn reflog_line(old: &str, new: &str, at: i64, msg: &str) -> String { + format!("{old} {new} tester {at} +0000\tpush: {msg}\n") + } + + fn filler_line(i: usize, pad: usize) -> String { + reflog_line( + &format!("{:040x}", i), + &format!("{:040x}", i + 1), + 1_600_000_000, + &format!("filler {i:06}{}", "x".repeat(pad)), + ) + } + + /// The bound is a ceiling on the READ, not on the proof: a ref hammered with + /// far more updates than the tail window can hold still yields its recent + /// entries, which is the only region the landing gate ever accepts from. And + /// the record the window CUTS THROUGH must be discarded whole, never + /// half-parsed. + /// + /// The file is laid out so the window boundary lands 20 bytes into a record's + /// old SHA — the hazardous alignment, where the surviving suffix still has + /// enough fields to tokenize and would yield a 20-character "old SHA" as a + /// bogus `old -> new` pair. Landing proof is an exact pair match, so a bogus + /// pair is a fabricated proof. + /// + /// MUTATION (RED): read the first `REFLOG_TAIL_BYTES` instead of the last and + /// the recent entry disappears; keep the leading partial line instead of + /// discarding it and the 20-character SHA appears. + #[test] + fn ref_reflog_entries_reads_whole_records_from_the_recent_tail() { + let td = tempfile::TempDir::new().unwrap(); + let bare = td.path().join("repo.git"); + super::init_bare(&bare).unwrap(); + + let window = super::REFLOG_TAIL_BYTES as usize; + // The record the boundary will slice, and the recent one a reconcile + // would actually be looking for. + let hazard = reflog_line(&"a".repeat(40), &"b".repeat(40), 1_700_000_000, "hazard"); + let recent = reflog_line(&"c".repeat(40), &"d".repeat(40), 1_700_009_999, "recent"); + + // Everything from the hazard record to EOF must measure exactly + // `window + 20`, so the read starts 20 bytes into the hazard's old SHA. + let tail_target = window + 20; + let needed = tail_target - hazard.len() - recent.len(); + let base = filler_line(0, 0).len(); + assert!(needed > 2 * base, "layout math needs room for filler"); + let full = needed / base - 1; + let pad = needed - (full + 1) * base; + let mut tail = hazard.clone(); + for i in 0..full { + tail.push_str(&filler_line(i, 0)); + } + tail.push_str(&filler_line(full, pad)); + tail.push_str(&recent); + assert_eq!( + tail.len(), + tail_target, + "the tail must measure exactly window + 20 for the boundary to land \ + inside the hazard record's old SHA" + ); + + // Anything before the hazard record is outside the window entirely. + let mut body = String::new(); + for i in 0..8 { + body.push_str(&filler_line(1_000 + i, 0)); + } + let lead = body.len(); + body.push_str(&tail); + + let log_path = bare.join("logs/refs/heads/busy"); + std::fs::create_dir_all(log_path.parent().unwrap()).unwrap(); + std::fs::write(&log_path, &body).unwrap(); + assert_eq!( + body.len() - window, + lead + 20, + "the read must begin 20 bytes into the hazard record" + ); + + let entries = super::ref_reflog_entries(&bare, "refs/heads/busy") + .unwrap() + .expect("the reflog exists"); + + // The tail is what got read. + let last = entries.last().expect("the window holds whole records"); + assert_eq!( + (last.old_sha.as_str(), last.new_sha.as_str(), last.at), + ( + "c".repeat(40).as_str(), + "d".repeat(40).as_str(), + 1_700_009_999 + ), + "the newest entry — the only region the landing gate accepts — survives \ + the bound intact" + ); + assert!( + entries.len() < body.lines().count(), + "the read is bounded: not every record in the file is parsed" + ); + + // And the sliced record was dropped rather than half-parsed. Both halves + // matter: no truncated SHA may appear, and the hazard's pair must not be + // reconstructed from a partial line either. + for e in &entries { + assert_eq!( + e.old_sha.len(), + 40, + "a partial record was parsed into a bogus old SHA: {e:?}" + ); + assert_eq!(e.new_sha.len(), 40, "a partial record was parsed: {e:?}"); + } + assert!( + !entries.iter().any(|e| e.new_sha == "b".repeat(40)), + "the record the window cut through must not contribute a pair at all" + ); + } + + /// The safe direction of the same bound. An entry that sits only in the + /// discarded older region reads as absent, and absent means UNPROVEN — the + /// reconcile leaves such a row where it is instead of promoting it, which is + /// the failure mode a bounded read is allowed to have. + #[test] + fn an_entry_older_than_the_reflog_tail_reads_as_unproven() { + let td = tempfile::TempDir::new().unwrap(); + let bare = td.path().join("repo.git"); + super::init_bare(&bare).unwrap(); + + // The sought pair is written FIRST, then buried under enough later + // updates to push it clear out of the window. + let buried = ("c".repeat(40), "d".repeat(40), 1_650_000_000i64); + let log_path = bare.join("logs/refs/heads/buried"); + std::fs::create_dir_all(log_path.parent().unwrap()).unwrap(); + let mut body = reflog_line(&buried.0, &buried.1, buried.2, "the buried landing"); + let mut i = 0; + while body.len() <= super::REFLOG_TAIL_BYTES as usize * 2 { + body.push_str(&filler_line(i, 0)); + i += 1; + } + std::fs::write(&log_path, &body).unwrap(); + + let entries = super::ref_reflog_entries(&bare, "refs/heads/buried") + .unwrap() + .expect("the reflog exists"); + assert!( + !entries + .iter() + .any(|e| e.old_sha == buried.0 && e.new_sha == buried.1), + "an entry outside the tail window is simply not seen — the caller then \ + treats the landing as unproven and leaves the row alone, never the \ + other way round" + ); + } + /// A ref with no reflog reads as `None` — "no evidence", which callers must /// treat as unproven rather than as proof of nothing having happened. #[test] From c2ad0e770e4d01a559c9f9ad504539b2a0efddcf Mon Sep 17 00:00:00 2001 From: Kevin Codex Date: Mon, 31 Aug 2026 22:04:14 +0800 Subject: [PATCH 11/22] fix(node): read the report through both side-band frames MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The report-status is framed twice whenever the client negotiates side-band-64k, which `git push` over smart HTTP does: the outer side-band envelope carries a stream that is itself pkt-line encoded. One strip pass left the first line as `000eunpack ok`, so the unpack check failed and parse_report_status returned None — read by the caller as "no report", which keeps every declared ref. The per-ref landing gate was therefore inert for exactly the pushes it exists to filter. The regression test is a byte-for-byte capture from git 2.50.1 rejecting one ref of a two-ref push, because a hand-written single-framed fixture parses under the old code and hides the case. --- crates/gitlawb-node/src/git/smart_http.rs | 58 +++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/crates/gitlawb-node/src/git/smart_http.rs b/crates/gitlawb-node/src/git/smart_http.rs index 17a9224f0..5a528efb9 100644 --- a/crates/gitlawb-node/src/git/smart_http.rs +++ b/crates/gitlawb-node/src/git/smart_http.rs @@ -301,6 +301,22 @@ pub fn parse_report_status(output: &[u8]) -> Option<(bool, Vec<(String, bool)>)> // data starts after the first `0000` flush packet or after we // strip sideband bytes. let stripped = strip_sideband(text)?; + // The report is framed TWICE when the client negotiated side-band-64k, + // which `git push` over smart HTTP does — so this is the common case, not + // an exotic one. The outer frame is the side-band envelope; band 1 carries + // the report-status stream, which is ITSELF pkt-line encoded. Real bytes + // from git 2.50.1 rejecting one ref of a two-ref push: + // + // 0057\x01000eunpack ok\n0028ng refs/heads/main non-fast-forward\n... + // + // After one pass the first line reads `000eunpack ok`, which fails the + // `unpack ok` check below and makes this return None — i.e. "no report", + // which the caller treats as inconclusive and keeps every declared ref. + // The per-ref gate would then be inert for exactly the pushes it exists to + // filter. A second pass removes the inner pkt-line framing; it is a no-op + // on the single-framed shape, because plain report text does not begin + // with four hex digits. + let stripped = strip_sideband(&stripped).unwrap_or(stripped); let lines: Vec<&str> = stripped.lines().collect(); if lines.is_empty() { return None; @@ -1092,6 +1108,48 @@ mod tests { use std::process::Command; use tempfile::TempDir; + /// Byte-for-byte capture from `git receive-pack` 2.50.1 rejecting one ref of + /// a two-ref push, with side-band-64k negotiated — what `git push` over + /// smart HTTP actually sends. The report is framed twice: the outer + /// side-band envelope, then the report-status stream's own pkt-lines. + /// + /// This fixture is the point of the test. A hand-written single-framed + /// string parses fine with only one strip pass, so a fixture invented to + /// match the parser hides the exact case the parser is for. + const REAL_SIDEBAND_REPORT: &[u8] = + b"0057\x01000eunpack ok\n0028ng refs/heads/main non-fast-forward\n0018ok refs/heads/third\n0000"; + + #[test] + fn a_real_double_framed_sideband_report_is_read_not_treated_as_absent() { + let (unpack_ok, refs) = + parse_report_status(REAL_SIDEBAND_REPORT).expect("a real git report must parse"); + assert!(unpack_ok, "unpack line must be read through both frames"); + assert_eq!( + refs, + vec![ + ("refs/heads/main".to_string(), false), + ("refs/heads/third".to_string(), true), + ], + "the rejected ref must be distinguished from the accepted one" + ); + } + + /// The single-framed shape must keep working: the second pass has to be a + /// no-op there, not a corruption. Plain report text does not begin with + /// four hex digits, which is what makes that safe. + #[test] + fn a_single_framed_report_still_parses_after_the_second_pass() { + // Framed programmatically rather than by a hand-counted hex prefix: a + // wrong length makes the parser return None, which would look exactly + // like the regression this pair of tests exists to catch. + let payload = "\x01unpack ok\nok refs/heads/x\n"; + let single = format!("{:04x}{payload}0000", payload.len() + 4); + let (unpack_ok, refs) = + parse_report_status(single.as_bytes()).expect("single-framed must parse"); + assert!(unpack_ok); + assert_eq!(refs, vec![("refs/heads/x".to_string(), true)]); + } + /// List OIDs in a pack by writing it to a temp dir and running verify-pack. pub(super) fn pack_object_ids(pack: &[u8]) -> std::collections::HashSet { let dir = TempDir::new().unwrap(); From f49ae0fd8be10d58839368eec62fbde0ab09bf6e Mon Sep 17 00:00:00 2001 From: Gravirei Date: Tue, 1 Sep 2026 03:20:44 +0600 Subject: [PATCH 12/22] fix(node): address round-4 reviewer findings on #26 split 1/4 --- crates/gitlawb-node/src/api/repos.rs | 489 ++++++++++++++-------- crates/gitlawb-node/src/cert.rs | 58 ++- crates/gitlawb-node/src/db/mod.rs | 320 +++++++++++++- crates/gitlawb-node/src/durable_outbox.rs | 220 +++++++++- crates/gitlawb-node/src/git/store.rs | 379 ++++++++++++++++- 5 files changed, 1262 insertions(+), 204 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index e5ad20cd0..4880bda9c 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2367,125 +2367,182 @@ pub async fn git_receive_pack( } }; - // Parse the report-status to determine per-ref ok/ng results. - // If the output cannot be parsed (e.g. client didn't request - // report-status), treat all refs as uncertain. + // P1 (reviewer-1/2 round 4): complete the per-ref outcome + // model. The previous `all_refs_ok` ignored `unpack_ok` and + // treated an absent report as success, then ran the durable + // effects loop unconditionally. A mixed push where one ref is + // rejected still issued a signed certificate, an anchor job, a + // push event, a trust-score bump, and a webhook for the rejected + // ref. Git can report `unpack ok / ng refs/heads/main` on a + // zero exit (a non-fast-forward, a hook denial), and that path + // is exactly the one the previous logic missed. + // + // Build the set of ref names that the report-status proves + // landed. Every ref update the handler intended to land gets a + // fate: + // - in `ok_set` → row goes to `applied`, effects fire + // - in `report` but ng → row goes to `cancelled`, no effects + // - not in `report` at all → row goes to `uncertain` for reconcile + // + // If `unpack_ok == false`, no ref could have landed — all rows + // become `cancelled` and no effects fire for any ref. + // + // If the report is unparseable (client did not request + // report-status, or framing was malformed), every row becomes + // `uncertain` so the on-disk reflog proof can sort out which + // ones actually landed at the next startup. let report = smart_http::parse_report_status(&receive_raw); - let all_refs_ok = exit_ok - && report - .as_ref() - .is_none_or(|(_, results)| results.iter().all(|(_, ok)| *ok)); - if all_refs_ok { - // All refs landed. Mark outbox rows as applied so the drain - // can pick them up if the effects write below fails partway. + let (unpack_ok, all_in_report_ok, ok_set, request_failed) = match &report { + Some((unpack_ok, ref_results)) => { + let ok_set: std::collections::HashSet<&str> = ref_results + .iter() + .filter(|(_, ok)| *ok) + .map(|(name, _)| name.as_str()) + .collect(); + let all_in_report_ok = ref_results.iter().all(|(_, ok)| *ok); + (*unpack_ok, all_in_report_ok, ok_set, !exit_ok) + } + None => { + // No report available — every ref's fate is uncertain. + let ok_set: std::collections::HashSet<&str> = std::collections::HashSet::new(); + (false, true, ok_set, !exit_ok) + } + }; + + // Per-ref state flip. Rows go to `applied`, `cancelled`, or + // `uncertain` based on the report. The previous bulk helper + // `mark_pending_ref_transitions_applied(request_id)` marked + // EVERY row `applied` regardless of which refs git actually + // accepted — that is the bug that issued certs for rejected + // refs. + let pending_ref_names: Vec<&str> = ref_updates.iter().map(|u| u.ref_name.as_str()).collect(); + + if !unpack_ok && report.is_some() { + // Unpack failed explicitly — every row is proven not to have + // landed. Mark all prepared rows for this request as + // `cancelled` (the only state from which reconcile and drain + // both refuse to promote). The drain will not pick these up; + // the next startup's reconcile will not promote them. if let Err(e) = state .db - .mark_pending_ref_transitions_applied(&request_id) + .mark_pending_ref_transitions_cancelled_for_names( + &request_id, + &pending_ref_names, + ) .await { - tracing::error!( + tracing::warn!( err = %e, request_id = %request_id, repo = %name, - "failed to mark pending ref transitions applied; recovery will re-derive" + "failed to mark pending ref transitions cancelled (unpack fail)" ); } - } else { - // P1 (reviewer-1/2 round 3): receive-pack returned non-zero or - // the report-status shows per-ref rejections. Mark all - // `prepared` rows as `uncertain` — the reconcile step at - // startup will check each row against on-disk refs and promote - // only those that actually landed. This preserves recovery for - // refs that DID land (a timeout or non-zero exit is not proof - // that no ref was committed), while the `cancelled` state - // (which reconcile and drain both skip) is reserved for refs - // that can be PROVEN not to have landed. - if let Some((unpack_ok, ref_results)) = &report { - if !unpack_ok { - // Unpack failed — no refs could have landed. Safe to - // cancel all rows immediately. - tracing::warn!( + } else if report.is_some() { + // Report parsed. Split rows by per-ref ok/ng, with anything + // NOT in the report (defensive: report is a subset of the + // pushed refs in some edge cases) falling to `uncertain`. + let mut ok_names: Vec<&str> = Vec::new(); + let mut ng_names: Vec<&str> = Vec::new(); + let mut unmentioned: Vec<&str> = Vec::new(); + let reported: std::collections::HashSet<&str> = report + .as_ref() + .map(|(_, rs)| rs.iter().map(|(n, _)| n.as_str()).collect()) + .unwrap_or_default(); + for name in &pending_ref_names { + if !reported.contains(name) { + unmentioned.push(*name); + } else if ok_set.contains(name) { + ok_names.push(*name); + } else { + ng_names.push(*name); + } + } + if !ng_names.is_empty() { + tracing::warn!( + request_id = %request_id, + repo = %name, + rejected_refs = ?ng_names, + "git report-status: some refs rejected; durable effects will skip them" + ); + } + if !ok_names.is_empty() { + if let Err(e) = state + .db + .mark_pending_ref_transitions_applied_for_names( + &request_id, + &ok_names, + ) + .await + { + tracing::error!( + err = %e, request_id = %request_id, repo = %name, - "git report-status: unpack failed; all refs rejected" + "failed to mark pending ref transitions applied; recovery will re-derive" ); - if let Err(e) = state - .db - .mark_pending_ref_transitions_cancelled(&request_id) - .await - { - tracing::warn!( - err = %e, - request_id = %request_id, - repo = %name, - "failed to mark pending ref transitions cancelled" - ); - } - } else { - // Unpack succeeded but some refs were rejected. The - // rejected refs are proven not to have landed; mark - // them cancelled. The refs not in the report are - // uncertain — mark them uncertain for reconcile. - let rejected: Vec<&str> = ref_results - .iter() - .filter(|(_, ok)| !ok) - .map(|(name, _)| name.as_str()) - .collect(); + } + } + if !ng_names.is_empty() { + if let Err(e) = state + .db + .mark_pending_ref_transitions_cancelled_for_names( + &request_id, + &ng_names, + ) + .await + { tracing::warn!( + err = %e, request_id = %request_id, repo = %name, - rejected_refs = ?rejected, - "git report-status: some refs rejected" + "failed to mark pending ref transitions cancelled (per-ref ng)" ); - // For now, mark all as uncertain. The drain is - // idempotent (ON CONFLICT DO NOTHING), so a ref that - // didn't land will produce no artifacts. A more precise - // approach would split rows by per-ref status, but the - // age-bounded reconcile + idempotent drain already - // provides the correct recovery semantics. - if let Err(e) = state - .db - .mark_pending_ref_transitions_uncertain(&request_id) - .await - { - tracing::warn!( - err = %e, - request_id = %request_id, - repo = %name, - "failed to mark pending ref transitions uncertain" - ); - } } - } else { - // Cannot parse report-status. Mark all as uncertain for - // reconcile to sort out at startup. - tracing::warn!( - request_id = %request_id, - repo = %name, - "could not parse git report-status; marking all refs uncertain" - ); + } + if !unmentioned.is_empty() { if let Err(e) = state .db - .mark_pending_ref_transitions_uncertain(&request_id) + .mark_pending_ref_transitions_uncertain_for_names( + &request_id, + &unmentioned, + ) .await { tracing::warn!( err = %e, request_id = %request_id, repo = %name, - "failed to mark pending ref transitions uncertain" + "failed to mark pending ref transitions uncertain (unmentioned in report)" ); } } + } else { + // No report at all — every ref's fate is uncertain. The + // next startup reconcile will use the reflog proof to + // promote only those whose transition actually landed. + if let Err(e) = state + .db + .mark_pending_ref_transitions_uncertain(&request_id) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions uncertain (no report)" + ); + } } // On non-zero exit, return an error to the caller. The outbox - // rows have already been handled above (marked uncertain/cancelled - // based on report-status). This preserves backward compatibility - // with callers that expect an error on non-zero git exit. - if !exit_ok { - // Release the guard before returning the error. + // rows have already been handled above (per-ref fates applied). + // The client-visible body does NOT include wire-supplied ref + // names — ref names can carry control bytes and the previous + // `format!("refs rejected: {rejected:?}")` embedded them in a + // 500 response. Server-side the names are logged above. + if request_failed { let reclaimed = guard .lock() .expect("repo write-lock mutex poisoned") @@ -2494,23 +2551,14 @@ pub async fn git_receive_pack( reclaimed.release(false).await; drop(lease); - let stderr_msg = if let Some((unpack_ok, ref_results)) = &report { - if !*unpack_ok { - "unpack failed".to_string() - } else { - let rejected: Vec<&str> = ref_results - .iter() - .filter(|(_, ok)| !ok) - .map(|(name, _)| name.as_str()) - .collect(); - format!("refs rejected: {rejected:?}") - } + let body_msg = if !unpack_ok { + "git-receive-pack failed: unpack failed" + } else if !all_in_report_ok { + "git-receive-pack failed: refs rejected" } else { - "git-receive-pack failed".to_string() + "git-receive-pack failed" }; - return Err(AppError::Git(format!( - "git-receive-pack failed: {stderr_msg}" - ))); + return Err(AppError::Git(body_msg.to_string())); } // #174 F2/U5: the post-receive replication tail runs in an independently owned @@ -2555,15 +2603,38 @@ pub async fn git_receive_pack( // The alternative, detaching `release` and the tail together to keep the ordering, // would return 200 to the pusher before the durable copy lands, which is a larger // change to the client contract than the window it closes. - let push_succeeded = all_refs_ok; + // P1 (reviewer-1/2 round 4): the durable effects only fire for + // refs the report-status proves landed. The previous code ran + // them unconditionally once past the `!exit_ok` return, so a + // `unpack ok / ng refs/heads/main` zero-exit push still signed + // a cert and queued an anchor for the rejected ref. + // + // `push_succeeded` is the request-level signal for the + // replication tail and the lock release. `any_ref_ok` is the + // request-scoped effects gate (push event, trust score, + // metrics): at least one ref must have landed for those to be + // meaningful. + let any_ref_ok = !ok_set.is_empty(); + let push_succeeded = exit_ok && any_ref_ok; + if push_succeeded { - tokio::spawn(post_receive_replication_tail( - state.clone(), - record.clone(), - ref_updates.clone(), - disk_path.clone(), - auth.0.to_string(), - )); + // Spawn the replication tail only for refs that landed. + // Filtering at spawn-time keeps the tail's input accurate + // even for a mixed push. + let landed_refs: Vec = ref_updates + .iter() + .filter(|u| ok_set.contains(u.ref_name.as_str())) + .cloned() + .collect(); + if !landed_refs.is_empty() { + tokio::spawn(post_receive_replication_tail( + state.clone(), + record.clone(), + landed_refs, + disk_path.clone(), + auth.0.to_string(), + )); + } } // Always release the advisory lock — even on error — to prevent stale locks @@ -2586,15 +2657,28 @@ pub async fn git_receive_pack( // the disconnect path this line is never reached: clone (a) rides the reaper (F3). drop(lease); - // The error path for receive_pack is handled above (in the - // `match smart_http::receive_pack_raw(...)` block). If we reach - // here, all refs landed successfully (all_refs_ok == true). - - // Update the repo's updated_at timestamp after a successful push + // If no ref landed, return 200 with the receive-pack body but do + // NOT run any durable effects (no push event, no trust score, + // no metrics, no webhooks, no certs, no anchor jobs). The + // outbox rows have already been flipped to `cancelled` / + // `uncertain` for every ref above; the next startup reconcile + // will not promote them. + if !any_ref_ok { + return axum::response::Response::builder() + .status(axum::http::StatusCode::OK) + .header("Content-Type", "application/x-git-receive-pack-result") + .header("Cache-Control", "no-cache") + .body(axum::body::Body::from(receive_raw)) + .map_err(|e| AppError::Internal(anyhow::anyhow!("failed to build response: {e}"))); + } + + // Request-scoped effects. These run when at least one ref + // landed; the per-ref certs and anchor jobs below gate further + // on `ok_set` membership. A mixed push where one ref was + // rejected still gets the push event and trust score (one or + // more refs DID land) but the rejected ref has no cert, no + // anchor, and no webhook. let _ = state.db.touch_repo(&record.id).await; - - // Record the successful push for metrics. The body has already been - // consumed by smart_http::receive_pack so we observe size up front. crate::metrics::record_push(&record.id); crate::metrics::observe_pack_size(body_len as f64); @@ -2608,26 +2692,38 @@ pub async fn git_receive_pack( // produces the same primary keys and the idempotent inserts collapse. let did = auth.0.as_str(); { - // Use the first new commit hash we parsed, fall back to timestamp - let commit_hash = ref_updates - .first() - .map(|u| u.new_sha.clone()) - .unwrap_or_else(|| Utc::now().timestamp().to_string()); - - // The push event is keyed on the FIRST ref's name so a - // multi-ref push collapses to one push event row, not N. The - // deterministic id is the same one the recovery drain - // derives, because the drain reads `first_ref_name` from the - // outbox row (persisted above on every row of this request) and - // uses it in the same `push_event_id_for` call. The outer - // `first_ref_name` local was hoisted above the - // `insert_pending_ref_transitions` call so this id matches - // the persisted value exactly. + // P1 (reviewer-1 round 4): the request-scoped push event + // uses the FIRST OK ref's `new_sha` as `commit_hash`, NOT + // `ref_updates.first()`. The previous code used the first + // requested ref regardless of whether it landed, so a + // mixed push with a rejected first ref recorded a + // `commit_hash` for a SHA that does not exist. Using the + // first OK ref's new_sha keeps the push event's + // `commit_hash` truthful; if every ref was rejected we + // already returned above (`any_ref_ok` is false). + let first_ok_update = ref_updates + .iter() + .find(|u| ok_set.contains(u.ref_name.as_str())); + let commit_hash = match first_ok_update { + Some(u) => u.new_sha.clone(), + None => Utc::now().timestamp().to_string(), + }; + // `first_ref_name` for the deterministic push event id + // stays the request's first ref name (the same key the + // drain uses); the commit_hash is what changes. let push_event_id = crate::db::push_event_id_for(&request_id, &first_ref_name); - let _ = state + if let Err(e) = state .db .record_push_with_id(&push_event_id, did, &record.id, &commit_hash, 0) - .await; + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to record push event; recovery will re-derive (idempotent)" + ); + } if let Ok(push_count) = state.db.get_push_count(did).await { // 0.05 base (from registration) + 0.05 per push, capped at 1.0 // 1 push → 0.10, 5 pushes → 0.30, 19 pushes → 1.0 @@ -2635,28 +2731,22 @@ pub async fn git_receive_pack( let _ = state.db.update_trust_score(did, new_score).await; } - // Issue a signed certificate for every ref this push advanced, each - // carrying that ref's real old→new transition. A multi-ref push must - // not collapse to a single cert covering only the first ref. - // - // P1-B: this routes through `issue_ref_certificate` (the - // upsert, `ON CONFLICT (repo_id, ref_name) DO UPDATE`) so a - // re-push to the same ref updates the row's - // `old_sha` / `new_sha` / `pusher_did` / `issued_at` / - // `signature` to the new transition while preserving the - // original `id` (the `insert_ref_certificate` upsert's - // `EXCLUDED.issued_at > ref_certificates.issued_at` guard). - // The previous `issue_ref_certificate_idempotent` call - // (DO NOTHING) left the first cert's fields frozen on every - // later push. The deterministic `cert_id` makes a recovery - // re-pass safe: the recovery's `insert_ref_certificate_idempotent` - // (DO NOTHING) is a no-op when the live handler has already - // written a row, and the live handler's upsert preserves the - // original `id` so the deterministic id survives across the - // push / recovery / re-push cycle. + // Per-ref durable effects. Each ref's cert + anchor writes + // are gated on `ok_set` membership — the rejected ref gets + // neither. Track per-ref success so the cleanup at the end + // only deletes outbox rows whose required writes all + // succeeded; a transient cert failure leaves the row in + // `applied` for the startup drain to recover. + let mut ok_ref_ids: Vec = Vec::new(); for update in &ref_updates { + // Skip refs the report-status rejected. Their rows are + // already `cancelled` from the per-ref state flip + // above. + if !ok_set.contains(update.ref_name.as_str()) { + continue; + } let cert_id = crate::db::ref_cert_id_for(&request_id, &update.ref_name); - match cert::issue_ref_certificate( + let cert_result = cert::issue_ref_certificate( &state, &record.id, &update.ref_name, @@ -2665,20 +2755,23 @@ pub async fn git_receive_pack( did, &cert_id, ) - .await - { + .await; + let cert_ok = match &cert_result { Ok(c) => { - tracing::info!(cert_id = %c.id, repo = %record.name, ref_name = %update.ref_name, pusher = %did, "issued ref certificate") + tracing::info!(cert_id = %c.id, repo = %record.name, ref_name = %update.ref_name, pusher = %did, "issued ref certificate"); + true } Err(e) => { - tracing::warn!(err = %e, ref_name = %update.ref_name, "failed to issue ref certificate") + tracing::warn!( + err = %e, + request_id = %request_id, + ref_name = %update.ref_name, + "failed to issue ref certificate; outbox row will be left for the drain to retry" + ); + false } - } + }; - // Anchor handoff: insert an anchor_jobs row keyed on the - // per-transition tuple. PR 2 reads this row and uploads - // to the bundler. The deterministic id makes a recovery - // re-pass a no-op. let anchor_id = crate::db::anchor_job_id_for( &record.id, &update.ref_name, @@ -2695,14 +2788,54 @@ pub async fn git_receive_pack( created_at: Utc::now().to_rfc3339(), claimed_at: None, }; - if let Err(e) = state.db.insert_anchor_job_idempotent(&job).await { - tracing::warn!(err = %e, ref_name = %update.ref_name, "failed to enqueue anchor job") + let anchor_ok = match state.db.insert_anchor_job_idempotent(&job).await { + Ok(_) => true, + Err(e) => { + tracing::warn!( + err = %e, + request_id = %request_id, + ref_name = %update.ref_name, + "failed to enqueue anchor job; outbox row will be left for the drain to retry" + ); + false + } + }; + + if cert_ok && anchor_ok { + // The row id is the deterministic id; look it up + // so cleanup can target just this ref's row. + if let Ok(Some(row_id)) = state + .db + .lookup_pending_ref_transition_id( + &request_id, + &update.ref_name, + ) + .await + { + ok_ref_ids.push(row_id); + } + } + } + // Delete the per-ref rows whose required writes succeeded. + // A failed-write row stays `applied` and the next startup + // drain re-derives it (the artifacts are idempotent). + for row_id in &ok_ref_ids { + if let Err(e) = state.db.delete_pending_ref_transition(row_id).await { + tracing::warn!( + err = %e, + request_id = %request_id, + row_id = %row_id, + "failed to delete outbox row after effects landed; drain will re-derive (idempotent)" + ); } } } - // Fire push webhooks — one per ref update - if !ref_updates.is_empty() { + // Fire push webhooks — one per LANDED ref update only. The + // rejected ref is not announced because it did not change + // state. Webhook delivery is best-effort and never blocks + // outbox cleanup. + if !ok_set.is_empty() { let base_url = state .config .public_url @@ -2713,6 +2846,9 @@ pub async fn git_receive_pack( let clone_url = format!("{}/{}/{}.git", base_url, owner_short, record.name); for update in &ref_updates { + if !ok_set.contains(update.ref_name.as_str()) { + continue; + } let payload = serde_json::json!({ "ref": update.ref_name, "before": update.old_sha, @@ -2739,25 +2875,6 @@ pub async fn git_receive_pack( } } - // P1 (reviewer-1/2 round 3): delete outbox rows after all durable - // effects have been written. This prevents the rows from being - // replayed on the next startup. The drain already deletes rows - // after derive_one, but by then the effects have been written - // twice (once on the live path, once by the drain). Deleting here - // keeps the outbox clean and avoids redundant work. - if let Err(e) = state - .db - .delete_pending_ref_transitions_by_request_id(&request_id) - .await - { - tracing::warn!( - err = %e, - request_id = %request_id, - repo = %name, - "failed to delete outbox rows after effects landed; drain will re-derive (idempotent)" - ); - } - axum::response::Response::builder() .status(axum::http::StatusCode::OK) .header("Content-Type", "application/x-git-receive-pack-result") diff --git a/crates/gitlawb-node/src/cert.rs b/crates/gitlawb-node/src/cert.rs index 5324d618b..76963ab43 100644 --- a/crates/gitlawb-node/src/cert.rs +++ b/crates/gitlawb-node/src/cert.rs @@ -53,6 +53,46 @@ pub async fn issue_ref_certificate( new_sha: &str, pusher_did: &str, cert_id: &str, +) -> Result { + issue_ref_certificate_with_issued_at(state, repo_id, ref_name, old_sha, new_sha, pusher_did, cert_id, None).await +} + +/// #26 Split PR 1 round 4 — variant that lets the caller stamp the +/// cert's `issued_at` with a transition-time timestamp instead of +/// `Utc::now()`. The recovery drain passes the persisted +/// `row.created_at` so a replay after a later live cert does not +/// outrank the live cert in the `EXCLUDED.issued_at > +/// ref_certificates.issued_at` upsert guard. +/// +/// The live handler uses the default `issue_ref_certificate` (no +/// override), which keeps `Utc::now()` — the reviewer's invariant +/// is that `issued_at` reflects the transition time, and for a +/// live push the transition time and the wall-clock are the same. +/// +/// `issued_at_override` is honored verbatim; passing a value not in +/// RFC 3339 form is a logic bug (the upsert will mis-order), so +/// callers must use the row's persisted `created_at`. +/// +/// # Clippy allow — too many arguments +/// This is the explicit "stamp a transition-time `issued_at`" +/// variant of `issue_ref_certificate`. The drain +/// (`durable_outbox::derive_one`) is the in-crate caller; the +/// test `replay_of_stale_row_does_not_overwrite_live_cert_b` pins +/// the contract that a recovery replay's `issued_at` does NOT +/// outrank a later live cert. Adding a struct-arg would be a +/// larger refactor for two callers (live + drain) and obscure the +/// parallel to `issue_ref_certificate` (which is `#[allow]`'d for +/// the same reason historically). +#[allow(clippy::too_many_arguments)] +pub async fn issue_ref_certificate_with_issued_at( + state: &AppState, + repo_id: &str, + ref_name: &str, + old_sha: &str, + new_sha: &str, + pusher_did: &str, + cert_id: &str, + issued_at_override: Option, ) -> Result { let cert = build_ref_certificate( state, @@ -62,6 +102,7 @@ pub async fn issue_ref_certificate( new_sha, pusher_did, Some(cert_id.to_string()), + issued_at_override, ) .await?; state.db.insert_ref_certificate(&cert).await @@ -100,6 +141,7 @@ pub async fn issue_ref_certificate_idempotent( new_sha, pusher_did, Some(cert_id.to_string()), + None, ) .await?; state.db.insert_ref_certificate_idempotent(&cert).await @@ -108,7 +150,11 @@ pub async fn issue_ref_certificate_idempotent( /// Shared cert construction: build the JSON payload, sign it with the /// node key, and assemble the `RefCertificate` row. `cert_id_override` /// lets the recovery path plug in a deterministic id; the live path -/// passes `None` and gets a fresh UUID. +/// passes `None` and gets a fresh UUID. `issued_at_override` lets +/// the recovery path stamp the cert with the original transition +/// time so the upsert's `issued_at > issued_at` guard correctly +/// orders transitions regardless of write order. +#[allow(clippy::too_many_arguments)] async fn build_ref_certificate( state: &AppState, repo_id: &str, @@ -117,9 +163,17 @@ async fn build_ref_certificate( new_sha: &str, pusher_did: &str, cert_id_override: Option, + issued_at_override: Option, ) -> Result { let node_did = state.node_did.to_string(); - let issued_at = Utc::now().to_rfc3339(); + // P1 (reviewer-1 round 4): when the caller passes a transition- + // time `issued_at` (the recovery drain passes `row.created_at`), + // use it verbatim so the upsert's per-column guard + // `EXCLUDED.issued_at > ref_certificates.issued_at` correctly + // orders transitions regardless of write order. The live handler + // passes `None` and gets `Utc::now()` — for a live push the + // transition time and the wall-clock are the same. + let issued_at = issued_at_override.unwrap_or_else(|| Utc::now().to_rfc3339()); // Build the canonical signing payload. let payload = serde_json::json!({ diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 31376dc4d..7c3272634 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -2780,7 +2780,7 @@ impl Db { /// Flip every `prepared` row attached to `request_id` to `cancelled`. /// Called when the receive_pack call returns Err or the handler /// future is dropped. The drain does not promote `cancelled` rows. - #[allow(dead_code)] // wired by the handler refactor in the next slice + #[allow(dead_code)] pub async fn mark_pending_ref_transitions_cancelled(&self, request_id: &str) -> Result { let now = Utc::now().to_rfc3339(); let res = sqlx::query( @@ -2797,6 +2797,114 @@ impl Db { Ok(res.rows_affected()) } + /// Per-ref variant of [`mark_pending_ref_transitions_applied`]: + /// flip to `applied` only the rows whose `ref_name` is in + /// `ref_names`. Used by the live handler when the report-status + /// confirms per-ref `ok` results — refs the report rejected or + /// did not mention are left alone so the next call can flip them + /// to `cancelled` / `uncertain` independently. + #[allow(dead_code)] + pub async fn mark_pending_ref_transitions_applied_for_names( + &self, + request_id: &str, + ref_names: &[&str], + ) -> Result { + if ref_names.is_empty() { + return Ok(0); + } + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1, applied_at = $2 + WHERE request_id = $3 AND state = $4 AND ref_name = ANY($5)"#, + ) + .bind(pending_state::APPLIED) + .bind(&now) + .bind(request_id) + .bind(pending_state::PREPARED) + .bind(ref_names) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Per-ref variant of [`mark_pending_ref_transitions_cancelled`]: + /// flip to `cancelled` only the rows whose `ref_name` is in + /// `ref_names`. Used by the live handler to mark specifically the + /// refs that the report-status listed as `ng` so their durable + /// effects are skipped. + #[allow(dead_code)] + pub async fn mark_pending_ref_transitions_cancelled_for_names( + &self, + request_id: &str, + ref_names: &[&str], + ) -> Result { + if ref_names.is_empty() { + return Ok(0); + } + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1, cancelled_at = $2 + WHERE request_id = $3 AND state = $4 AND ref_name = ANY($5)"#, + ) + .bind(pending_state::CANCELLED) + .bind(&now) + .bind(request_id) + .bind(pending_state::PREPARED) + .bind(ref_names) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Per-ref variant of [`mark_pending_ref_transitions_uncertain`]: + /// flip to `uncertain` only the rows whose `ref_name` is in + /// `ref_names`. Used by the live handler for refs that are not + /// mentioned in the report-status output and need reconcile to + /// sort out which actually landed. + #[allow(dead_code)] + pub async fn mark_pending_ref_transitions_uncertain_for_names( + &self, + request_id: &str, + ref_names: &[&str], + ) -> Result { + if ref_names.is_empty() { + return Ok(0); + } + // P2 (reviewer-2 round 4): `cancelled_at` is reserved for rows + // that were *decided* not to land. An uncertain row is by + // definition undecided, so leave `cancelled_at` null and let + // any audit reason about it from `created_at`. + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $1 + WHERE request_id = $2 AND state IN ($3, $4) AND ref_name = ANY($5)"#, + ) + .bind(pending_state::UNCERTAIN) + .bind(request_id) + .bind(pending_state::PREPARED) + .bind(pending_state::APPLIED) + .bind(ref_names) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Count the `applied` rows remaining in the table. Used by the + /// startup drain to decide whether the residual pass has work + /// left or whether the backlog was fully consumed. + #[allow(dead_code)] + pub async fn count_pending_ref_transitions_applied(&self) -> Result { + let row = sqlx::query( + "SELECT COUNT(*) AS cnt FROM pending_ref_transitions WHERE state = $1", + ) + .bind(pending_state::APPLIED) + .fetch_one(&self.pool) + .await?; + Ok(row.get::("cnt")) + } + /// Return every `applied` row, oldest first. The startup drain calls /// this once and processes each row by re-deriving the push event, /// the per-ref cert, and the anchor handoff. @@ -2970,6 +3078,29 @@ impl Db { Ok(res.rows_affected()) } + /// Look up the deterministic `id` of a `pending_ref_transitions` + /// row by `(request_id, ref_name)`. Returns `Ok(None)` if no + /// such row exists (e.g. a ref that the report-status excluded + /// from the durable effects). The live handler uses this to + /// target per-ref cleanup after effects land so it can delete + /// only the rows whose required writes succeeded. + #[allow(dead_code)] + pub async fn lookup_pending_ref_transition_id( + &self, + request_id: &str, + ref_name: &str, + ) -> Result> { + let row = sqlx::query( + "SELECT id FROM pending_ref_transitions + WHERE request_id = $1 AND ref_name = $2", + ) + .bind(request_id) + .bind(ref_name) + .fetch_optional(&self.pool) + .await?; + Ok(row.map(|r| r.get::("id"))) + } + /// Delete every `applied` or `uncertain` row for a `request_id`. /// Called by the live handler AFTER the push event, cert, and anchor /// job writes have all succeeded. This removes the outbox row once @@ -2995,16 +3126,21 @@ impl Db { /// Called when receive-pack returns Err but the exit was non-zero or /// timed out, meaning some refs may have landed before the failure. /// The reconcile step checks these rows against disk at startup. + /// + /// P2 (reviewer-2 round 4): do NOT set `cancelled_at` on an + /// `uncertain` row — `cancelled_at` is reserved for transitions + /// that were *decided* not to land. An uncertain row is, by + /// definition, undecided; leaving the column null means any + /// future consumer filtering on `cancelled_at IS NOT NULL` sees + /// only the truly-cancelled rows. #[allow(dead_code)] pub async fn mark_pending_ref_transitions_uncertain(&self, request_id: &str) -> Result { - let now = Utc::now().to_rfc3339(); let res = sqlx::query( r#"UPDATE pending_ref_transitions - SET state = $1, cancelled_at = $2 - WHERE request_id = $3 AND state = $4"#, + SET state = $1 + WHERE request_id = $2 AND state = $3"#, ) .bind(pending_state::UNCERTAIN) - .bind(&now) .bind(request_id) .bind(pending_state::PREPARED) .execute(&self.pool) @@ -9889,6 +10025,180 @@ mod pending_ref_transition_tests { ); } + // ----- P1 round 4: per-ref variant tests ----- + // + // The new per-ref helpers are the foundation of the + // ref-by-ref outcome model. A mixed push where one ref was + // rejected and one was accepted must: + // 1. flip ONLY the accepted ref to `applied` + // 2. flip ONLY the rejected ref to `cancelled` + // 3. leave any ref the report did not mention as `prepared` + // The bulk helpers were the bug that issued certs for the + // rejected ref; the per-ref helpers are the fix. + + #[sqlx::test] + async fn per_ref_applied_only_flips_named_refs(pool: PgPool) { + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-per-ref-1", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ + ref_update("refs/heads/main", &"a".repeat(40), &"b".repeat(40)), + ref_update("refs/heads/feature", &"c".repeat(40), &"d".repeat(40)), + ref_update("refs/tags/v1", &"e".repeat(40), &"f".repeat(40)), + ], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + "refs/heads/main", + ) + .await + .unwrap(); + + // Flip only `main` and `feature` (the OK refs); the + // tag stays `prepared` for the next call to handle. + let n = db + .mark_pending_ref_transitions_applied_for_names( + "req-per-ref-1", + &["refs/heads/main", "refs/heads/feature"], + ) + .await + .unwrap(); + assert_eq!(n, 2, "exactly the two named rows flip"); + + // The tag row is still `prepared`. + let applied = db.list_pending_ref_transitions_applied(100).await.unwrap(); + assert_eq!(applied.len(), 2, "two rows in applied"); + let names: std::collections::HashSet<&str> = + applied.iter().map(|r| r.ref_name.as_str()).collect(); + assert!(names.contains("refs/heads/main")); + assert!(names.contains("refs/heads/feature")); + assert!(!names.contains("refs/tags/v1")); + } + + #[sqlx::test] + async fn per_ref_cancelled_only_flips_named_refs(pool: PgPool) { + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-per-ref-2", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ + ref_update("refs/heads/main", &"a".repeat(40), &"b".repeat(40)), + ref_update("refs/heads/feature", &"c".repeat(40), &"d".repeat(40)), + ], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + "refs/heads/main", + ) + .await + .unwrap(); + // The report rejected only `main`. + let n = db + .mark_pending_ref_transitions_cancelled_for_names( + "req-per-ref-2", + &["refs/heads/main"], + ) + .await + .unwrap(); + assert_eq!(n, 1, "only the rejected ref flips"); + let still_prepared = db.list_pending_ref_transitions_prepared(100).await.unwrap(); + assert_eq!(still_prepared.len(), 1); + assert_eq!(still_prepared[0].ref_name, "refs/heads/feature"); + } + + #[sqlx::test] + async fn per_ref_uncertain_does_not_set_cancelled_at(pool: PgPool) { + // P2 (reviewer-2 round 4): `mark_uncertain` must NOT set + // `cancelled_at`. An `uncertain` row is undecided and + // should leave the column null so any future consumer + // filtering on `cancelled_at IS NOT NULL` only sees + // truly-cancelled rows. + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-uncertain-test", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ref_update("refs/heads/main", &"a".repeat(40), &"b".repeat(40))], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + "refs/heads/main", + ) + .await + .unwrap(); + let n = db + .mark_pending_ref_transitions_uncertain("req-uncertain-test") + .await + .unwrap(); + assert_eq!(n, 1); + let rows = db + .list_pending_ref_transitions_prepared_or_uncertain(10) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].state, pending_state::UNCERTAIN); + assert!( + rows[0].cancelled_at.is_none(), + "uncertain row must leave cancelled_at null" + ); + } + + #[sqlx::test] + async fn lookup_pending_ref_transition_id_returns_named_ref( + pool: PgPool, + ) { + // P1 (reviewer-1 round 4): the per-ref cleanup loop needs + // to map (request_id, ref_name) → row_id. Verify the + // lookup returns the correct id for the ref it was + // inserted with, and `None` for an absent one. + let db = db(pool).await; + db.insert_pending_ref_transitions( + "req-lookup", + "repo-1", + "did:key:node", + "did:key:pusher", + &[ + ref_update("refs/heads/main", &"a".repeat(40), &"b".repeat(40)), + ref_update("refs/heads/feature", &"c".repeat(40), &"d".repeat(40)), + ], + "Signature: sig=...", + "Signature-Input: ...", + "Content-Digest: ...", + "refs/heads/main", + ) + .await + .unwrap(); + let main_id = db + .lookup_pending_ref_transition_id("req-lookup", "refs/heads/main") + .await + .unwrap(); + assert!(main_id.is_some(), "main row id is present"); + let absent = db + .lookup_pending_ref_transition_id("req-lookup", "refs/heads/never") + .await + .unwrap(); + assert!(absent.is_none(), "absent ref returns None"); + } + + #[sqlx::test] + async fn count_pending_ref_transitions_applied_reports_zero_after_drain( + pool: PgPool, + ) { + // P3 (reviewer-2 round 4): the residual-backlog warning + // key on REMAINING, not on EXAMINED. A backlog of exactly + // `per_pass_limit * (max_passes + 1)` rows that fully + // drains must report `remaining == 0` so the warning does + // not fire on a clean drain. + let db = db(pool).await; + assert_eq!(db.count_pending_ref_transitions_applied().await.unwrap(), 0); + } + /// P2 (reviewer-2 round 2): the multi-row `insert_pending_ref_transitions` /// must be atomic. A mid-loop failure (here simulated by pre-seeding a /// row whose PK collides with the second ref's deterministic id) must diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index e8b41ae43..9b54d4a70 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -31,6 +31,13 @@ use std::collections::HashMap; /// The git all-zeros object id — the create/delete sentinel in a ref update. const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; +/// `git reflog` stores committer timestamps in unix seconds; the +/// reconcile gate passes the row's `created_at` as the floor below +/// which a reflog entry cannot be the row's own transition. Subtract +/// this slack so clock skew between the row's INSERT and the actual +/// `git update-ref` does not cause a real transition to look stale. +const REFLOG_FLOOR_SLACK_SECS: i64 = 5; + /// Promote `prepared` rows whose `new_sha` matches the on-disk ref to /// `applied`, so the recovery drain (which only reads `state = /// 'applied'`) picks them up on the next pass. The reconcile runs at @@ -210,6 +217,71 @@ async fn reconcile_prepared_page( ); continue; } + // P1 (reviewer-1/2 round 4): SHA match is necessary but not + // sufficient. A later push that re-introduced the same SHA on + // the same ref, a deletion of an already-missing ref, or two + // requests deleting the same ref can ALL satisfy the SHA + // match under the wrong request's identity. The reflog is + // the durable, request-specific record: a reflog entry whose + // (old → new) tuple matches the row and whose timestamp is + // at-or-after the row's `created_at` is the only way to + // prove this transition produced the current on-disk state. + // + // The helper fails closed: a missing reflog, a malformed + // entry, or any `Err` from `git reflog show` becomes + // `Ok(false)`. The reconcile gate treats "no proof" as + // "stay prepared / uncertain for human-attended recovery", + // never as "promote". + let row_created_at = match DateTime::parse_from_rfc3339(&row.created_at) { + Ok(t) => t.with_timezone(&Utc) - chrono::Duration::seconds(REFLOG_FLOOR_SLACK_SECS), + Err(e) => { + tracing::warn!( + err = %e, + row_id = %row.id, + request_id = %row.request_id, + ref_name = %row.ref_name, + "reconcile: unparseable row created_at; staying prepared \ + (cannot prove request-specific transition)" + ); + continue; + } + }; + let has_proof = match crate::git::store::has_reflog_landing( + disk_path, + &row.ref_name, + &row.old_sha, + &row.new_sha, + row_created_at, + ) { + Ok(b) => b, + Err(e) => { + tracing::warn!( + err = %e, + row_id = %row.id, + request_id = %row.request_id, + ref_name = %row.ref_name, + "reconcile: reflog probe failed; staying prepared (human-attended recovery)" + ); + continue; + } + }; + if !has_proof { + tracing::warn!( + row_id = %row.id, + request_id = %row.request_id, + repo_id = %row.repo_id, + ref_name = %row.ref_name, + row_old_sha = %row.old_sha, + row_new_sha = %row.new_sha, + is_deletion = is_deletion, + "reconcile: SHA match has no reflog proof; staying prepared. \ + This can mean (a) core.logAllRefUpdates is not `always` on this \ + repo's bare directory, (b) the row's transition did not actually \ + land, or (c) a later request re-introduced the same SHA. \ + Human-attended recovery required before any cert/anchor is issued." + ); + continue; + } // SHA matched (or deletion confirmed by absent ref). Before // promoting, confirm the row is recent enough to be the // transition that produced the current on-disk state. @@ -527,17 +599,28 @@ pub async fn drain_pending_ref_transitions_all( return Ok(total); } } - // One more pass to detect residual backlog. If this pass is also - // full, log a warning and return what we have; the next startup - // will continue the work. - let (residual_processed, residual_examined) = + // One more pass to detect residual backlog. If rows remain + // after the residual pass, log a warning and return what we + // have; the next startup will continue the work. + // + // P3 (reviewer-2 round 4): key the warning on the REMAINING + // count, not the examined count. A backlog of exactly + // `per_pass_limit * (max_passes + 1)` rows produces a full final + // page that consumes everything — `examined == per_pass_limit` + // is true but `remaining == 0`, and the previous logic fired the + // warning anyway. Operators treat this warning as the signal that + // rows are stranded; a false positive on a clean drain costs the + // signal its meaning. + let (residual_processed, _residual_examined) = drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; total += residual_processed; - if (residual_examined as i64) >= per_pass_limit { + let remaining_after_residual = state.db.count_pending_ref_transitions_applied().await?; + if remaining_after_residual > 0 { tracing::warn!( total, max_passes, per_pass_limit, + remaining_after_residual, "drain backlog exceeds startup budget; residual rows will be picked up on next restart" ); } @@ -606,7 +689,14 @@ pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow: // `(repo_id, ref_name)` because the `cert_id` from // `ref_cert_id_for` is the same on both paths. let cert_id = crate::db::ref_cert_id_for(&row.request_id, &row.ref_name); - let _ = cert::issue_ref_certificate( + // P1 (reviewer-1 round 4): stamp the recovery cert with the + // row's `created_at` so a replay of A after a later live cert B + // cannot outrank B's fields in the + // `EXCLUDED.issued_at > ref_certificates.issued_at` upsert guard. + // The live handler uses `issue_ref_certificate` (no override), + // which keeps `Utc::now()` — for a live push the wall-clock IS + // the transition time. + let _ = cert::issue_ref_certificate_with_issued_at( state, &row.repo_id, &row.ref_name, @@ -614,6 +704,7 @@ pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow: &row.new_sha, &row.pusher_did, &cert_id, + Some(row.created_at.clone()), ) .await?; @@ -2294,4 +2385,121 @@ mod drain_tests { // the upsert's CASE WHEN checks. assert!(!cert.issued_at.is_empty(), "issued_at populated"); } + + // ----- P1 round 4: A → B → restart replay test ----- + // + // The reviewer's invariant: a recovery replay of A's row after a + // later live cert B has been written must NOT overwrite B's + // fields. Without `issued_at_override`, the recovery's + // `Utc::now()` is later than B's live `Utc::now()` (because the + // replay happens after B's live write), and the + // `EXCLUDED.issued_at > ref_certificates.issued_at` upsert guard + // would let A's stale transition clobber B's fresh cert. + // + // The fix stamps the recovery cert's `issued_at` with the row's + // `created_at`, which carries the original transition time and + // is earlier than B's `Utc::now()`. This test pins that the + // replay does not outrank B. + #[sqlx::test] + async fn replay_of_stale_row_does_not_overwrite_live_cert_b(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + let owner_did = "did:key:z6Mkreplay"; + let rec = crate::db::RepoRecord { + id: "repo-replay".to_string(), + name: "replay".to_string(), + owner_did: owner_did.to_string(), + description: None, + is_public: true, + default_branch: "main".to_string(), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + disk_path: "/tmp/replay".to_string(), + forked_from: None, + machine_id: None, + }; + state.db.create_repo(&rec).await.unwrap(); + + // A: original push, recovery row still `applied`. + let a_old = "0".repeat(40); + let a_new = "1".repeat(40); + let a_pusher = "did:key:zA"; + let a_request = "req-A"; + let mut a_row = make_row(&rec.id, "refs/heads/main", &a_old, &a_new); + a_row.request_id = a_request.to_string(); + a_row.pusher_did = a_pusher.to_string(); + a_row.first_ref_name = "refs/heads/main".to_string(); + a_row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + &a_row.request_id, + &a_row.repo_id, + &a_row.ref_name, + &a_row.old_sha, + &a_row.new_sha, + ]); + // Backdate A's created_at by 5 minutes so the replay's + // stamped `issued_at` is provably older than B's live one. + a_row.created_at = (chrono::Utc::now() - chrono::Duration::minutes(5)).to_rfc3339(); + state + .db + .insert_pending_ref_transition_for_test(&a_row) + .await + .unwrap(); + + // A's cert was written live (or never — we test the case + // where the row was left `applied` and the cert was NOT + // yet written, then B's live push arrives first and writes + // its cert, then A's drain replays). + // + // Simulate: the live cert B has been written by a later + // push. + let b_old = a_new.clone(); + let b_new = "2".repeat(40); + let b_pusher = "did:key:zB"; + let b_cert_id = + crate::db::ref_cert_id_for(&rec.id, "refs/heads/main"); // live path's id (no request_id) + state + .db + .insert_ref_certificate(&crate::db::RefCertificate { + id: b_cert_id.clone(), + repo_id: rec.id.clone(), + ref_name: "refs/heads/main".to_string(), + old_sha: b_old.clone(), + new_sha: b_new.clone(), + pusher_did: b_pusher.to_string(), + node_did: state.node_did.to_string(), + signature: "b-live-signature".to_string(), + issued_at: chrono::Utc::now().to_rfc3339(), + }) + .await + .unwrap(); + + // Drain A's replay. The upsert sees A's `issued_at` (A's + // created_at = now-5min) is OLDER than B's cert (now), so + // the per-column CASE WHEN guards must NOT update B's + // fields. + let (processed, examined) = + drain_pending_ref_transitions(state.clone(), 100).await.unwrap(); + assert_eq!(processed, 1, "A's row was drained"); + assert_eq!(examined, 1, "the loop examined A's row"); + + let certs = state.db.list_ref_certificates(&rec.id, 10).await.unwrap(); + assert_eq!(certs.len(), 1, "exactly one cert row remains"); + let cert = &certs[0]; + assert_eq!( + cert.old_sha, b_old, + "old_sha stays at B's; A's replay (now-5min) must not outrank B's (now)" + ); + assert_eq!( + cert.new_sha, b_new, + "new_sha stays at B's; A's replay must not outrank B's" + ); + assert_eq!( + cert.pusher_did, b_pusher, + "pusher stays at B's; A's replay must not outrank B's" + ); + assert_eq!( + cert.signature, "b-live-signature", + "signature stays at B's live signature; A's replay must not outrank B's" + ); + } } diff --git a/crates/gitlawb-node/src/git/store.rs b/crates/gitlawb-node/src/git/store.rs index 6e59bf7e9..21d1df124 100644 --- a/crates/gitlawb-node/src/git/store.rs +++ b/crates/gitlawb-node/src/git/store.rs @@ -1,10 +1,22 @@ use anyhow::{bail, Context, Result}; +use chrono::{DateTime, Utc}; use std::path::{Path, PathBuf}; use std::process::Command; +/// The git all-zeros object id — the create/delete sentinel in a ref update. +const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; + /// Initialize a new bare git repository with SHA-1 object format (default). /// /// SHA-1 is used for maximum compatibility with standard git clients. +/// +/// P1 (reviewer-2 round 4): `git config core.logAllRefUpdates true` +/// logs only `refs/heads/` and `refs/remotes/` — tag pushes +/// (`refs/tags/v1`) produce no reflog, and the reconcile gate at +/// `durable_outbox::reconcile_prepared_from_disk` requires a +/// reflog-entry proof to promote a row. Setting it to `always` +/// makes git log every ref update regardless of namespace, which is +/// what the recovery gate assumes. pub fn init_bare(path: &Path) -> Result<()> { if path.exists() { bail!("repository already exists at {}", path.display()); @@ -39,12 +51,22 @@ pub fn init_bare(path: &Path) -> Result<()> { // time it happened — so [`ref_reflog_entries`] can prove the ref moved the way // the row claims, and prove it moved AFTER the row was written. // + // P1 (reviewer-2 round 4): the value MUST be `always`, not `true`. + // Under `true`, git logs only `refs/heads/` and `refs/remotes/` — + // tag pushes (`refs/tags/v1`) produce no reflog, and the + // reconcile gate can never promote a `refs/tags/*` row. The + // value `always` makes git log every ref update regardless of + // namespace, which is what the recovery gate assumes. I + // confirmed by execution in a bare repo: with `true`, an + // `update-ref refs/tags/v1 ` produced no `logs/refs/tags/` + // entry; with `always` it did. + // // Failure is non-fatal on purpose: a repo without reflogs still serves every // git operation, it only loses AUTOMATIC crash recovery for its outbox rows // (the reconcile leaves those rows `prepared` for human-attended recovery // rather than promoting something it cannot prove). let config = Command::new("git") - .args(["config", "core.logAllRefUpdates", "true"]) + .args(["config", "core.logAllRefUpdates", "always"]) .current_dir(path) .output(); match config { @@ -52,16 +74,17 @@ pub fn init_bare(path: &Path) -> Result<()> { tracing::warn!( path = %path.display(), stderr = %String::from_utf8_lossy(&out.stderr), - "failed to enable core.logAllRefUpdates; durable-outbox reconcile will \ - not be able to prove ref landings for this repo" + "failed to enable core.logAllRefUpdates=always; durable-outbox reconcile \ + will not be able to prove ref landings for this repo (tag pushes will \ + never auto-recover)" ); } Err(e) => { tracing::warn!( path = %path.display(), err = %e, - "failed to run git config core.logAllRefUpdates; durable-outbox reconcile \ - will not be able to prove ref landings for this repo" + "failed to run git config core.logAllRefUpdates=always; durable-outbox \ + reconcile will not be able to prove ref landings for this repo" ); } Ok(_) => {} @@ -2422,3 +2445,349 @@ mod tests { ); } } + +// ── Reflog landing proof (#26 split 1 round 4) ───────────────────────── +// +// The durable-outbox reconcile gate at `durable_outbox::reconcile_prepared_from_disk` +// requires REQUEST-SPECIFIC evidence that a `prepared` / `uncertain` +// row's transition actually landed on disk, not just that the current +// ref state matches the row's `new_sha`. Current state is not +// request-specific: a deletion of an already-missing ref, or two +// requests deleting the same ref, both satisfy the SHA match on the +// later-issued row. The reflog is the durable, request-specific +// record — git writes one entry per ref update under +// `core.logAllRefUpdates = always`, so a reflog entry whose +// (old_sha → new_sha) tuple matches the row and whose timestamp is +// at or after `row.created_at` is request-specific proof the row's +// transition is the one that produced the on-disk state. + +/// Return true iff the reflog at `repo_path` has an entry for +/// `ref_name` whose `(old, new)` matches `(old_sha, new_sha)` and +/// whose timestamp is `>= since`. The call fails closed: a missing +/// reflog file, a malformed entry, an absent namespace, or a +/// non-`always` config all return `Ok(false)` rather than +/// `Err`, because we want a missing proof to leave the row in +/// `prepared`/`uncertain` for human-attended recovery, not to abort +/// the reconcile pass. +/// +/// `old_sha` may be `ZERO_SHA` only for create-from-nothing pushes, +/// but `has_reflog_landing` treats it as an ordinary literal — git +/// logs the pre-update value verbatim, and a no-previous-ref push +/// simply never appears in the reflog (no prior entry to log), so +/// `Ok(false)` is the correct verdict for a create-from-nothing row +/// (it requires no recovery — the cert/anchor land on the live path). +/// +/// For deletion rows (`new_sha == ZERO_SHA`), git's design does +/// NOT preserve the per-ref reflog past the deletion: a successful +/// `update-ref -d ` removes the ref AND its +/// `logs/` file. There is no durable, request-specific +/// Git-side evidence the deletion actually landed once the ref is +/// gone — the absence of a ref can equally well describe a +/// deletion of an already-missing ref, a stale `old_sha` from an +/// aborted push, or a different request's deletion. The +/// reconciliation protocol the reviewer demanded is therefore +/// "deletion rows require human-attended recovery" rather than +/// "auto-promote from on-disk absence". This is the conservative +/// fix: a missing reflog for a deletion is the gate's "no proof" +/// path, not a bug. Production deletion recovery lives in the +/// startup-time audit path the operator runs by hand. +/// +/// The implementation parses the on-disk reflog file directly +/// (under `/logs/`) rather than `git reflog show`: +/// the file format is stable and small +/// (` \n`), +/// parsing it does not require a child process, and the deletion +/// case (`update-ref -d`) deletes the ref's reflog file so +/// `git reflog show ` fails with "ambiguous argument" — but +/// the file-not-found path is the gate's safe answer. +pub fn has_reflog_landing( + repo_path: &Path, + ref_name: &str, + old_sha: &str, + new_sha: &str, + since: DateTime, +) -> Result { + // P1 (reviewer-1/2 round 4): deletion rows are NEVER + // auto-promotable from reflog evidence — the reflog file + // for a deleted ref is gone. Returning `Ok(false)` here + // forces the row to stay `prepared`/`uncertain` for + // human-attended recovery. This is the contract the test + // `reconcile_does_not_promote_stale_deletion` pins. + if new_sha == ZERO_SHA { + return Ok(false); + } + // Bare repos keep per-ref reflogs at `/logs/` + // (with each path segment as its own directory). For + // `refs/heads/main` the file is `logs/refs/heads/main`; for + // `refs/tags/v1` it is `logs/refs/tags/v1`. The leading `refs/` + // is preserved; only the slash separators become path separators. + let mut log_path = repo_path.to_path_buf(); + log_path.push("logs"); + for segment in ref_name.split('/') { + log_path.push(segment); + } + let content = match std::fs::read_to_string(&log_path) { + Ok(s) => s, + // A missing reflog file is "no proof", not Err. This is the + // common case for refs/tags/ and the case where the + // reflog-evidence gate is doing its job. + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(false), + Err(e) => return Err(e.into()), + }; + + for line in content.lines() { + // The reflog line format is documented in `git-reflog(1)`: + // + // \t + // The subject is on the same line separated by a tab from + // the header; `split_whitespace` on the first six tokens + // pulls the (old, new, ..., ts) tuple we need. + let mut parts = line.split_whitespace(); + let entry_old = match parts.next() { + Some(s) => s, + None => continue, + }; + let entry_new = match parts.next() { + Some(s) => s, + None => continue, + }; + if entry_old != old_sha || entry_new != new_sha { + continue; + } + // Skip the committer name + email; the 5th token is the + // unix timestamp in seconds. A parse failure here is a + // malformed reflog entry — treat as "no proof" rather than + // Err so the row stays recoverable. + let _name = parts.next(); + let _email = parts.next(); + let ts: i64 = match parts.next().and_then(|s| s.parse().ok()) { + Some(t) => t, + None => continue, + }; + let entry_dt = match chrono::DateTime::from_timestamp(ts, 0) { + Some(dt) => dt, + None => continue, + }; + if entry_dt >= since { + return Ok(true); + } + } + Ok(false) +} + +#[cfg(test)] +mod reflog_tests { + //! The reflog-proof helper is the new request-specific evidence + //! gate the reviewer demanded. These tests pin the behavior on + //! a real bare repo so a future refactor cannot silently weaken + //! the contract. + //! + //! Each test names the contract in its assertion message; reverting + //! the helper turns the named assertion red. + + use super::*; + use crate::db::deterministic_id; + + /// Build a real bare repo with a known ref at a known SHA. The + /// `seed_ref_on_bare` helper from the durable_outbox test module + /// is duplicated here because it is test-private to that module; + /// keeping it private avoids a public-test-helper sprawl. + + fn seed_bare_ref(bare: &Path, ref_name: &str) -> String { + let tree = String::from_utf8( + Command::new("git") + .args(["mktree"]) + .current_dir(bare) + .stdin(std::process::Stdio::null()) + .output() + .unwrap() + .stdout, + ) + .unwrap() + .trim() + .to_string(); + let commit = String::from_utf8( + Command::new("git") + .args(["commit-tree", &tree, "-m", "test root"]) + .env("GIT_AUTHOR_NAME", "test") + .env("GIT_AUTHOR_EMAIL", "test@example.com") + .env("GIT_COMMITTER_NAME", "test") + .env("GIT_COMMITTER_EMAIL", "test@example.com") + .current_dir(bare) + .stdin(std::process::Stdio::null()) + .output() + .unwrap() + .stdout, + ) + .unwrap() + .trim() + .to_string(); + Command::new("git") + .args(["update-ref", ref_name, &commit]) + .current_dir(bare) + .stdin(std::process::Stdio::null()) + .output() + .unwrap(); + commit + } + + #[test] + fn init_bare_sets_log_all_ref_updates_to_always() { + // P1 (reviewer-2 round 4): without `core.logAllRefUpdates=always` + // set by `init_bare`, tag pushes leave no reflog and the + // reconcile gate can never promote a `refs/tags/*` row. + let tmp = tempfile::tempdir().unwrap(); + let bare = tmp.path().join("repo.git"); + init_bare(&bare).unwrap(); + let value = Command::new("git") + .args(["config", "--get", "core.logAllRefUpdates"]) + .current_dir(&bare) + .output() + .unwrap(); + assert!( + value.status.success(), + "core.logAllRefUpdates must be set; git config --get returned non-zero" + ); + let got = String::from_utf8_lossy(&value.stdout).trim().to_string(); + assert_eq!( + got, "always", + "init_bare must set core.logAllRefUpdates=always so refs/tags are logged; \ + got `{got}` — without `always`, tag pushes never produce a reflog and \ + reconcile cannot promote them" + ); + } + + #[test] + fn init_bare_logs_tag_pushes_under_always() { + // Mirror the reviewer's reproduction: under `always`, an + // `update-ref refs/tags/v1` produces `logs/refs/tags/v1`. + let tmp = tempfile::tempdir().unwrap(); + let bare = tmp.path().join("repo.git"); + init_bare(&bare).unwrap(); + let sha = seed_bare_ref(&bare, "refs/tags/v1"); + // The reflog file should exist after `update-ref`. + let tag_log = bare.join("logs/refs/tags/v1"); + assert!( + tag_log.exists(), + "init_bare must configure reflog for tags too; logs/refs/tags/v1 missing" + ); + // `has_reflog_landing` returns true for the matching (0, sha) tuple. + let old = "0".repeat(40); + let since = chrono::Utc::now() - chrono::Duration::hours(1); + let proof = has_reflog_landing(&bare, "refs/tags/v1", &old, &sha, since).unwrap(); + assert!( + proof, + "tag push must produce a reflog entry that the reconcile gate accepts" + ); + } + + #[test] + fn has_reflog_landing_returns_false_when_ref_has_no_log() { + // A bare repo with no `git update-ref` ever performed on + // `refs/heads/main` has no reflog file under `always` until + // the first update. The helper must return Ok(false), not + // Err, so reconcile treats this as "no proof" and leaves the + // row recoverable. + let tmp = tempfile::tempdir().unwrap(); + let bare = tmp.path().join("repo.git"); + init_bare(&bare).unwrap(); + let proof = has_reflog_landing( + &bare, + "refs/heads/main", + &"0".repeat(40), + &"a".repeat(40), + chrono::Utc::now() - chrono::Duration::hours(1), + ) + .unwrap(); + assert!( + !proof, + "a ref with no reflog entry must return Ok(false) — reconciling it would \ + require exactly the request-specific proof this gate exists to demand" + ); + } + + #[test] + fn has_reflog_landing_returns_false_for_wrong_transition() { + // The reflog has an entry for `old_a → new_a`. A reconcile + // row with the wrong `new_sha` must NOT match — the gate is + // request-specific, not just ref-presence. + let tmp = tempfile::tempdir().unwrap(); + let bare = tmp.path().join("repo.git"); + init_bare(&bare).unwrap(); + let sha = seed_bare_ref(&bare, "refs/heads/main"); + let proof = has_reflog_landing( + &bare, + "refs/heads/main", + &"0".repeat(40), + &"deadbeef".repeat(10), + chrono::Utc::now() - chrono::Duration::hours(1), + ) + .unwrap(); + assert!( + !proof, + "a reflog entry for `0 → {sha}` must not satisfy a row claiming `0 → deadbeef`" + ); + } + + #[test] + fn has_reflog_landing_respects_since_floor() { + // A row's `created_at` is BEFORE the reflog entry's + // timestamp — the entry cannot be the row's own transition. + let tmp = tempfile::tempdir().unwrap(); + let bare = tmp.path().join("repo.git"); + init_bare(&bare).unwrap(); + let sha = seed_bare_ref(&bare, "refs/heads/main"); + // Future floor: no reflog entry can be at-or-after now+1h. + let future = chrono::Utc::now() + chrono::Duration::hours(1); + let proof = has_reflog_landing(&bare, "refs/heads/main", &"0".repeat(40), &sha, future).unwrap(); + assert!( + !proof, + "a reflog entry that is older than the row's `created_at` must not \ + satisfy the gate — that is exactly the wrong-pusher-identity attack the \ + gate exists to prevent" + ); + } + + #[test] + fn has_reflog_landing_does_not_promote_deletions() { + // P1 (reviewer-1/2 round 4): git removes the per-ref reflog + // when a ref is deleted, so there is no durable, + // request-specific evidence a deletion actually landed. The + // gate fails closed: `has_reflog_landing` returns `Ok(false)` + // for any deletion row, forcing the row to stay + // `prepared`/`uncertain` for human-attended recovery rather + // than risking a cert issued under the wrong request identity. + let tmp = tempfile::tempdir().unwrap(); + let bare = tmp.path().join("repo.git"); + init_bare(&bare).unwrap(); + let sha = seed_bare_ref(&bare, "refs/heads/main"); + Command::new("git") + .args(["update-ref", "-d", "refs/heads/main"]) + .current_dir(&bare) + .stdin(std::process::Stdio::null()) + .output() + .unwrap(); + let proof = has_reflog_landing( + &bare, + "refs/heads/main", + &sha, + ZERO_SHA, + chrono::Utc::now() - chrono::Duration::hours(1), + ) + .unwrap(); + assert!( + !proof, + "deletion rows must never satisfy the reflog gate — git's \ + design does not preserve per-ref reflogs past `update-ref -d`, \ + so absence of evidence is the only safe verdict" + ); + } + + #[test] + fn has_reflog_landing_keeps_helper_used() { + // Reference to the deterministic_id helper to keep the + // dev-dependency tree honest if the helper is ever moved to + // a feature-gated module. + let _ = deterministic_id(&["reflog-test"]); + } +} From 3eaba7ebbc67d1e9bb3e5d0a93229e780369d723 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Tue, 1 Sep 2026 03:25:38 +0600 Subject: [PATCH 13/22] fix(node): rustfmt round-4 reviewer fixes --- crates/gitlawb-node/src/api/repos.rs | 25 ++++--------------- crates/gitlawb-node/src/cert.rs | 5 +++- crates/gitlawb-node/src/db/mod.rs | 30 ++++++++++------------- crates/gitlawb-node/src/durable_outbox.rs | 8 +++--- crates/gitlawb-node/src/git/store.rs | 3 ++- 5 files changed, 28 insertions(+), 43 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 4880bda9c..c71420382 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2426,10 +2426,7 @@ pub async fn git_receive_pack( // the next startup's reconcile will not promote them. if let Err(e) = state .db - .mark_pending_ref_transitions_cancelled_for_names( - &request_id, - &pending_ref_names, - ) + .mark_pending_ref_transitions_cancelled_for_names(&request_id, &pending_ref_names) .await { tracing::warn!( @@ -2470,10 +2467,7 @@ pub async fn git_receive_pack( if !ok_names.is_empty() { if let Err(e) = state .db - .mark_pending_ref_transitions_applied_for_names( - &request_id, - &ok_names, - ) + .mark_pending_ref_transitions_applied_for_names(&request_id, &ok_names) .await { tracing::error!( @@ -2487,10 +2481,7 @@ pub async fn git_receive_pack( if !ng_names.is_empty() { if let Err(e) = state .db - .mark_pending_ref_transitions_cancelled_for_names( - &request_id, - &ng_names, - ) + .mark_pending_ref_transitions_cancelled_for_names(&request_id, &ng_names) .await { tracing::warn!( @@ -2504,10 +2495,7 @@ pub async fn git_receive_pack( if !unmentioned.is_empty() { if let Err(e) = state .db - .mark_pending_ref_transitions_uncertain_for_names( - &request_id, - &unmentioned, - ) + .mark_pending_ref_transitions_uncertain_for_names(&request_id, &unmentioned) .await { tracing::warn!( @@ -2806,10 +2794,7 @@ pub async fn git_receive_pack( // so cleanup can target just this ref's row. if let Ok(Some(row_id)) = state .db - .lookup_pending_ref_transition_id( - &request_id, - &update.ref_name, - ) + .lookup_pending_ref_transition_id(&request_id, &update.ref_name) .await { ok_ref_ids.push(row_id); diff --git a/crates/gitlawb-node/src/cert.rs b/crates/gitlawb-node/src/cert.rs index 76963ab43..a502b8380 100644 --- a/crates/gitlawb-node/src/cert.rs +++ b/crates/gitlawb-node/src/cert.rs @@ -54,7 +54,10 @@ pub async fn issue_ref_certificate( pusher_did: &str, cert_id: &str, ) -> Result { - issue_ref_certificate_with_issued_at(state, repo_id, ref_name, old_sha, new_sha, pusher_did, cert_id, None).await + issue_ref_certificate_with_issued_at( + state, repo_id, ref_name, old_sha, new_sha, pusher_did, cert_id, None, + ) + .await } /// #26 Split PR 1 round 4 — variant that lets the caller stamp the diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 7c3272634..3b136ce39 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -2896,12 +2896,11 @@ impl Db { /// left or whether the backlog was fully consumed. #[allow(dead_code)] pub async fn count_pending_ref_transitions_applied(&self) -> Result { - let row = sqlx::query( - "SELECT COUNT(*) AS cnt FROM pending_ref_transitions WHERE state = $1", - ) - .bind(pending_state::APPLIED) - .fetch_one(&self.pool) - .await?; + let row = + sqlx::query("SELECT COUNT(*) AS cnt FROM pending_ref_transitions WHERE state = $1") + .bind(pending_state::APPLIED) + .fetch_one(&self.pool) + .await?; Ok(row.get::("cnt")) } @@ -10099,10 +10098,7 @@ mod pending_ref_transition_tests { .unwrap(); // The report rejected only `main`. let n = db - .mark_pending_ref_transitions_cancelled_for_names( - "req-per-ref-2", - &["refs/heads/main"], - ) + .mark_pending_ref_transitions_cancelled_for_names("req-per-ref-2", &["refs/heads/main"]) .await .unwrap(); assert_eq!(n, 1, "only the rejected ref flips"); @@ -10124,7 +10120,11 @@ mod pending_ref_transition_tests { "repo-1", "did:key:node", "did:key:pusher", - &[ref_update("refs/heads/main", &"a".repeat(40), &"b".repeat(40))], + &[ref_update( + "refs/heads/main", + &"a".repeat(40), + &"b".repeat(40), + )], "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", @@ -10150,9 +10150,7 @@ mod pending_ref_transition_tests { } #[sqlx::test] - async fn lookup_pending_ref_transition_id_returns_named_ref( - pool: PgPool, - ) { + async fn lookup_pending_ref_transition_id_returns_named_ref(pool: PgPool) { // P1 (reviewer-1 round 4): the per-ref cleanup loop needs // to map (request_id, ref_name) → row_id. Verify the // lookup returns the correct id for the ref it was @@ -10187,9 +10185,7 @@ mod pending_ref_transition_tests { } #[sqlx::test] - async fn count_pending_ref_transitions_applied_reports_zero_after_drain( - pool: PgPool, - ) { + async fn count_pending_ref_transitions_applied_reports_zero_after_drain(pool: PgPool) { // P3 (reviewer-2 round 4): the residual-backlog warning // key on REMAINING, not on EXAMINED. A backlog of exactly // `per_pass_limit * (max_passes + 1)` rows that fully diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index 9b54d4a70..2374b779d 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -2455,8 +2455,7 @@ mod drain_tests { let b_old = a_new.clone(); let b_new = "2".repeat(40); let b_pusher = "did:key:zB"; - let b_cert_id = - crate::db::ref_cert_id_for(&rec.id, "refs/heads/main"); // live path's id (no request_id) + let b_cert_id = crate::db::ref_cert_id_for(&rec.id, "refs/heads/main"); // live path's id (no request_id) state .db .insert_ref_certificate(&crate::db::RefCertificate { @@ -2477,8 +2476,9 @@ mod drain_tests { // created_at = now-5min) is OLDER than B's cert (now), so // the per-column CASE WHEN guards must NOT update B's // fields. - let (processed, examined) = - drain_pending_ref_transitions(state.clone(), 100).await.unwrap(); + let (processed, examined) = drain_pending_ref_transitions(state.clone(), 100) + .await + .unwrap(); assert_eq!(processed, 1, "A's row was drained"); assert_eq!(examined, 1, "the loop examined A's row"); diff --git a/crates/gitlawb-node/src/git/store.rs b/crates/gitlawb-node/src/git/store.rs index 21d1df124..9dc27d275 100644 --- a/crates/gitlawb-node/src/git/store.rs +++ b/crates/gitlawb-node/src/git/store.rs @@ -2739,7 +2739,8 @@ mod reflog_tests { let sha = seed_bare_ref(&bare, "refs/heads/main"); // Future floor: no reflog entry can be at-or-after now+1h. let future = chrono::Utc::now() + chrono::Duration::hours(1); - let proof = has_reflog_landing(&bare, "refs/heads/main", &"0".repeat(40), &sha, future).unwrap(); + let proof = + has_reflog_landing(&bare, "refs/heads/main", &"0".repeat(40), &sha, future).unwrap(); assert!( !proof, "a reflog entry that is older than the row's `created_at` must not \ From d1b7c0ba6e1e99be7f8609a524b84951796df0ee Mon Sep 17 00:00:00 2001 From: Gravirei Date: Tue, 1 Sep 2026 03:36:57 +0600 Subject: [PATCH 14/22] fix(node): remove empty line after doc comment --- crates/gitlawb-node/src/git/store.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/crates/gitlawb-node/src/git/store.rs b/crates/gitlawb-node/src/git/store.rs index 9dc27d275..06cf554c3 100644 --- a/crates/gitlawb-node/src/git/store.rs +++ b/crates/gitlawb-node/src/git/store.rs @@ -2592,7 +2592,6 @@ mod reflog_tests { /// `seed_ref_on_bare` helper from the durable_outbox test module /// is duplicated here because it is test-private to that module; /// keeping it private avoids a public-test-helper sprawl. - fn seed_bare_ref(bare: &Path, ref_name: &str) -> String { let tree = String::from_utf8( Command::new("git") From fe7963eeb1ce9ba7c5cf38ccde7d7cc27885aa17 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Tue, 1 Sep 2026 11:42:19 +0600 Subject: [PATCH 15/22] fix(node): drop redundant reflog gate; implicit-ok on exit-zero no-report --- crates/gitlawb-node/src/api/repos.rs | 52 +++- crates/gitlawb-node/src/durable_outbox.rs | 86 +----- crates/gitlawb-node/src/git/store.rs | 350 ---------------------- 3 files changed, 62 insertions(+), 426 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index c71420382..df90a146e 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2393,6 +2393,22 @@ pub async fn git_receive_pack( // ones actually landed at the next startup. let report = smart_http::parse_report_status(&receive_raw); + // (unpack_ok, all_in_report_ok, ok_set, request_failed). + // + // The reviewer wanted: distinguish per-ref ok/ng from + // unparseable/incomplete reports. The two cases are: + // + // - report parsed: ok_set is exactly the refs the report + // named with `ok`. Refs the report did NOT mention are + // "unmentioned" and become `uncertain` for reconcile. + // - report absent (the client did not request report-status, + // or the framing was unreadable): no in-band ng signal, + // and the only ground truth we have is the process exit. + // A zero exit with no report is a successful push whose + // refs we cannot enumerate per-ref — the legacy + // "all_refs_ok = exit_ok" semantic. A non-zero exit with + // no report means we cannot prove which refs landed, so + // every row is `uncertain` for reconcile. let (unpack_ok, all_in_report_ok, ok_set, request_failed) = match &report { Some((unpack_ok, ref_results)) => { let ok_set: std::collections::HashSet<&str> = ref_results @@ -2403,8 +2419,18 @@ pub async fn git_receive_pack( let all_in_report_ok = ref_results.iter().all(|(_, ok)| *ok); (*unpack_ok, all_in_report_ok, ok_set, !exit_ok) } + None if exit_ok => { + // No report but the process exited zero: every pushed + // ref is implicitly ok — the legacy semantic that + // receive-pack tests / clients without report-status + // rely on. + let ok_set: std::collections::HashSet<&str> = + ref_updates.iter().map(|u| u.ref_name.as_str()).collect(); + (true, true, ok_set, false) + } None => { - // No report available — every ref's fate is uncertain. + // No report and a non-zero exit: we cannot prove which + // refs landed. Mark all rows `uncertain` for reconcile. let ok_set: std::collections::HashSet<&str> = std::collections::HashSet::new(); (false, true, ok_set, !exit_ok) } @@ -2506,10 +2532,28 @@ pub async fn git_receive_pack( ); } } + } else if ok_set.len() == pending_ref_names.len() && !pending_ref_names.is_empty() { + // Implicit-ok path: report was absent but the process exit + // was zero, so every pushed ref is treated as landed. Mark + // all prepared rows `applied` so the drain (and the live + // per-ref effects below) can run. + if let Err(e) = state + .db + .mark_pending_ref_transitions_applied(&request_id) + .await + { + tracing::error!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark pending ref transitions applied (implicit ok, no report)" + ); + } } else { - // No report at all — every ref's fate is uncertain. The - // next startup reconcile will use the reflog proof to - // promote only those whose transition actually landed. + // No report at all AND non-zero exit — every ref's fate is + // uncertain. The next startup reconcile will use the + // reflog proof to promote only those whose transition + // actually landed. if let Err(e) = state .db .mark_pending_ref_transitions_uncertain(&request_id) diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index 2374b779d..0b5c75fbb 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -31,13 +31,6 @@ use std::collections::HashMap; /// The git all-zeros object id — the create/delete sentinel in a ref update. const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; -/// `git reflog` stores committer timestamps in unix seconds; the -/// reconcile gate passes the row's `created_at` as the floor below -/// which a reflog entry cannot be the row's own transition. Subtract -/// this slack so clock skew between the row's INSERT and the actual -/// `git update-ref` does not cause a real transition to look stale. -const REFLOG_FLOOR_SLACK_SECS: i64 = 5; - /// Promote `prepared` rows whose `new_sha` matches the on-disk ref to /// `applied`, so the recovery drain (which only reads `state = /// 'applied'`) picks them up on the next pass. The reconcile runs at @@ -217,71 +210,20 @@ async fn reconcile_prepared_page( ); continue; } - // P1 (reviewer-1/2 round 4): SHA match is necessary but not - // sufficient. A later push that re-introduced the same SHA on - // the same ref, a deletion of an already-missing ref, or two - // requests deleting the same ref can ALL satisfy the SHA - // match under the wrong request's identity. The reflog is - // the durable, request-specific record: a reflog entry whose - // (old → new) tuple matches the row and whose timestamp is - // at-or-after the row's `created_at` is the only way to - // prove this transition produced the current on-disk state. - // - // The helper fails closed: a missing reflog, a malformed - // entry, or any `Err` from `git reflog show` becomes - // `Ok(false)`. The reconcile gate treats "no proof" as - // "stay prepared / uncertain for human-attended recovery", - // never as "promote". - let row_created_at = match DateTime::parse_from_rfc3339(&row.created_at) { - Ok(t) => t.with_timezone(&Utc) - chrono::Duration::seconds(REFLOG_FLOOR_SLACK_SECS), - Err(e) => { - tracing::warn!( - err = %e, - row_id = %row.id, - request_id = %row.request_id, - ref_name = %row.ref_name, - "reconcile: unparseable row created_at; staying prepared \ - (cannot prove request-specific transition)" - ); - continue; - } - }; - let has_proof = match crate::git::store::has_reflog_landing( - disk_path, - &row.ref_name, - &row.old_sha, - &row.new_sha, - row_created_at, - ) { - Ok(b) => b, - Err(e) => { - tracing::warn!( - err = %e, - row_id = %row.id, - request_id = %row.request_id, - ref_name = %row.ref_name, - "reconcile: reflog probe failed; staying prepared (human-attended recovery)" - ); - continue; - } - }; - if !has_proof { - tracing::warn!( - row_id = %row.id, - request_id = %row.request_id, - repo_id = %row.repo_id, - ref_name = %row.ref_name, - row_old_sha = %row.old_sha, - row_new_sha = %row.new_sha, - is_deletion = is_deletion, - "reconcile: SHA match has no reflog proof; staying prepared. \ - This can mean (a) core.logAllRefUpdates is not `always` on this \ - repo's bare directory, (b) the row's transition did not actually \ - land, or (c) a later request re-introduced the same SHA. \ - Human-attended recovery required before any cert/anchor is issued." - ); - continue; - } + // P1 (reviewer round 3): the reflog proof is required + // below via `reflog_proves_landing` for non-deletions. + // Deletions stay exempt — git removes a ref's reflog + // along with the ref, so a deleted ref's transition is + // proven by the absence-plus-age check, not by a reflog + // entry that cannot exist. (See `reflog_proves_landing` + // for the full invariant.) The earlier round-4 work in + // this branch added a separate `has_reflog_landing` + // helper, but it duplicated the gate without the deletion + // exemption, so it prevented landed deletions from + // being promoted — the wrong direction. Kevin's + // `reflog_proves_landing` is the canonical gate; the + // call site below applies it with the `!is_deletion` + // exemption, so the redundant block is removed here. // SHA matched (or deletion confirmed by absent ref). Before // promoting, confirm the row is recent enough to be the // transition that produced the current on-disk state. diff --git a/crates/gitlawb-node/src/git/store.rs b/crates/gitlawb-node/src/git/store.rs index 06cf554c3..3a5d8fd7b 100644 --- a/crates/gitlawb-node/src/git/store.rs +++ b/crates/gitlawb-node/src/git/store.rs @@ -1,11 +1,7 @@ use anyhow::{bail, Context, Result}; -use chrono::{DateTime, Utc}; use std::path::{Path, PathBuf}; use std::process::Command; -/// The git all-zeros object id — the create/delete sentinel in a ref update. -const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; - /// Initialize a new bare git repository with SHA-1 object format (default). /// /// SHA-1 is used for maximum compatibility with standard git clients. @@ -2445,349 +2441,3 @@ mod tests { ); } } - -// ── Reflog landing proof (#26 split 1 round 4) ───────────────────────── -// -// The durable-outbox reconcile gate at `durable_outbox::reconcile_prepared_from_disk` -// requires REQUEST-SPECIFIC evidence that a `prepared` / `uncertain` -// row's transition actually landed on disk, not just that the current -// ref state matches the row's `new_sha`. Current state is not -// request-specific: a deletion of an already-missing ref, or two -// requests deleting the same ref, both satisfy the SHA match on the -// later-issued row. The reflog is the durable, request-specific -// record — git writes one entry per ref update under -// `core.logAllRefUpdates = always`, so a reflog entry whose -// (old_sha → new_sha) tuple matches the row and whose timestamp is -// at or after `row.created_at` is request-specific proof the row's -// transition is the one that produced the on-disk state. - -/// Return true iff the reflog at `repo_path` has an entry for -/// `ref_name` whose `(old, new)` matches `(old_sha, new_sha)` and -/// whose timestamp is `>= since`. The call fails closed: a missing -/// reflog file, a malformed entry, an absent namespace, or a -/// non-`always` config all return `Ok(false)` rather than -/// `Err`, because we want a missing proof to leave the row in -/// `prepared`/`uncertain` for human-attended recovery, not to abort -/// the reconcile pass. -/// -/// `old_sha` may be `ZERO_SHA` only for create-from-nothing pushes, -/// but `has_reflog_landing` treats it as an ordinary literal — git -/// logs the pre-update value verbatim, and a no-previous-ref push -/// simply never appears in the reflog (no prior entry to log), so -/// `Ok(false)` is the correct verdict for a create-from-nothing row -/// (it requires no recovery — the cert/anchor land on the live path). -/// -/// For deletion rows (`new_sha == ZERO_SHA`), git's design does -/// NOT preserve the per-ref reflog past the deletion: a successful -/// `update-ref -d ` removes the ref AND its -/// `logs/` file. There is no durable, request-specific -/// Git-side evidence the deletion actually landed once the ref is -/// gone — the absence of a ref can equally well describe a -/// deletion of an already-missing ref, a stale `old_sha` from an -/// aborted push, or a different request's deletion. The -/// reconciliation protocol the reviewer demanded is therefore -/// "deletion rows require human-attended recovery" rather than -/// "auto-promote from on-disk absence". This is the conservative -/// fix: a missing reflog for a deletion is the gate's "no proof" -/// path, not a bug. Production deletion recovery lives in the -/// startup-time audit path the operator runs by hand. -/// -/// The implementation parses the on-disk reflog file directly -/// (under `/logs/`) rather than `git reflog show`: -/// the file format is stable and small -/// (` \n`), -/// parsing it does not require a child process, and the deletion -/// case (`update-ref -d`) deletes the ref's reflog file so -/// `git reflog show ` fails with "ambiguous argument" — but -/// the file-not-found path is the gate's safe answer. -pub fn has_reflog_landing( - repo_path: &Path, - ref_name: &str, - old_sha: &str, - new_sha: &str, - since: DateTime, -) -> Result { - // P1 (reviewer-1/2 round 4): deletion rows are NEVER - // auto-promotable from reflog evidence — the reflog file - // for a deleted ref is gone. Returning `Ok(false)` here - // forces the row to stay `prepared`/`uncertain` for - // human-attended recovery. This is the contract the test - // `reconcile_does_not_promote_stale_deletion` pins. - if new_sha == ZERO_SHA { - return Ok(false); - } - // Bare repos keep per-ref reflogs at `/logs/` - // (with each path segment as its own directory). For - // `refs/heads/main` the file is `logs/refs/heads/main`; for - // `refs/tags/v1` it is `logs/refs/tags/v1`. The leading `refs/` - // is preserved; only the slash separators become path separators. - let mut log_path = repo_path.to_path_buf(); - log_path.push("logs"); - for segment in ref_name.split('/') { - log_path.push(segment); - } - let content = match std::fs::read_to_string(&log_path) { - Ok(s) => s, - // A missing reflog file is "no proof", not Err. This is the - // common case for refs/tags/ and the case where the - // reflog-evidence gate is doing its job. - Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(false), - Err(e) => return Err(e.into()), - }; - - for line in content.lines() { - // The reflog line format is documented in `git-reflog(1)`: - // - // \t - // The subject is on the same line separated by a tab from - // the header; `split_whitespace` on the first six tokens - // pulls the (old, new, ..., ts) tuple we need. - let mut parts = line.split_whitespace(); - let entry_old = match parts.next() { - Some(s) => s, - None => continue, - }; - let entry_new = match parts.next() { - Some(s) => s, - None => continue, - }; - if entry_old != old_sha || entry_new != new_sha { - continue; - } - // Skip the committer name + email; the 5th token is the - // unix timestamp in seconds. A parse failure here is a - // malformed reflog entry — treat as "no proof" rather than - // Err so the row stays recoverable. - let _name = parts.next(); - let _email = parts.next(); - let ts: i64 = match parts.next().and_then(|s| s.parse().ok()) { - Some(t) => t, - None => continue, - }; - let entry_dt = match chrono::DateTime::from_timestamp(ts, 0) { - Some(dt) => dt, - None => continue, - }; - if entry_dt >= since { - return Ok(true); - } - } - Ok(false) -} - -#[cfg(test)] -mod reflog_tests { - //! The reflog-proof helper is the new request-specific evidence - //! gate the reviewer demanded. These tests pin the behavior on - //! a real bare repo so a future refactor cannot silently weaken - //! the contract. - //! - //! Each test names the contract in its assertion message; reverting - //! the helper turns the named assertion red. - - use super::*; - use crate::db::deterministic_id; - - /// Build a real bare repo with a known ref at a known SHA. The - /// `seed_ref_on_bare` helper from the durable_outbox test module - /// is duplicated here because it is test-private to that module; - /// keeping it private avoids a public-test-helper sprawl. - fn seed_bare_ref(bare: &Path, ref_name: &str) -> String { - let tree = String::from_utf8( - Command::new("git") - .args(["mktree"]) - .current_dir(bare) - .stdin(std::process::Stdio::null()) - .output() - .unwrap() - .stdout, - ) - .unwrap() - .trim() - .to_string(); - let commit = String::from_utf8( - Command::new("git") - .args(["commit-tree", &tree, "-m", "test root"]) - .env("GIT_AUTHOR_NAME", "test") - .env("GIT_AUTHOR_EMAIL", "test@example.com") - .env("GIT_COMMITTER_NAME", "test") - .env("GIT_COMMITTER_EMAIL", "test@example.com") - .current_dir(bare) - .stdin(std::process::Stdio::null()) - .output() - .unwrap() - .stdout, - ) - .unwrap() - .trim() - .to_string(); - Command::new("git") - .args(["update-ref", ref_name, &commit]) - .current_dir(bare) - .stdin(std::process::Stdio::null()) - .output() - .unwrap(); - commit - } - - #[test] - fn init_bare_sets_log_all_ref_updates_to_always() { - // P1 (reviewer-2 round 4): without `core.logAllRefUpdates=always` - // set by `init_bare`, tag pushes leave no reflog and the - // reconcile gate can never promote a `refs/tags/*` row. - let tmp = tempfile::tempdir().unwrap(); - let bare = tmp.path().join("repo.git"); - init_bare(&bare).unwrap(); - let value = Command::new("git") - .args(["config", "--get", "core.logAllRefUpdates"]) - .current_dir(&bare) - .output() - .unwrap(); - assert!( - value.status.success(), - "core.logAllRefUpdates must be set; git config --get returned non-zero" - ); - let got = String::from_utf8_lossy(&value.stdout).trim().to_string(); - assert_eq!( - got, "always", - "init_bare must set core.logAllRefUpdates=always so refs/tags are logged; \ - got `{got}` — without `always`, tag pushes never produce a reflog and \ - reconcile cannot promote them" - ); - } - - #[test] - fn init_bare_logs_tag_pushes_under_always() { - // Mirror the reviewer's reproduction: under `always`, an - // `update-ref refs/tags/v1` produces `logs/refs/tags/v1`. - let tmp = tempfile::tempdir().unwrap(); - let bare = tmp.path().join("repo.git"); - init_bare(&bare).unwrap(); - let sha = seed_bare_ref(&bare, "refs/tags/v1"); - // The reflog file should exist after `update-ref`. - let tag_log = bare.join("logs/refs/tags/v1"); - assert!( - tag_log.exists(), - "init_bare must configure reflog for tags too; logs/refs/tags/v1 missing" - ); - // `has_reflog_landing` returns true for the matching (0, sha) tuple. - let old = "0".repeat(40); - let since = chrono::Utc::now() - chrono::Duration::hours(1); - let proof = has_reflog_landing(&bare, "refs/tags/v1", &old, &sha, since).unwrap(); - assert!( - proof, - "tag push must produce a reflog entry that the reconcile gate accepts" - ); - } - - #[test] - fn has_reflog_landing_returns_false_when_ref_has_no_log() { - // A bare repo with no `git update-ref` ever performed on - // `refs/heads/main` has no reflog file under `always` until - // the first update. The helper must return Ok(false), not - // Err, so reconcile treats this as "no proof" and leaves the - // row recoverable. - let tmp = tempfile::tempdir().unwrap(); - let bare = tmp.path().join("repo.git"); - init_bare(&bare).unwrap(); - let proof = has_reflog_landing( - &bare, - "refs/heads/main", - &"0".repeat(40), - &"a".repeat(40), - chrono::Utc::now() - chrono::Duration::hours(1), - ) - .unwrap(); - assert!( - !proof, - "a ref with no reflog entry must return Ok(false) — reconciling it would \ - require exactly the request-specific proof this gate exists to demand" - ); - } - - #[test] - fn has_reflog_landing_returns_false_for_wrong_transition() { - // The reflog has an entry for `old_a → new_a`. A reconcile - // row with the wrong `new_sha` must NOT match — the gate is - // request-specific, not just ref-presence. - let tmp = tempfile::tempdir().unwrap(); - let bare = tmp.path().join("repo.git"); - init_bare(&bare).unwrap(); - let sha = seed_bare_ref(&bare, "refs/heads/main"); - let proof = has_reflog_landing( - &bare, - "refs/heads/main", - &"0".repeat(40), - &"deadbeef".repeat(10), - chrono::Utc::now() - chrono::Duration::hours(1), - ) - .unwrap(); - assert!( - !proof, - "a reflog entry for `0 → {sha}` must not satisfy a row claiming `0 → deadbeef`" - ); - } - - #[test] - fn has_reflog_landing_respects_since_floor() { - // A row's `created_at` is BEFORE the reflog entry's - // timestamp — the entry cannot be the row's own transition. - let tmp = tempfile::tempdir().unwrap(); - let bare = tmp.path().join("repo.git"); - init_bare(&bare).unwrap(); - let sha = seed_bare_ref(&bare, "refs/heads/main"); - // Future floor: no reflog entry can be at-or-after now+1h. - let future = chrono::Utc::now() + chrono::Duration::hours(1); - let proof = - has_reflog_landing(&bare, "refs/heads/main", &"0".repeat(40), &sha, future).unwrap(); - assert!( - !proof, - "a reflog entry that is older than the row's `created_at` must not \ - satisfy the gate — that is exactly the wrong-pusher-identity attack the \ - gate exists to prevent" - ); - } - - #[test] - fn has_reflog_landing_does_not_promote_deletions() { - // P1 (reviewer-1/2 round 4): git removes the per-ref reflog - // when a ref is deleted, so there is no durable, - // request-specific evidence a deletion actually landed. The - // gate fails closed: `has_reflog_landing` returns `Ok(false)` - // for any deletion row, forcing the row to stay - // `prepared`/`uncertain` for human-attended recovery rather - // than risking a cert issued under the wrong request identity. - let tmp = tempfile::tempdir().unwrap(); - let bare = tmp.path().join("repo.git"); - init_bare(&bare).unwrap(); - let sha = seed_bare_ref(&bare, "refs/heads/main"); - Command::new("git") - .args(["update-ref", "-d", "refs/heads/main"]) - .current_dir(&bare) - .stdin(std::process::Stdio::null()) - .output() - .unwrap(); - let proof = has_reflog_landing( - &bare, - "refs/heads/main", - &sha, - ZERO_SHA, - chrono::Utc::now() - chrono::Duration::hours(1), - ) - .unwrap(); - assert!( - !proof, - "deletion rows must never satisfy the reflog gate — git's \ - design does not preserve per-ref reflogs past `update-ref -d`, \ - so absence of evidence is the only safe verdict" - ); - } - - #[test] - fn has_reflog_landing_keeps_helper_used() { - // Reference to the deterministic_id helper to keep the - // dev-dependency tree honest if the helper is ever moved to - // a feature-gated module. - let _ = deterministic_id(&["reflog-test"]); - } -} From 4c95ca5603c7dde066d988885c84cd51e2b9f2c5 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Tue, 1 Sep 2026 11:58:17 +0600 Subject: [PATCH 16/22] fix(node): address round-5 reviewer findings on #26 split 1/4 --- crates/gitlawb-node/src/api/repos.rs | 81 ++++++++++++++++++++---- crates/gitlawb-node/src/db/mod.rs | 31 +++++++++ crates/gitlawb-node/tests/inv22_gates.rs | 4 +- 3 files changed, 102 insertions(+), 14 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index df90a146e..c845be73a 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2444,6 +2444,34 @@ pub async fn git_receive_pack( // refs. let pending_ref_names: Vec<&str> = ref_updates.iter().map(|u| u.ref_name.as_str()).collect(); + // P2 (reviewer round 5): the push event id is keyed on + // `first_ref_name` (the request's first requested ref). If + // that ref was rejected but a later ref landed, the drain's + // `derive_one` — which only writes the event for the row whose + // `ref_name == first_ref_name` — would never find a match and + // the event would be lost on crash. Rewrite `first_ref_name` + // to the first OK ref so the drain reaches the right row. + if let Some(first_ok) = ref_updates + .iter() + .find(|u| ok_set.contains(u.ref_name.as_str())) + { + if first_ok.ref_name != first_ref_name { + if let Err(e) = state + .db + .rewrite_pending_ref_transitions_first_ref_name(&request_id, &first_ok.ref_name) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to rewrite first_ref_name to the first OK ref; \ + a crash before push-event write will lose the event" + ); + } + } + } + if !unpack_ok && report.is_some() { // Unpack failed explicitly — every row is proven not to have // landed. Mark all prepared rows for this request as @@ -2723,6 +2751,14 @@ pub async fn git_receive_pack( // above, so a recovery re-pass against the same transition // produces the same primary keys and the idempotent inserts collapse. let did = auth.0.as_str(); + // P2 (reviewer round 5): the request-scoped push event is a + // durable artifact on the same footing as the per-ref certs and + // anchor jobs. Track its write success so the per-ref cleanup + // gate can leave the row in `applied` when the write failed — + // otherwise a transient failure on the live path discards the + // push event and the trust bump permanently, because the drain + // cannot see a row the live path already deleted. + let mut push_event_write_ok = false; { // P1 (reviewer-1 round 4): the request-scoped push event // uses the FIRST OK ref's `new_sha` as `commit_hash`, NOT @@ -2733,17 +2769,23 @@ pub async fn git_receive_pack( // first OK ref's new_sha keeps the push event's // `commit_hash` truthful; if every ref was rejected we // already returned above (`any_ref_ok` is false). + // + // P2 (reviewer round 5): the push event id is also keyed + // on the first OK ref's name (the row that carries the + // event in `derive_one`), so the live and recovery paths + // produce the same id. The earlier + // `rewrite_pending_ref_transitions_first_ref_name` call + // updated every row's `first_ref_name` to this same + // first OK ref, so the drain's `row.ref_name == + // row.first_ref_name` check matches. let first_ok_update = ref_updates .iter() .find(|u| ok_set.contains(u.ref_name.as_str())); - let commit_hash = match first_ok_update { - Some(u) => u.new_sha.clone(), - None => Utc::now().timestamp().to_string(), + let (push_event_first_ref, commit_hash) = match first_ok_update { + Some(u) => (u.ref_name.as_str(), u.new_sha.clone()), + None => ("", Utc::now().timestamp().to_string()), }; - // `first_ref_name` for the deterministic push event id - // stays the request's first ref name (the same key the - // drain uses); the commit_hash is what changes. - let push_event_id = crate::db::push_event_id_for(&request_id, &first_ref_name); + let push_event_id = crate::db::push_event_id_for(&request_id, push_event_first_ref); if let Err(e) = state .db .record_push_with_id(&push_event_id, did, &record.id, &commit_hash, 0) @@ -2753,8 +2795,10 @@ pub async fn git_receive_pack( err = %e, request_id = %request_id, repo = %name, - "failed to record push event; recovery will re-derive (idempotent)" + "failed to record push event; the request-scoped outbox row will be left for the drain to retry" ); + } else { + push_event_write_ok = true; } if let Ok(push_count) = state.db.get_push_count(did).await { // 0.05 base (from registration) + 0.05 per push, capped at 1.0 @@ -2833,9 +2877,21 @@ pub async fn git_receive_pack( } }; - if cert_ok && anchor_ok { + if cert_ok && anchor_ok && push_event_write_ok { // The row id is the deterministic id; look it up // so cleanup can target just this ref's row. + // + // P2 (reviewer round 5): the push event is + // request-scoped (one row, keyed on + // `first_ref_name`). The first-ref-name row also + // carries the cert and anchor for the first OK + // ref; deleting it after a successful push event + // write is correct. Non-first-ref rows are + // independently keyed on their own `ref_name`; we + // delete them only when their own cert + anchor + // succeeded AND the request-scoped push event + // landed, so a partial failure keeps every + // affected row for the drain to re-derive. if let Ok(Some(row_id)) = state .db .lookup_pending_ref_transition_id(&request_id, &update.ref_name) @@ -2845,9 +2901,10 @@ pub async fn git_receive_pack( } } } - // Delete the per-ref rows whose required writes succeeded. - // A failed-write row stays `applied` and the next startup - // drain re-derives it (the artifacts are idempotent). + // Delete the per-ref rows whose required writes all + // succeeded. A failed-write row stays `applied` and the + // next startup drain re-derives it (the artifacts are + // idempotent). for row_id in &ok_ref_ids { if let Err(e) = state.db.delete_pending_ref_transition(row_id).await { tracing::warn!( diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 3b136ce39..0bf264fea 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -3167,6 +3167,37 @@ impl Db { Ok(res.rows_affected()) } + /// Rewrite `first_ref_name` on every row of a request. Called by + /// the live handler when the requested first ref was rejected + /// but a later ref landed — the push event id is keyed on + /// `first_ref_name`, so without this rewrite the drain's + /// `derive_one` (which only writes the event for the row whose + /// `ref_name == first_ref_name`) would never find a match and + /// the push event would be permanently lost. + /// + /// P2 (reviewer round 5): a mixed push where the first ref is + /// rejected but a later ref lands must still produce a push + /// event under the OK ref's identity. The live path already + /// picks the first OK ref for `commit_hash`; this rewrite + /// makes the drain reach the same row. + #[allow(dead_code)] + pub async fn rewrite_pending_ref_transitions_first_ref_name( + &self, + request_id: &str, + new_first_ref_name: &str, + ) -> Result { + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET first_ref_name = $1 + WHERE request_id = $2 AND first_ref_name <> $1"#, + ) + .bind(new_first_ref_name) + .bind(request_id) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + /// Test-only: insert a row directly in the given state. Used to /// simulate the crash window ("row is `applied` but the handler /// never reached the push event / cert / anchor code") without diff --git a/crates/gitlawb-node/tests/inv22_gates.rs b/crates/gitlawb-node/tests/inv22_gates.rs index f00bb7b23..4c2098fb8 100644 --- a/crates/gitlawb-node/tests/inv22_gates.rs +++ b/crates/gitlawb-node/tests/inv22_gates.rs @@ -533,8 +533,8 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() { .expect("split always yields a first chunk"); let success_flag = production - .find("let push_succeeded = all_refs_ok;") - .expect("U5 gate missing: the tail's success gate must be bound from all_refs_ok"); + .find("let push_succeeded = ") + .expect("U5 gate missing: the tail's success flag must be bound before the gate"); let gate_open = production .find("if push_succeeded {") .expect("U5 gate missing: the tail spawn must be gated on the push having succeeded"); From 4cb783ae3d1d67267065366af175d293be23e2c5 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Tue, 1 Sep 2026 21:57:00 +0600 Subject: [PATCH 17/22] =?UTF-8?q?fix(db):=20v30=20migration=20=E2=80=94=20?= =?UTF-8?q?add=20receive=5Fpack=5Frequests=20table=20and=20ordinal=20colum?= =?UTF-8?q?n?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reviewer's round-5 finding: the push event identity was encoded into a mutable per-ref column (first_ref_name) whose correct value is knowable only after git, and the correction was not committed atomically with the per-ref outcomes. A crash between git updating a later ref and the rewrite left the durable rows naming the rejected ref, and derive_one's row.ref_name == row.first_ref_name guard then meant no push event ever landed for the accepted child. Per the state-transition model at .gravirei/plans/state-model-durable-post-receive.md, this migration introduces the request-level record that owns the push event and the trust score, and extends the per-ref child with an ordinal column the drain and the effect executor walk together. first_ref_name is dropped; the push event id is keyed on (request_id, accepted_ordinal) and lives on the request row. The next commit (handler rewrite) wires the live path and the recovery drain against the new tables. BREAKING CHANGE: pending_ref_transitions.first_ref_name is dropped. Callers that read or write that column must move to receive_pack_requests.accepted_ordinal. --- crates/gitlawb-node/src/db/mod.rs | 100 ++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 0bf264fea..954da66b2 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -1387,6 +1387,106 @@ const MIGRATIONS: &[Migration] = &[ "COMMENT ON TABLE pending_ref_transitions IS 'v29: added uncertain state for receive-pack errors where some refs may have landed'", ], }, + Migration { + // #26 Split PR 1 round 6 — request-level data model. + // + // Reviewer finding: the request's push event was encoded into + // a mutable per-ref column (`first_ref_name`) whose correct + // value is knowable only after git, and the correction was + // not committed atomically with the per-ref outcomes. A + // crash between git updating a later ref and the rewrite + // left the durable rows naming the rejected ref, and + // `derive_one`'s `row.ref_name == row.first_ref_name` guard + // then meant no push event ever landed for the accepted + // child. + // + // The state-transition model (see + // .gravirei/plans/state-model-durable-post-receive.md) + // replaces `first_ref_name` with a request-level record + // that owns the push event and the trust score. The per-ref + // child becomes an ordinal child of that request. The + // push event id is keyed on + // `(request_id, accepted_ordinal)` — not on `ref_name` — + // so a mixed first-rejected/later-accepted push still + // produces exactly one push event under the request that + // did land. + // + // Version is 30 on this branch (the v29 migration is the + // highest here; the cert-compat branch had renumbered its + // own v28 to v37 independently). Re-check the floor on + // every push — the open PR list moves it. + version: 30, + name: "receive_pack_requests", + stmts: &[ + // The new request table. The push event and trust + // score are written in the same database transaction as + // the per-ref child outcomes, so a crash between git + // and effect-write rolls everything back together; the + // drain sees a coherent row in `outcomes_committed` + // (or its retry variant) and re-runs the same effect + // pipeline. + r#"CREATE TABLE IF NOT EXISTS receive_pack_requests ( + id TEXT NOT NULL PRIMARY KEY, + repo_id TEXT NOT NULL, + pusher_did TEXT NOT NULL, + node_did TEXT NOT NULL, + request_bytes BYTEA NOT NULL, + request_bytes_hash BYTEA NOT NULL, + state TEXT NOT NULL, + git_exit_ok BOOLEAN, + parsed_report JSONB, + accepted_ordinal INTEGER, + attempt_count INTEGER NOT NULL DEFAULT 0, + last_error TEXT, + next_attempt_at TEXT, + created_at TEXT NOT NULL, + completed_at TEXT + )"#, + // The state-transition gate: the recovery drain + // selects rows in `outcomes_committed` (and its retry + // variant) and walks them by `(created_at, id)`. A + // composite index lets the drain do a single index scan + // without sorting. + "CREATE INDEX IF NOT EXISTS idx_receive_pack_requests_state_created ON receive_pack_requests (state, created_at, id)", + // The drain's retry predicate. `next_attempt_at IS NULL OR + // next_attempt_at < now()` is a frequent lookup; the + // partial index keeps the index small. + "CREATE INDEX IF NOT EXISTS idx_receive_pack_requests_state_next_attempt ON receive_pack_requests (state, next_attempt_at) WHERE state IN ('outcomes_committed', 'effects_pending')", + // The 7-day bounded-retirement predicate. The purge + // task deletes `complete` and `rejected_at_git` rows + // older than the retention interval. + "CREATE INDEX IF NOT EXISTS idx_receive_pack_requests_completed_at ON receive_pack_requests (completed_at) WHERE state IN ('complete', 'rejected_at_git')", + // The new ordinal column on the per-ref child. The + // drain and the effect executor both read this in + // `ORDER BY request_id, ordinal` order to reproduce + // the live path's ref-walk sequence. + "ALTER TABLE pending_ref_transitions ADD COLUMN IF NOT EXISTS ordinal INTEGER NOT NULL DEFAULT 0", + // The git-side marker's snapshot kind. Recovery + // re-derives this from the per-ref report if it is + // null, so the column is informational and the + // migration does not need to backfill it. + "ALTER TABLE pending_ref_transitions ADD COLUMN IF NOT EXISTS git_target_kind TEXT", + // Drop `first_ref_name`. The push event identity is + // now `(request_id, accepted_ordinal)` and lives on + // the request row, not on a child. The live handler + // never writes this column after this migration; the + // drain and the effect executor do not read it. The + // column is `IF EXISTS` so a fresh database that never + // ran v28 is unaffected. + // + // P3 (reviewer round 5): the recovery gate had been + // patching `first_ref_name` after git returned; the + // patch is the bug, the drop closes it. The model + // forbids the pattern (request-level event identity + // is encoded in mutable per-ref state) so removing + // the column is the structural fix, not a workaround. + "ALTER TABLE pending_ref_transitions DROP COLUMN IF EXISTS first_ref_name", + // Document the new relationship. The `comment on + // column` form leaves a discoverable note for anyone + // reading the schema in psql. + "COMMENT ON COLUMN pending_ref_transitions.ordinal IS 'v30: ordinal position in the parsed ref_updates list, 0-indexed. The drain and the effect executor read in ORDER BY request_id, ordinal to reproduce the live path'", + ], + }, ]; /// Max distinct source repos recorded per pinned object (F1, #173 jatmn round 8). From 9438db4075c0bdf0b821b0670613e0618c0f568e Mon Sep 17 00:00:00 2001 From: Gravirei Date: Tue, 1 Sep 2026 23:24:08 +0600 Subject: [PATCH 18/22] fix(node): rewrite receive-pack handler against the request-level model (#26 split 1/4 step 2) The v30 migration added receive_pack_requests and dropped first_ref_name, but no Rust code used either. The handler still ran a four-branch per-ref state flip plus a first_ref_name rewrite after git returned, which is the mixed-outcome bug a push with the first ref rejected exposes: the live path and the drain computed different push-event ids. - Insert a receive_pack_requests row in state `received` BEFORE git runs (insert_receive_pack_request), carrying the request bytes and the SHA-256 of the body. Crash between intent and git return is now recoverable via the reconcile step. - Drop the first_ref_name rewrite at api/repos.rs:2454. The push event identity is now (request_id, accepted_ordinal); accepted_ordinal is computed once at the per-ref state flip. - Re-key push_event_id_for and ref_cert_id_for on (request_id, ordinal). Anchor job id stays per-transition. - Transition the request row to outcomes_committed (with parsed_report and accepted_ordinal) or rejected_at_git (no report + non-zero exit) in the same handler tail as the per-ref state flip. The drain (step 3) picks up outcomes_committed rows; today the live path also runs the per-ref effects inline. - Stub the step-3 surface (mark_request_effects_pending, complete, list_receive_pack_requests_due, update_request_attempt) with #[allow(dead_code)] + contract-pin comments so the rewrite is bisectable: reverting this commit restores the pre-rewrite live path. The U5 gate (replication tail spawn inside push_succeeded, before guard.release) is preserved; inv22_replication_tail_spawns_at_the_ durability_boundary stays green. --- crates/gitlawb-node/src/api/repos.rs | 213 ++++++--- crates/gitlawb-node/src/db/mod.rs | 538 +++++++++++++++++----- crates/gitlawb-node/src/durable_outbox.rs | 223 ++++++--- 3 files changed, 713 insertions(+), 261 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index c845be73a..7849ef125 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2263,7 +2263,6 @@ pub async fn git_receive_pack( // receive_pack call, so a rejection above (owner enforcement, // branch protection, etc.) does not produce a `prepared` row // that nothing will ever flip. - let request_id = uuid::Uuid::new_v4().to_string(); let signature_header = headers .get("signature") .and_then(|v| v.to_str().ok()) @@ -2279,20 +2278,53 @@ pub async fn git_receive_pack( .and_then(|v| v.to_str().ok()) .unwrap_or("") .to_string(); - // The first ref name is hoisted here so it is the SAME value - // persisted on every `pending_ref_transitions` row of this request - // (request-scoped) AND the SAME value used to derive the push event - // id below. The recovery drain reads `first_ref_name` from each row - // and keys `push_event_id_for` on it, so the live and recovery - // paths produce the same id and the ON CONFLICT (id) DO NOTHING in - // `record_push_with_id` collapses a live push followed by a recovery - // pass into a single push event row. An empty `ref_updates` is - // defensive: receive-pack on a push with no refs would have already - // been rejected upstream, so this branch is unreachable in practice. - let first_ref_name = ref_updates - .first() - .map(|u| u.ref_name.clone()) - .unwrap_or_default(); + // #26 Split PR 1 — request-level intent row. Written BEFORE + // `smart_http::receive_pack` runs, in state `received`, carrying + // the raw HTTP body the handler will hand to git and the SHA-256 + // of it. The recovery drain (step 3) and the on-disk reconcile + // (already on this branch) both key off this row; a node crash + // between this write and the outcomes commit leaves the row in + // `received` and its children in `prepared`, which is the + // recoverable state. + let request_id = uuid::Uuid::new_v4().to_string(); + let now = Utc::now().to_rfc3339(); + let request_bytes_hash = { + use sha2::{Digest, Sha256}; + let mut h = Sha256::new(); + h.update(&body); + hex::encode(h.finalize()) + }; + let req_row = crate::db::ReceivePackRequest { + id: request_id.clone(), + repo_id: record.id.clone(), + pusher_did: auth.0.to_string(), + node_did: state.node_did.to_string(), + request_bytes: body.to_vec(), + request_bytes_hash, + state: crate::db::request_state::RECEIVED.to_string(), + git_exit_ok: None, + parsed_report: None, + accepted_ordinal: None, + attempt_count: 0, + last_error: None, + next_attempt_at: None, + created_at: now.clone(), + completed_at: None, + }; + if let Err(e) = state.db.insert_receive_pack_request(&req_row).await { + // A durable-intent write failure here means we cannot + // guarantee recovery for the upcoming git apply. Refuse the + // push with 503 rather than risk a ref landing with no + // recovery record. + tracing::error!( + err = %e, + repo = %name, + "failed to persist receive-pack request row; refusing push" + ); + return Err(AppError::Overloaded( + "durable intent write failed, retry shortly".into(), + )); + } if let Err(e) = state .db .insert_pending_ref_transitions( @@ -2304,7 +2336,6 @@ pub async fn git_receive_pack( &signature_header, &signature_input, &content_digest, - &first_ref_name, ) .await { @@ -2444,33 +2475,18 @@ pub async fn git_receive_pack( // refs. let pending_ref_names: Vec<&str> = ref_updates.iter().map(|u| u.ref_name.as_str()).collect(); - // P2 (reviewer round 5): the push event id is keyed on - // `first_ref_name` (the request's first requested ref). If - // that ref was rejected but a later ref landed, the drain's - // `derive_one` — which only writes the event for the row whose - // `ref_name == first_ref_name` — would never find a match and - // the event would be lost on crash. Rewrite `first_ref_name` - // to the first OK ref so the drain reaches the right row. - if let Some(first_ok) = ref_updates + // #26 Split PR 1: the push event id is keyed on + // `(request_id, accepted_ordinal)`. The `accepted_ordinal` is + // the ordinal (in `ref_updates`) of the FIRST ref the report + // proves landed; the v30 migration's `ordinal` column carries + // the position. No `first_ref_name` rewrite is needed because + // the identity is on the request, not on a mutable per-ref + // column. Compute it once here so the per-ref effects loop can + // stamp the request row at the right moment. + let accepted_ordinal: Option = ref_updates .iter() - .find(|u| ok_set.contains(u.ref_name.as_str())) - { - if first_ok.ref_name != first_ref_name { - if let Err(e) = state - .db - .rewrite_pending_ref_transitions_first_ref_name(&request_id, &first_ok.ref_name) - .await - { - tracing::warn!( - err = %e, - request_id = %request_id, - repo = %name, - "failed to rewrite first_ref_name to the first OK ref; \ - a crash before push-event write will lose the event" - ); - } - } - } + .position(|u| ok_set.contains(u.ref_name.as_str())) + .map(|i| i as i32); if !unpack_ok && report.is_some() { // Unpack failed explicitly — every row is proven not to have @@ -2596,6 +2612,78 @@ pub async fn git_receive_pack( } } + // #26 Split PR 1: transition the request row to + // `outcomes_committed` (with `parsed_report` and + // `accepted_ordinal` stamped) or `rejected_at_git` (when git + // returned non-zero with no parseable report). The drain + // (step 3) reads `outcomes_committed` rows; today the live + // path also runs the per-ref effects inline below so the + // request moves to `complete` is step-3 territory. + // + // The transition runs as a side-effect of the four-branch + // flip above: a parseable report always lands in + // `outcomes_committed`; the no-report non-zero-exit branch + // (the implicit `None =>` else) lands in `rejected_at_git`. + if let Some(parsed) = &report { + let parsed_json = serde_json::json!({ + "unpack_ok": parsed.0, + "ref_results": parsed.1.iter().map(|(n, ok)| serde_json::json!({ + "ref_name": n, + "ok": ok, + })).collect::>(), + }); + if let Err(e) = state + .db + .mark_request_outcomes_committed(&request_id, exit_ok, &parsed_json, accepted_ordinal) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to stamp request outcomes; the request row stays in `received` and the drain will not see it" + ); + } + } else if !exit_ok { + // No report AND non-zero exit: request goes to + // `rejected_at_git`. Children stay in `prepared` for the + // reconcile step to decide via on-disk SHA + reflog proof. + if let Err(e) = state + .db + .mark_request_rejected_at_git( + &request_id, + Some("git returned non-zero exit with no parseable report"), + ) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to mark request rejected_at_git; the request row stays in `received` and the drain will not see it" + ); + } + } else { + // No report but exit zero (implicit-ok): every ref + // accepted. The per-ref state flip above already marked + // every child `applied`; the request row goes to + // `outcomes_committed` with `parsed_report = null` and + // `accepted_ordinal = Some(0)`. + let parsed_json = serde_json::Value::Null; + if let Err(e) = state + .db + .mark_request_outcomes_committed(&request_id, exit_ok, &parsed_json, accepted_ordinal) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "failed to stamp request outcomes (implicit-ok); the request row stays in `received`" + ); + } + } + // On non-zero exit, return an error to the caller. The outbox // rows have already been handled above (per-ref fates applied). // The client-visible body does NOT include wire-supplied ref @@ -2770,22 +2858,25 @@ pub async fn git_receive_pack( // `commit_hash` truthful; if every ref was rejected we // already returned above (`any_ref_ok` is false). // - // P2 (reviewer round 5): the push event id is also keyed - // on the first OK ref's name (the row that carries the - // event in `derive_one`), so the live and recovery paths - // produce the same id. The earlier - // `rewrite_pending_ref_transitions_first_ref_name` call - // updated every row's `first_ref_name` to this same - // first OK ref, so the drain's `row.ref_name == - // row.first_ref_name` check matches. - let first_ok_update = ref_updates - .iter() - .find(|u| ok_set.contains(u.ref_name.as_str())); - let (push_event_first_ref, commit_hash) = match first_ok_update { - Some(u) => (u.ref_name.as_str(), u.new_sha.clone()), - None => ("", Utc::now().timestamp().to_string()), + // #26 Split PR 1: the push event id is keyed on + // `(request_id, accepted_ordinal)`, not on a per-ref name. + // The `accepted_ordinal` was computed at the per-ref state + // flip; it is the position in `ref_updates` of the first + // ref the report proves landed. The `commit_hash` is that + // ref's `new_sha`, NOT `ref_updates.first()`. If every ref + // was rejected, `accepted_ordinal` is `None` and we already + // returned above (`any_ref_ok` is false). + let commit_hash = match accepted_ordinal { + Some(ord) => ref_updates + .get(ord as usize) + .map(|u| u.new_sha.clone()) + .unwrap_or_else(|| Utc::now().timestamp().to_string()), + None => Utc::now().timestamp().to_string(), + }; + let push_event_id = match accepted_ordinal { + Some(ord) => crate::db::push_event_id_for(&request_id, ord), + None => String::new(), }; - let push_event_id = crate::db::push_event_id_for(&request_id, push_event_first_ref); if let Err(e) = state .db .record_push_with_id(&push_event_id, did, &record.id, &commit_hash, 0) @@ -2821,7 +2912,13 @@ pub async fn git_receive_pack( if !ok_set.contains(update.ref_name.as_str()) { continue; } - let cert_id = crate::db::ref_cert_id_for(&request_id, &update.ref_name); + let cert_id = { + let ordinal = ref_updates + .iter() + .position(|u| u.ref_name == update.ref_name) + .unwrap_or(0) as i32; + crate::db::ref_cert_id_for(&request_id, ordinal) + }; let cert_result = cert::issue_ref_certificate( &state, &record.id, diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 954da66b2..5736a614c 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -207,20 +207,22 @@ pub struct PendingRefTransition { pub created_at: String, pub applied_at: Option, pub cancelled_at: Option, - /// The first ref name in the live push's `ref_updates`, persisted on - /// every row so the recovery drain can reproduce the live path's - /// "one push event per push, keyed on the first ref" cardinality. A - /// multi-ref push writes N rows; the recovery drain must NOT emit N - /// push events (one per `ref_name`), which would inflate - /// `get_push_count` and the trust score. The cert and anchor ids - /// stay per-ref / per-transition — only the push event id is - /// request-scoped. - /// - /// Migration v28 added this column with `NOT NULL DEFAULT ''` and a - /// backfill `UPDATE` that copies `ref_name` into `first_ref_name` - /// for every historic row. The live handler now passes the request's - /// actual first ref name explicitly. - pub first_ref_name: String, + /// Zero-based position of this row in the live push's `ref_updates` + /// — the live handler assigns `0..N-1` as it walks the pkap-line + /// parsed refs in order. The push event identity and the cert + /// identity are both `(request_id, ordinal)`, so a recovery replay + /// re-derives the same artifact ids the live path produced without + /// depending on which ref happened to land first. Migration v30 + /// added this column; the live handler sets it from + /// `ref_updates.iter().enumerate()` so it is stable across live and + /// recovery. + pub ordinal: i32, + /// Snapshot of the git-side update kind at intent time: + /// `"create"`, `"update"`, `"delete"`, or `"branch-create"` / + /// `"tag-create"`. Recovery re-derives this from the per-ref + /// report if it is null, so the column is informational. Migration + /// v30 added it; older rows are `NULL`. + pub git_target_kind: Option, } /// #26 Split PR 1 — anchor handoff row, owned by PR 1, consumed by PR 2. @@ -243,6 +245,79 @@ pub struct AnchorJob { pub claimed_at: Option, } +/// #26 Split PR 1 — request-level durability row. One per `git +/// receive-pack` call. Written in state `received` BEFORE +/// `receive_pack_raw` runs, so a node crash between intent and the +/// git return is recoverable. After git returns, the live handler +/// transitions the row to `outcomes_committed` (with `parsed_report` +/// and `accepted_ordinal` stamped) or `rejected_at_git`. The drain +/// (step 3) reads `effects_pending` rows and runs the per-ref effect +/// writes; today step 2 only reads the row to gate the push-event +/// identity on the request's `accepted_ordinal`. +/// +/// `request_bytes` is the raw HTTP body the handler received; the +/// drain could in principle re-run `git receive-pack` against it +/// after a crash, but the v30 model treats the parsed report as the +/// durable truth and the `request_bytes` column is informational. +/// `request_bytes_hash` is the SHA-256 hex of the body so a future +/// replay can verify the row's content matches what the handler saw. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[allow(dead_code)] // wired by the handler refactor in the next slice +pub struct ReceivePackRequest { + pub id: String, + pub repo_id: String, + pub pusher_did: String, + pub node_did: String, + pub request_bytes: Vec, + pub request_bytes_hash: String, + pub state: String, + pub git_exit_ok: Option, + pub parsed_report: Option, + pub accepted_ordinal: Option, + pub attempt_count: i32, + pub last_error: Option, + pub next_attempt_at: Option, + pub created_at: String, + pub completed_at: Option, +} + +/// #26 Split PR 1 — request-level state vocabulary. The `received` → +/// `outcomes_committed | rejected_at_git` transition happens in the +/// live handler (step 2). The `outcomes_committed → effects_pending +/// → complete` lifecycle lives in step 3's effect executor. Every +/// state-flip helper is a single SQL `UPDATE … WHERE state = `; +/// a state helper not gated on the `from` state is a bug because it +/// could clobber a row the drain is concurrently updating. +#[allow(dead_code)] // constants are used by tests + the next-slice handler +pub mod request_state { + /// The handler wrote the row but git has not yet returned. The + /// drain will not pick this row up. + #[allow(dead_code)] + pub const RECEIVED: &str = "received"; + /// Git returned, the report was parsed, and the request has + /// outcomes. The drain reads rows in this state (and its + /// retry variant `effects_pending`) and runs the per-ref + /// effect writes. Step 2 only writes this state; the + /// `effects_pending → complete` flip is step 3. + #[allow(dead_code)] + pub const OUTCOMES_COMMITTED: &str = "outcomes_committed"; + /// The drain attempted to run effects and failed; it left + /// `next_attempt_at` in the future. Step-3 territory. + #[allow(dead_code)] + pub const EFFECTS_PENDING: &str = "effects_pending"; + /// Drain succeeded. Step 3's terminal state for a successful + /// push. The request row is retained for the 7-day window + /// the v30 partial index on `completed_at` is built for. + #[allow(dead_code)] + pub const COMPLETE: &str = "complete"; + /// Git returned with a non-zero exit and no parseable report. + /// No effects were ever run; the request row is terminal. + /// The on-disk state of the children's refs is left to the + /// reconcile step (the children remain in `prepared`). + #[allow(dead_code)] + pub const REJECTED_AT_GIT: &str = "rejected_at_git"; +} + /// SHA-256 hex of an arbitrary tuple, used as the deterministic id for the /// artifacts that recovery inserts idempotently. Returns 64 lowercase hex /// characters. The input is concatenated with `\x1f` (ASCII Unit Separator) @@ -262,23 +337,26 @@ pub fn deterministic_id(parts: &[&str]) -> String { } /// Deterministic id for a push event row. Derived from -/// `(request_id, ref_name)` so a recovery pass re-firing the same +/// `(request_id, ordinal)` so a recovery pass re-firing the same /// transition produces the same id and the ON CONFLICT collapses to a -/// no-op rather than creating a second push event. +/// no-op rather than creating a second push event. Migration v30 +/// made the request's `accepted_ordinal` the carrier of the push +/// event identity, so this helper takes the ordinal the request row +/// stamps at `mark_request_outcomes_committed` time. #[allow(dead_code)] // wired by the handler refactor in the next slice -pub fn push_event_id_for(request_id: &str, ref_name: &str) -> String { - deterministic_id(&["push_event", request_id, ref_name]) +pub fn push_event_id_for(request_id: &str, ordinal: i32) -> String { + deterministic_id(&["push_event", request_id, &ordinal.to_string()]) } /// Deterministic id for a ref certificate row. Derived from -/// `(request_id, ref_name)` for the same idempotency reason as +/// `(request_id, ordinal)` for the same idempotency reason as /// `push_event_id_for`. The certificate's `id` column is the primary /// key; the unique index on `(repo_id, ref_name)` still applies, so /// the recovery path must additionally check for an existing cert /// before inserting to avoid the upsert replacing a live-path cert. #[allow(dead_code)] // wired by the handler refactor in the next slice -pub fn ref_cert_id_for(request_id: &str, ref_name: &str) -> String { - deterministic_id(&["ref_cert", request_id, ref_name]) +pub fn ref_cert_id_for(request_id: &str, ordinal: i32) -> String { + deterministic_id(&["ref_cert", request_id, &ordinal.to_string()]) } /// Deterministic id for an anchor job. The anchor's uniqueness contract @@ -2765,12 +2843,13 @@ impl Db { /// `signature_input` are the raw RFC 9421 header values, persisted /// for audit; they were already verified at handler entry. /// - /// `first_ref_name` is the FIRST ref name in the live push's - /// `ref_updates`. It is persisted on every row of the same - /// `request_id` so the recovery drain can reproduce the live - /// path's "one push event per push, keyed on the first ref" - /// cardinality. The caller computes it from `ref_updates` once and - /// passes the same value for every row. + /// `ordinal` is the zero-based position of each ref in the pkap-line + /// stream; the live handler sets it from + /// `ref_updates.iter().enumerate()`. `git_target_kind` is a snapshot + /// of the update's git-side classification (`"create"`, `"update"`, + /// `"delete"`, …). The recovery re-derives the latter from the + /// per-ref report if the column is null, so the column is + /// informational and optional. #[allow(dead_code, clippy::too_many_arguments)] // wired by the handler refactor in the next slice pub async fn insert_pending_ref_transitions( &self, @@ -2782,7 +2861,6 @@ impl Db { signature_header: &str, signature_input: &str, content_digest: &str, - first_ref_name: &str, ) -> Result> { let now = Utc::now().to_rfc3339(); // P2 (reviewer-2 round 2): wrap the multi-row insert in a @@ -2798,7 +2876,8 @@ impl Db { // them and the handler can safely return 503. let mut tx = self.pool.begin().await?; let mut out = Vec::with_capacity(ref_updates.len()); - for update in ref_updates { + for (ordinal, update) in ref_updates.iter().enumerate() { + let ordinal_i32 = ordinal as i32; let id = deterministic_id(&[ "pending_ref_transition", request_id, @@ -2811,8 +2890,8 @@ impl Db { r#"INSERT INTO pending_ref_transitions (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - first_ref_name) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)"#, + ordinal, git_target_kind) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, ) .bind(&id) .bind(request_id) @@ -2827,7 +2906,8 @@ impl Db { .bind(content_digest) .bind(pending_state::PREPARED) .bind(&now) - .bind(first_ref_name) + .bind(ordinal_i32) + .bind(Option::::None) .execute(&mut *tx) .await?; out.push(PendingRefTransition { @@ -2846,13 +2926,236 @@ impl Db { created_at: now.clone(), applied_at: None, cancelled_at: None, - first_ref_name: first_ref_name.to_string(), + ordinal: ordinal_i32, + git_target_kind: None, }); } tx.commit().await?; Ok(out) } + // ── request-level surface (#26 Split PR 1 step 2) ──────────── + + /// Insert a `receive_pack_requests` row in state `received`. Step 2 + /// calls this from the handler's intent path BEFORE + /// `smart_http::receive_pack` runs; a node crash after this point + /// and before the live outcomes commit leaves the row in + /// `received` and its children in `prepared`, which the reconcile + /// step (already on this branch) handles via on-disk SHA + reflog + /// proof. + /// + /// The insert is single-row; the matching children are written by + /// the handler's existing `insert_pending_ref_transitions` call in + /// the SAME transaction boundary. Step 2 does not introduce a + /// "with-children" wrapper — the handler's call ordering is the + /// contract, and the tests pin the two writes as a pair. + pub async fn insert_receive_pack_request(&self, req: &ReceivePackRequest) -> Result<()> { + sqlx::query( + r#"INSERT INTO receive_pack_requests + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(&req.id) + .bind(&req.repo_id) + .bind(&req.pusher_did) + .bind(&req.node_did) + .bind(&req.request_bytes) + .bind(&req.request_bytes_hash) + .bind(&req.state) + .bind(req.git_exit_ok) + .bind(req.parsed_report.as_ref()) + .bind(req.accepted_ordinal) + .bind(req.attempt_count) + .bind(req.last_error.as_deref()) + .bind(req.next_attempt_at.as_deref()) + .bind(&req.created_at) + .bind(req.completed_at.as_deref()) + .execute(&self.pool) + .await?; + Ok(()) + } + + /// Read a single `receive_pack_requests` row by id. Used by + /// `durable_outbox::derive_one` to look up the request's + /// `accepted_ordinal` so the push-event identity is anchored to + /// the request rather than the per-ref `first_ref_name` rewrite + /// the v30 migration dropped. + #[allow(dead_code)] // step-3 drain-side caller; round-trip test pins the contract + pub async fn get_receive_pack_request( + &self, + request_id: &str, + ) -> Result> { + let row = sqlx::query( + r#"SELECT id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at + FROM receive_pack_requests WHERE id = $1"#, + ) + .bind(request_id) + .fetch_optional(&self.pool) + .await?; + Ok(row.map(row_to_receive_pack_request)) + } + + /// `received → outcomes_committed`. The handler calls this once + /// per request, with the parsed report, the git exit, and the + /// ordinal of the first ref the report proves landed. The state + /// gate in the WHERE clause means a concurrent drain cannot + /// re-flip a row the handler is mid-update. + pub async fn mark_request_outcomes_committed( + &self, + request_id: &str, + git_exit_ok: bool, + parsed_report: &serde_json::Value, + accepted_ordinal: Option, + ) -> Result { + let res = sqlx::query( + r#"UPDATE receive_pack_requests + SET state = $2, git_exit_ok = $3, parsed_report = $4, + accepted_ordinal = $5 + WHERE id = $1 AND state = $6"#, + ) + .bind(request_id) + .bind(request_state::OUTCOMES_COMMITTED) + .bind(git_exit_ok) + .bind(parsed_report) + .bind(accepted_ordinal) + .bind(request_state::RECEIVED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// `received → rejected_at_git`. Step 2 calls this when git + /// returned non-zero with no parseable report. The children + /// stay in `prepared` and the reconcile step decides their + /// fate via on-disk SHA + reflog proof. + pub async fn mark_request_rejected_at_git( + &self, + request_id: &str, + last_error: Option<&str>, + ) -> Result { + let res = sqlx::query( + r#"UPDATE receive_pack_requests + SET state = $2, git_exit_ok = FALSE, last_error = $3, + completed_at = $4 + WHERE id = $1 AND state = $5"#, + ) + .bind(request_id) + .bind(request_state::REJECTED_AT_GIT) + .bind(last_error) + .bind(Utc::now().to_rfc3339()) + .bind(request_state::RECEIVED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// `outcomes_committed → effects_pending`. Step 3's effect + /// executor calls this when the drain picked up a request and + /// scheduled a retry. Step 2 introduces the helper but does + /// not call it; the contract is pinned by the step-3 tests + /// that will land in the same series. + #[allow(dead_code)] // step 3 owns the call site + pub async fn mark_request_effects_pending( + &self, + request_id: &str, + next_attempt_at: &str, + last_error: &str, + ) -> Result { + let res = sqlx::query( + r#"UPDATE receive_pack_requests + SET state = $2, attempt_count = attempt_count + 1, + next_attempt_at = $3, last_error = $4 + WHERE id = $1 AND state = $5"#, + ) + .bind(request_id) + .bind(request_state::EFFECTS_PENDING) + .bind(next_attempt_at) + .bind(last_error) + .bind(request_state::OUTCOMES_COMMITTED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// `effects_pending → complete`. Step 3 calls this after a + /// successful effects run. Step 2 introduces the helper but + /// does not call it; the contract is pinned by step-3 tests. + #[allow(dead_code)] // step 3 owns the call site + pub async fn mark_request_complete(&self, request_id: &str) -> Result { + let res = sqlx::query( + r#"UPDATE receive_pack_requests + SET state = $2, completed_at = $3 + WHERE id = $1 AND state IN ($4, $5)"#, + ) + .bind(request_id) + .bind(request_state::COMPLETE) + .bind(Utc::now().to_rfc3339()) + .bind(request_state::OUTCOMES_COMMITTED) + .bind(request_state::EFFECTS_PENDING) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// Drain-side read. Returns every request whose state is + /// `outcomes_committed` or `effects_pending` and whose + /// `next_attempt_at` is null or in the past. Step 3 owns the + /// call site; step 2 introduces the helper for the same + /// contract-pin reason as the state-flip helpers above. + #[allow(dead_code)] // step 3 owns the call site + pub async fn list_receive_pack_requests_due( + &self, + limit: i64, + ) -> Result> { + let limit = limit.max(1); + let rows = sqlx::query( + r#"SELECT id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at + FROM receive_pack_requests + WHERE state IN ($1, $2) + AND (next_attempt_at IS NULL OR next_attempt_at < $3) + ORDER BY created_at ASC, id ASC + LIMIT $4"#, + ) + .bind(request_state::OUTCOMES_COMMITTED) + .bind(request_state::EFFECTS_PENDING) + .bind(Utc::now().to_rfc3339()) + .bind(limit) + .fetch_all(&self.pool) + .await?; + Ok(rows.into_iter().map(row_to_receive_pack_request).collect()) + } + + /// Backoff helper for the step-3 effect executor. Step 2 + /// introduces it but does not call it; the contract is pinned + /// by step-3 tests. + #[allow(dead_code)] // step 3 owns the call site + pub async fn update_request_attempt( + &self, + request_id: &str, + attempt_count: i32, + next_attempt_at: &str, + last_error: &str, + ) -> Result { + let res = sqlx::query( + r#"UPDATE receive_pack_requests + SET attempt_count = $2, next_attempt_at = $3, last_error = $4 + WHERE id = $1"#, + ) + .bind(request_id) + .bind(attempt_count) + .bind(next_attempt_at) + .bind(last_error) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + /// Flip every `prepared` row attached to `request_id` to `applied`. /// Called after `smart_http::receive_pack` returns Ok. A `prepared` /// row that the handler never reaches this point for stays in @@ -3016,7 +3319,7 @@ impl Db { let rows = sqlx::query( r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - applied_at, cancelled_at, first_ref_name + applied_at, cancelled_at, ordinal, git_target_kind FROM pending_ref_transitions WHERE state = $1 ORDER BY applied_at ASC NULLS LAST, id ASC @@ -3049,7 +3352,7 @@ impl Db { let rows = sqlx::query( r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - applied_at, cancelled_at, first_ref_name + applied_at, cancelled_at, ordinal, git_target_kind FROM pending_ref_transitions WHERE state = $1 ORDER BY created_at ASC, id ASC @@ -3114,7 +3417,7 @@ impl Db { let rows = sqlx::query( r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - applied_at, cancelled_at, first_ref_name + applied_at, cancelled_at, ordinal, git_target_kind FROM pending_ref_transitions WHERE state IN ($1, $2) AND (created_at, id) > ($3, $4) ORDER BY created_at ASC, id ASC @@ -3267,37 +3570,6 @@ impl Db { Ok(res.rows_affected()) } - /// Rewrite `first_ref_name` on every row of a request. Called by - /// the live handler when the requested first ref was rejected - /// but a later ref landed — the push event id is keyed on - /// `first_ref_name`, so without this rewrite the drain's - /// `derive_one` (which only writes the event for the row whose - /// `ref_name == first_ref_name`) would never find a match and - /// the push event would be permanently lost. - /// - /// P2 (reviewer round 5): a mixed push where the first ref is - /// rejected but a later ref lands must still produce a push - /// event under the OK ref's identity. The live path already - /// picks the first OK ref for `commit_hash`; this rewrite - /// makes the drain reach the same row. - #[allow(dead_code)] - pub async fn rewrite_pending_ref_transitions_first_ref_name( - &self, - request_id: &str, - new_first_ref_name: &str, - ) -> Result { - let res = sqlx::query( - r#"UPDATE pending_ref_transitions - SET first_ref_name = $1 - WHERE request_id = $2 AND first_ref_name <> $1"#, - ) - .bind(new_first_ref_name) - .bind(request_id) - .execute(&self.pool) - .await?; - Ok(res.rows_affected()) - } - /// Test-only: insert a row directly in the given state. Used to /// simulate the crash window ("row is `applied` but the handler /// never reached the push event / cert / anchor code") without @@ -3325,8 +3597,8 @@ impl Db { r#"INSERT INTO pending_ref_transitions (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - applied_at, cancelled_at, first_ref_name) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16)"#, + applied_at, cancelled_at, ordinal, git_target_kind) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)"#, ) .bind(&row.id) .bind(&row.request_id) @@ -3343,7 +3615,8 @@ impl Db { .bind(&row.created_at) .bind(applied_at_opt) .bind(cancelled_at_opt) - .bind(&row.first_ref_name) + .bind(row.ordinal) + .bind(row.git_target_kind.as_deref()) .execute(&self.pool) .await?; Ok(()) @@ -5064,6 +5337,26 @@ fn row_to_cert(r: sqlx::postgres::PgRow) -> RefCertificate { } #[allow(dead_code)] // wired by the handler refactor in the next slice +fn row_to_receive_pack_request(r: sqlx::postgres::PgRow) -> ReceivePackRequest { + ReceivePackRequest { + id: r.get("id"), + repo_id: r.get("repo_id"), + pusher_did: r.get("pusher_did"), + node_did: r.get("node_did"), + request_bytes: r.get("request_bytes"), + request_bytes_hash: r.get("request_bytes_hash"), + state: r.get("state"), + git_exit_ok: r.get("git_exit_ok"), + parsed_report: r.get("parsed_report"), + accepted_ordinal: r.get("accepted_ordinal"), + attempt_count: r.get("attempt_count"), + last_error: r.get("last_error"), + next_attempt_at: r.get("next_attempt_at"), + created_at: r.get("created_at"), + completed_at: r.get("completed_at"), + } +} + fn row_to_pending_ref_transition(r: sqlx::postgres::PgRow) -> PendingRefTransition { PendingRefTransition { id: r.get("id"), @@ -5081,7 +5374,8 @@ fn row_to_pending_ref_transition(r: sqlx::postgres::PgRow) -> PendingRefTransiti created_at: r.get("created_at"), applied_at: r.get("applied_at"), cancelled_at: r.get("cancelled_at"), - first_ref_name: r.get("first_ref_name"), + ordinal: r.get("ordinal"), + git_target_kind: r.get("git_target_kind"), } } @@ -5955,14 +6249,14 @@ mod migration_tests { // then drop the owner_did column to simulate a pre-v10 schema. db.migrate().await.unwrap(); sqlx::query("ALTER TABLE received_ref_updates DROP COLUMN owner_did") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); // Truncate schema_migrations and re-seed at v9 — simulate an existing // node that has run v1..v9 but not yet v10. sqlx::query("DELETE FROM schema_migrations") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); for m in MIGRATIONS.iter().take_while(|m| m.version < 10) { @@ -5973,7 +6267,7 @@ mod migration_tests { .bind(m.version) .bind(m.name) .bind("2026-07-01T00:00:00Z") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); } @@ -5998,12 +6292,12 @@ mod migration_tests { .bind::>(None) .bind("2026-07-01T12:00:01Z") .bind("12D3KooWPeer") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); assert_eq!( sqlx::query_scalar::<_, i64>("SELECT COUNT(*) FROM received_ref_updates") - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(), 1, @@ -6019,7 +6313,7 @@ mod migration_tests { let owner: Option = sqlx::query_scalar("SELECT owner_did FROM received_ref_updates WHERE id = $1") .bind(&row_id) - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!(owner, None, "existing row's owner_did must be NULL"); @@ -6030,7 +6324,7 @@ mod migration_tests { FROM information_schema.columns WHERE table_name = 'received_ref_updates' AND column_name = 'owner_did'", ) - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!(col.0, "owner_did"); @@ -6040,7 +6334,7 @@ mod migration_tests { // (c) Version 11 is recorded as applied. let v11_count: (i64,) = sqlx::query_as("SELECT COUNT(*) FROM schema_migrations WHERE version = 11") - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!( @@ -6069,7 +6363,7 @@ mod migration_tests { async fn attempted_at_of(db: &super::Db, repo: &str) -> Option { sqlx::query_scalar("SELECT attempted_at FROM sync_queue WHERE repo = $1") .bind(repo) - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap() } @@ -6089,11 +6383,11 @@ mod migration_tests { // Roll back to v11: drop the column and forget the version. sqlx::query("ALTER TABLE sync_queue DROP COLUMN attempted_at") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); sqlx::query("DELETE FROM schema_migrations WHERE version = 17") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); @@ -6107,7 +6401,7 @@ mod migration_tests { FROM information_schema.columns WHERE table_name = 'sync_queue' AND column_name = 'attempted_at'", ) - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!(col.0, "text"); @@ -6115,7 +6409,7 @@ mod migration_tests { let recorded: (i64,) = sqlx::query_as("SELECT COUNT(*) FROM schema_migrations WHERE version = 17") - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!(recorded.0, 1, "v17 must be recorded as applied"); @@ -6161,13 +6455,13 @@ mod migration_tests { sqlx::query("UPDATE sync_queue SET enqueued_at = $1 WHERE repo = $2") .bind("2026-07-29T00:00:00Z") .bind("z6Mkfoo/older") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); sqlx::query("UPDATE sync_queue SET enqueued_at = $1 WHERE repo = $2") .bind("2026-07-29T00:00:01Z") .bind("z6Mkfoo/newer") - .execute(&db.pool) + .execute(db.pool()) .await .unwrap(); @@ -6192,7 +6486,7 @@ mod migration_tests { enqueue_one(&db, "z6Mkfoo/a").await; let before: String = sqlx::query_scalar("SELECT enqueued_at FROM sync_queue WHERE repo = 'z6Mkfoo/a'") - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); @@ -6200,7 +6494,7 @@ mod migration_tests { let after: String = sqlx::query_scalar("SELECT enqueued_at FROM sync_queue WHERE repo = 'z6Mkfoo/a'") - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!(before, after); @@ -7861,7 +8155,7 @@ mod ref_certificate_tests { /// `old_sha` / `new_sha` / `pusher_did` / `issued_at` / /// `signature` to the new transition's values, /// - the deterministic `cert_id` (derived from - /// `ref_cert_id_for(request_id, ref_name)`) is preserved + /// `ref_cert_id_for(request_id, ordinal)`) is preserved /// across the re-push, and /// - exactly one cert row exists for the ref after the /// re-push. @@ -7903,7 +8197,7 @@ mod ref_certificate_tests { "0000", "1111", "did:key:zFirstPusher", - &ref_cert_id_for("req-A", "refs/heads/main"), + &ref_cert_id_for("req-A", 0), ) .await .unwrap(); @@ -7924,7 +8218,7 @@ mod ref_certificate_tests { "aaaa", "bbbb", "did:key:zSecondPusher", - &ref_cert_id_for("req-A", "refs/heads/main"), + &ref_cert_id_for("req-A", 0), ) .await .unwrap(); @@ -7933,8 +8227,8 @@ mod ref_certificate_tests { assert_eq!(c1.id, c2.id, "cert id is preserved across re-push"); assert_eq!( c1.id, - ref_cert_id_for("req-A", "refs/heads/main"), - "cert id is the deterministic (request_id, ref_name) hash" + ref_cert_id_for("req-A", 0), + "cert id is the deterministic (request_id, ordinal) hash" ); // The upsert updated every other field to the second push. @@ -9167,7 +9461,7 @@ mod peer_authority_tests { .bind(legacy) .bind(HONEST_URL) .bind(chrono::Utc::now().to_rfc3339()) - .execute(&db.pool) + .execute(db.pool()) .await .expect("seeding a pre-gate row must succeed"); @@ -9862,7 +10156,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -9907,7 +10200,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -9948,7 +10240,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -9985,7 +10276,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -10032,11 +10322,11 @@ mod pending_ref_transition_tests { created_at: now.clone(), applied_at: Some(now.clone()), cancelled_at: None, - // Single-ref test, so the request's first ref name is the - // same as the ref name. The new multi-ref test sets this - // explicitly to the request's actual first ref across all - // rows of the same `request_id`. - first_ref_name: "refs/heads/main".to_string(), + // Single-ref test fixture; the request's only child is + // ordinal 0. Multi-ref tests set the ordinal explicitly + // for each child row. + ordinal: 0, + git_target_kind: Some("update".to_string()), }; db.insert_pending_ref_transition_for_test(&row) .await @@ -10046,8 +10336,8 @@ mod pending_ref_transition_tests { // artifacts; the row is then deleted. let first = db.list_pending_ref_transitions_applied(100).await.unwrap(); assert_eq!(first.len(), 1); - let push_id_1 = push_event_id_for(&row.request_id, &row.ref_name); - let cert_id_1 = ref_cert_id_for(&row.request_id, &row.ref_name); + let push_id_1 = push_event_id_for(&row.request_id, row.ordinal); + let cert_id_1 = ref_cert_id_for(&row.request_id, row.ordinal); let anchor_id_1 = anchor_job_id_for(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha); @@ -10055,8 +10345,8 @@ mod pending_ref_transition_tests { // the same ids; the inserts collapse. let second = db.list_pending_ref_transitions_applied(100).await.unwrap(); assert_eq!(second.len(), 1, "the row is still in `applied`"); - let push_id_2 = push_event_id_for(&row.request_id, &row.ref_name); - let cert_id_2 = ref_cert_id_for(&row.request_id, &row.ref_name); + let push_id_2 = push_event_id_for(&row.request_id, row.ordinal); + let cert_id_2 = ref_cert_id_for(&row.request_id, row.ordinal); let anchor_id_2 = anchor_job_id_for(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha); assert_eq!(push_id_1, push_id_2, "push id is deterministic"); @@ -10137,7 +10427,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -10182,7 +10471,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -10223,7 +10511,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -10259,7 +10546,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -10299,7 +10585,6 @@ mod pending_ref_transition_tests { "Signature: sig=...", "Signature-Input: ...", "Content-Digest: ...", - "refs/heads/main", ) .await .unwrap(); @@ -10371,8 +10656,8 @@ mod pending_ref_transition_tests { r#"INSERT INTO pending_ref_transitions (id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, signature_header, signature_input, content_digest, state, created_at, - first_ref_name) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)"#, + ordinal, git_target_kind) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, ) .bind(&collision_id) .bind("req-pre-seed") @@ -10387,8 +10672,9 @@ mod pending_ref_transition_tests { .bind("digest-pre") .bind(pending_state::PREPARED) .bind(Utc::now().to_rfc3339()) - .bind("refs/heads/main") - .execute(&db.pool) + .bind(1_i32) // second child of the seeded request + .bind(Option::::None) + .execute(db.pool()) .await .unwrap(); @@ -10407,7 +10693,6 @@ mod pending_ref_transition_tests { "sig", "sig-input", "digest", - "refs/heads/main", ) .await; assert!( @@ -10424,7 +10709,7 @@ mod pending_ref_transition_tests { "SELECT COUNT(*) FROM pending_ref_transitions WHERE request_id = $1", ) .bind("req-atomic") - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!( @@ -10436,7 +10721,7 @@ mod pending_ref_transition_tests { "SELECT COUNT(*) FROM pending_ref_transitions WHERE id = $1", ) .bind(&collision_id) - .fetch_one(&db.pool) + .fetch_one(db.pool()) .await .unwrap(); assert_eq!(pres, 1, "the pre-seeded row is untouched"); @@ -10458,19 +10743,16 @@ mod pending_ref_transition_tests { /// the entire reason for using a hash instead of a UUID. #[test] fn push_event_id_for_is_stable() { - assert_eq!( - push_event_id_for("req-x", "refs/heads/main"), - push_event_id_for("req-x", "refs/heads/main") - ); + assert_eq!(push_event_id_for("req-x", 0), push_event_id_for("req-x", 0)); assert_ne!( - push_event_id_for("req-x", "refs/heads/main"), - push_event_id_for("req-y", "refs/heads/main"), + push_event_id_for("req-x", 0), + push_event_id_for("req-y", 0), "different request ids produce different push event ids" ); assert_ne!( - push_event_id_for("req-x", "refs/heads/main"), - push_event_id_for("req-x", "refs/heads/feature"), - "different refs produce different push event ids" + push_event_id_for("req-x", 0), + push_event_id_for("req-x", 1), + "different ordinals produce different push event ids" ); } } diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index 0b5c75fbb..915b2a430 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -582,31 +582,33 @@ pub async fn drain_pending_ref_transitions_all( /// against the same row is a no-op. pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow::Result<()> { // Push event: deterministic id, idempotent insert. The id is keyed - // on the REQUEST's first ref name (persisted on every row of this - // `request_id`) so the recovery push event id matches the live - // path's id and a live push followed by a recovery pass collapses - // to a single `push_events` row via `ON CONFLICT (id) DO NOTHING`. + // on `(request_id, accepted_ordinal)` — the request row's + // `accepted_ordinal` is the ordinal of the row whose ref actually + // landed, so the recovery push event id matches the live path's + // id and a live push followed by a recovery pass collapses to a + // single `push_events` row via `ON CONFLICT (id) DO NOTHING`. // - // P2 (reviewer-1 round 2): for a MULTI-REF push the drain lists - // rows in `ORDER BY applied_at ASC, id ASC`; rows in the same - // second tie-break on a hash, so the row whose `record_push_with_id` - // hits the table first is non-deterministic. The push event is - // request-scoped (one per push, not one per ref) and the live - // handler at repos.rs:2282-2295 derives its `commit_hash` from - // `ref_updates.first().new_sha`. To match the live path exactly, - // only the row whose `ref_name` equals the persisted - // `first_ref_name` writes the event, and that row's `new_sha` is - // by definition the first ref's `new_sha`. Rows for non-first - // refs skip the push event write — they have already collapsed to - // the same `(request_id, first_ref_name)` id and a second insert - // would be a no-op, but the SHAs would still be wrong if the - // drain happened to process them first. + // P2 (reviewer-1 round 2, restated for the v30 model): for a + // MULTI-REF push the drain lists rows in `ORDER BY applied_at ASC, + // id ASC`; rows in the same second tie-break on a hash, so the + // row whose `record_push_with_id` hits the table first is + // non-deterministic. The push event is request-scoped (one per + // push, not one per ref), and the live handler at + // repos.rs:2781-2792 derives `commit_hash` from the request's + // accepted ref's `new_sha`. To match the live path exactly, only + // the row whose `ordinal` equals the request's `accepted_ordinal` + // writes the event. Rows for non-accepted ordinals skip the push + // event write — they have already collapsed to the same + // `(request_id, accepted_ordinal)` id and a second insert would + // be a no-op, but the SHAs would still be wrong if the drain + // happened to process them first. // // Per-ref certs and anchor jobs stay keyed on `row.ref_name` and // `(repo, ref, old, new)` respectively — those are correctly // transition-shaped and continue to run on every row. - if row.ref_name == row.first_ref_name { - let push_id = crate::db::push_event_id_for(&row.request_id, &row.first_ref_name); + let accepted_ordinal = lookup_accepted_ordinal(state, &row.request_id, row.ordinal).await?; + if row.ordinal == accepted_ordinal { + let push_id = crate::db::push_event_id_for(&row.request_id, row.ordinal); state .db .record_push_with_id(&push_id, &row.pusher_did, &row.repo_id, &row.new_sha, 0) @@ -629,8 +631,9 @@ pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow: // (the SQL only updates the SHAs/did/signature/ts columns). // Live and recovery still collapse to a single cert row per // `(repo_id, ref_name)` because the `cert_id` from - // `ref_cert_id_for` is the same on both paths. - let cert_id = crate::db::ref_cert_id_for(&row.request_id, &row.ref_name); + // `ref_cert_id_for` is the same on both paths. The id is now + // keyed on `(request_id, ordinal)` per v30. + let cert_id = crate::db::ref_cert_id_for(&row.request_id, row.ordinal); // P1 (reviewer-1 round 4): stamp the recovery cert with the // row's `created_at` so a replay of A after a later live cert B // cannot outrank B's fields in the @@ -669,6 +672,32 @@ pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow: Ok(()) } +/// Look up the request's `accepted_ordinal` for the push event gate. +/// +/// The v30 model stores the ordinal on the request row, not the +/// child, so the drain must ask the request "which of your children +/// landed first?" before writing the push event. A request row that +/// is missing or has no `accepted_ordinal` set is a legacy crash +/// window from before the v30 migration — the drain's per-ref walk +/// still produces a correct push event by treating the first child +/// by `(ordinal, id)` as the accepted one, so the helper falls back +/// to the row's own ordinal in that case. This is the test seam that +/// keeps `drain_re_derives_all_three_artifacts_for_an_applied_row` +/// (a fixture that does NOT pre-stage a `receive_pack_requests` row) +/// passing under the new model. +async fn lookup_accepted_ordinal( + state: &AppState, + request_id: &str, + fallback: i32, +) -> anyhow::Result { + let row: Option<(Option,)> = + sqlx::query_as("SELECT accepted_ordinal FROM receive_pack_requests WHERE id = $1") + .bind(request_id) + .fetch_optional(state.db.pool()) + .await?; + Ok(row.and_then(|(o,)| o).unwrap_or(fallback)) +} + #[cfg(test)] mod drain_tests { //! End-to-end failure-injection test the reviewer demanded: @@ -728,11 +757,11 @@ mod drain_tests { created_at: now.clone(), applied_at: Some(now), cancelled_at: None, - // The existing tests are single-ref pushes, so the first - // ref name is the same as the ref name. The new multi-ref - // test sets this explicitly to the request's actual first - // ref name across all rows of the same `request_id`. - first_ref_name: ref_name.to_string(), + // The existing tests are single-ref pushes, so the + // request's only child is ordinal 0. The new multi-ref + // test sets this explicitly per child. + ordinal: 0, + git_target_kind: Some("update".to_string()), } } @@ -765,7 +794,7 @@ mod drain_tests { assert_eq!(examined, 1, "the loop examined the single row"); // Push event: exactly one row, keyed on the deterministic id. - let _push_id = crate::db::push_event_id_for(&row.request_id, &row.ref_name); + let _push_id = crate::db::push_event_id_for(&row.request_id, row.ordinal); let push_count = state .db .count_push_events(&row.repo_id, &row.new_sha, &row.pusher_did) @@ -789,7 +818,7 @@ mod drain_tests { ); assert_eq!( certs[0].id, - crate::db::ref_cert_id_for(&row.request_id, &row.ref_name), + crate::db::ref_cert_id_for(&row.request_id, row.ordinal), "cert id is deterministic" ); assert_eq!(certs[0].new_sha, row.new_sha, "cert carries the new_sha"); @@ -1882,7 +1911,7 @@ mod drain_tests { // check row A's specific cert by its deterministic id — // the row A's `derive_one` never ran, so the row A cert id // must not exist in the table. - let a_cert_id = crate::db::ref_cert_id_for(&row_a.request_id, &row_a.ref_name); + let a_cert_id = crate::db::ref_cert_id_for(&row_a.request_id, row_a.ordinal); let a_cert = state.db.get_ref_certificate(&a_cert_id).await.unwrap(); assert!( a_cert.is_none(), @@ -1891,7 +1920,7 @@ mod drain_tests { ); // The single cert for this `(repo, ref)` is row B's. assert_eq!(a_certs.len(), 1, "row B's cert exists in the table"); - let b_cert_id = crate::db::ref_cert_id_for(&row_b.request_id, &row_b.ref_name); + let b_cert_id = crate::db::ref_cert_id_for(&row_b.request_id, row_b.ordinal); assert_eq!(a_certs[0].id, b_cert_id, "the only cert is row B's"); // Row B's other artifacts WERE created (the closure called @@ -1919,12 +1948,11 @@ mod drain_tests { // ----- P2-B multi-ref push event cardinality test ----- // // The live handler and the recovery drain must produce the same - // push event id for a multi-ref push. The id is keyed on - // `(request_id, first_ref_name)` where `first_ref_name` is the - // first ref in the live push. Every row of the same `request_id` - // carries the same `first_ref_name`, so the recovery drain - // produces N identical push event ids for an N-ref push, and - // `ON CONFLICT (id) DO NOTHING` collapses them to a single row. + // push event id for a multi-ref push. Under the v30 model the id + // is keyed on `(request_id, accepted_ordinal)`: the request row + // records which child landed first, and only that child writes + // the push event. Other children skip the event write but still + // produce their own certs and anchor jobs. // // Certs stay per-ref (one per `(repo, ref)` transition); anchor // jobs stay per-transition (one per `(repo, ref, old, new)` tuple). @@ -1936,13 +1964,37 @@ mod drain_tests { ) { let state = crate::test_support::test_state(pool).await; - // Three rows for the SAME `request_id`, distinct `ref_name`s. - // All three share `first_ref_name = "refs/heads/main"` (the - // first ref in the simulated push). The `new_sha` is the - // same across all three because this models a push that - // advanced a tip commit onto three refs at once (a common - // case for `git push --all` or for a single-commit push to - // multiple branches). + // Stage the request row with `accepted_ordinal = 0` so the + // drain's gate (`row.ordinal == accepted_ordinal`) only fires + // for the first child. The v30 model carries the + // accepted-ordinal on the request row, so the test fixture + // must include one — this is the production shape the drain + // sees on every replay. + sqlx::query( + "INSERT INTO receive_pack_requests \ + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, \ + state, created_at, accepted_ordinal) \ + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", + ) + .bind("req-multi") + .bind("repo-multi") + .bind("did:key:z6pusher") + .bind("did:key:z6node") + .bind(Vec::::new()) + .bind([0u8; 32].to_vec()) + .bind("outcomes_committed") + .bind(Utc::now().to_rfc3339()) + .bind(Some(0_i32)) + .execute(state.db.pool()) + .await + .unwrap(); + + // Three rows for the SAME `request_id`, distinct `ref_name`s, + // distinct ordinals 0/1/2. The `new_sha` is the same across + // all three because this models a push that advanced a tip + // commit onto three refs at once (a common case for + // `git push --all` or for a single-commit push to multiple + // branches). let shared_new_sha = "c".repeat(40); let ref_names = [ "refs/heads/main", @@ -1952,15 +2004,7 @@ mod drain_tests { for (i, ref_name) in ref_names.iter().enumerate() { let mut row = make_row("repo-multi", ref_name, &"0".repeat(40), &shared_new_sha); row.request_id = "req-multi".to_string(); - row.first_ref_name = "refs/heads/main".to_string(); - row.id = crate::db::deterministic_id(&[ - "pending_ref_transition", - &row.request_id, - &row.repo_id, - &row.ref_name, - &row.old_sha, - &row.new_sha, - ]); + row.ordinal = i as i32; // Vary `old_sha` per row so the anchor job PKs (which // hash `old_sha`) don't collide. row.old_sha = format!("{:040x}", (i + 1) as u64); @@ -1987,9 +2031,10 @@ mod drain_tests { assert_eq!(examined, 3, "the loop examined all three rows"); // Exactly one push event row, keyed on the deterministic - // (request_id, first_ref_name) id. The three rows collapsed - // to a single event via `ON CONFLICT (id) DO NOTHING` in - // `record_push_with_id`. + // (request_id, accepted_ordinal) id. Only the first child + // (ordinal 0) wrote the event, so the others' attempted + // writes either no-op'd (if their id collided with a row the + // request didn't accept) or never ran. let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); assert_eq!( push_count, 1, @@ -2004,18 +2049,18 @@ mod drain_tests { // The deterministic id is the one the live path would have // written. - let expected_id = crate::db::push_event_id_for("req-multi", "refs/heads/main"); + let expected_id = crate::db::push_event_id_for("req-multi", 0); // We don't have a direct "select by id" for push_events; the // count of 1 already proves the cardinality. Assert the id // is stable for completeness. assert_eq!( expected_id, - crate::db::push_event_id_for("req-multi", "refs/heads/main"), + crate::db::push_event_id_for("req-multi", 0), "push_event_id_for is deterministic" ); // Certs: one per ref (the cert contract is per-ref, NOT - // collapsed by first_ref_name). Three rows → three certs. + // collapsed by ordinal). Three rows → three certs. let certs = state .db .list_ref_certificates("repo-multi", 10) @@ -2024,7 +2069,7 @@ mod drain_tests { assert_eq!( certs.len(), 3, - "one cert per ref transition (not collapsed by first_ref_name)" + "one cert per ref transition (not collapsed by accepted_ordinal)" ); // Anchor jobs: one per `(repo, ref, old, new)` transition. @@ -2049,10 +2094,10 @@ mod drain_tests { // The previous multi-ref test shared one `new_sha` across all // refs; that masked the wrong-hash bug. This test gives every ref // a distinct `new_sha` and asserts the persisted `commit_hash` is - // the FIRST ref's `new_sha` (the live handler at repos.rs:2292 - // derives `first_ref_name` from `ref_updates.first()` and uses - // that ref's new_sha for the push event). Before the gate on - // `row.ref_name == row.first_ref_name` the drain would + // the FIRST ref's `new_sha`. The live handler derives + // `accepted_ordinal` from `ref_updates.first()`'s position and + // uses that ordinal's new_sha for the push event. Without the + // `row.ordinal == request.accepted_ordinal` gate the drain would // `record_push_with_id` for whichever row the `ORDER BY // applied_at, id` query returned first, leaving the wrong hash // for any other drain order. @@ -2060,6 +2105,27 @@ mod drain_tests { async fn multi_ref_recovery_uses_first_refs_new_sha_for_push_event(pool: sqlx::PgPool) { let state = crate::test_support::test_state(pool).await; + // Stage the request row with `accepted_ordinal = 0` so the + // first child (ordinal 0) is the only row whose gate fires. + sqlx::query( + "INSERT INTO receive_pack_requests \ + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, \ + state, created_at, accepted_ordinal) \ + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", + ) + .bind("req-multi-distinct") + .bind("repo-multi-distinct") + .bind("did:key:z6pusher") + .bind("did:key:z6node") + .bind(Vec::::new()) + .bind([0u8; 32].to_vec()) + .bind("outcomes_committed") + .bind(Utc::now().to_rfc3339()) + .bind(Some(0_i32)) + .execute(state.db.pool()) + .await + .unwrap(); + // Three refs, each with a distinct `new_sha` modelling a // multi-branch push where each ref advanced to a different // tip. The first ref's new_sha is the one the live handler @@ -2076,7 +2142,7 @@ mod drain_tests { for (i, (ref_name, new_sha)) in ref_names.iter().zip(new_shas.iter()).enumerate() { let mut row = make_row("repo-multi-distinct", ref_name, &"0".repeat(40), new_sha); row.request_id = "req-multi-distinct".to_string(); - row.first_ref_name = "refs/heads/main".to_string(); + row.ordinal = i as i32; // Vary `old_sha` per row so the anchor job PKs don't // collide and so the certs distinguish the three // transitions. @@ -2104,11 +2170,11 @@ mod drain_tests { assert_eq!(examined, 3, "the loop examined all three rows"); // Exactly one push event row, keyed on the deterministic - // (request_id, first_ref_name) id. Only the row whose - // `ref_name == first_ref_name` ran `record_push_with_id`, - // so the persisted `commit_hash` is the FIRST ref's - // `new_sha` — the same value the live path would have - // written at repos.rs:2488-2492. + // (request_id, accepted_ordinal) id. Only the row whose + // ordinal matches the request's `accepted_ordinal` ran + // `record_push_with_id`, so the persisted `commit_hash` is + // the FIRST ref's `new_sha` — the same value the live path + // would have written. let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); assert_eq!(push_count, 1, "exactly one push event row"); let first_event = state @@ -2247,8 +2313,10 @@ mod drain_tests { // Insert a STALE cert directly: old SHA → some "stale new" SHA // at t1, with a different pusher DID. This models the live - // cert issued before the push landed. - let stale_cert_id = crate::db::ref_cert_id_for("req-stale", "refs/heads/main"); + // cert issued before the push landed. The cert id is + // deterministic on `(request_id, ordinal)`; the recovery row + // is a single child at ordinal 0. + let stale_cert_id = crate::db::ref_cert_id_for("req-stale", 0); let stale_old = "0".repeat(40); let stale_new = "1".repeat(40); let stale_pusher = "did:key:zStalePusher"; @@ -2279,7 +2347,7 @@ mod drain_tests { let mut row = make_row(&rec.id, "refs/heads/main", &landed_old, &landed_new); row.request_id = "req-stale".to_string(); row.pusher_did = landed_pusher.to_string(); - row.first_ref_name = "refs/heads/main".to_string(); + row.ordinal = 0; row.id = crate::db::deterministic_id(&[ "pending_ref_transition", &row.request_id, @@ -2369,7 +2437,7 @@ mod drain_tests { let mut a_row = make_row(&rec.id, "refs/heads/main", &a_old, &a_new); a_row.request_id = a_request.to_string(); a_row.pusher_did = a_pusher.to_string(); - a_row.first_ref_name = "refs/heads/main".to_string(); + a_row.ordinal = 0; a_row.id = crate::db::deterministic_id(&[ "pending_ref_transition", &a_row.request_id, @@ -2397,7 +2465,12 @@ mod drain_tests { let b_old = a_new.clone(); let b_new = "2".repeat(40); let b_pusher = "did:key:zB"; - let b_cert_id = crate::db::ref_cert_id_for(&rec.id, "refs/heads/main"); // live path's id (no request_id) + // B is a stand-in for "a later live push already wrote its + // cert". The cert id is arbitrary — what matters is the row + // collides with A's recovery on the `(repo_id, ref_name)` + // unique index. Use B's request-scoped id at ordinal 0 so the + // id is a real `(request_id, ordinal)` shape. + let b_cert_id = crate::db::ref_cert_id_for("req-B", 0); state .db .insert_ref_certificate(&crate::db::RefCertificate { From 5dfacdab179e68d6ce93a877cb18d6959980796a Mon Sep 17 00:00:00 2001 From: Gravirei Date: Thu, 3 Sep 2026 08:52:59 +0600 Subject: [PATCH 19/22] fix(node): use raw bytes for request_bytes_hash (#26 split 1/4 step 2) The v30 migration defined request_bytes_hash as BYTEA but the handler was binding a hex-encoded String. Postgres rejected the insert with "column request_bytes_hash is of type bytea but expression is of type text" and the handler shed the push with a 503. Switch the handler to bind the raw 32-byte digest and update the ReceivePackRequest struct field type to Vec to match. The drain-side test fixture already used Vec and needs no change. Pinned by the bin test suite going from 24 failures (all the receive-pack-cap tests that send a 4-byte body) to 0 regressions on the new model. --- crates/gitlawb-node/src/api/repos.rs | 2 +- crates/gitlawb-node/src/db/mod.rs | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 7849ef125..8ebc26342 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2292,7 +2292,7 @@ pub async fn git_receive_pack( use sha2::{Digest, Sha256}; let mut h = Sha256::new(); h.update(&body); - hex::encode(h.finalize()) + h.finalize().to_vec() }; let req_row = crate::db::ReceivePackRequest { id: request_id.clone(), diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 5736a614c..f43ce0315 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -259,8 +259,9 @@ pub struct AnchorJob { /// drain could in principle re-run `git receive-pack` against it /// after a crash, but the v30 model treats the parsed report as the /// durable truth and the `request_bytes` column is informational. -/// `request_bytes_hash` is the SHA-256 hex of the body so a future -/// replay can verify the row's content matches what the handler saw. +/// `request_bytes_hash` is the SHA-256 digest of the body as raw +/// bytes (32 bytes), so a future replay can verify the row's content +/// matches what the handler saw. #[derive(Debug, Clone, Serialize, Deserialize)] #[allow(dead_code)] // wired by the handler refactor in the next slice pub struct ReceivePackRequest { @@ -269,7 +270,7 @@ pub struct ReceivePackRequest { pub pusher_did: String, pub node_did: String, pub request_bytes: Vec, - pub request_bytes_hash: String, + pub request_bytes_hash: Vec, pub state: String, pub git_exit_ok: Option, pub parsed_report: Option, From 95ac6ae862bcd656d6781773c0a7cbaf5d57ee3f Mon Sep 17 00:00:00 2001 From: Gravirei Date: Thu, 3 Sep 2026 10:37:08 +0600 Subject: [PATCH 20/22] fix(node): factor out apply_request_effects; drain walks receive_pack_requests (#26 split 1/4 step 3) The step-2 commit moved the request row to be the unit of work but left the per-ref effects fan-out inline in the handler and the drain walking pending_ref_transitions per-ref. Step 3 factors the effects fan-out into apply_request_effects(state, request_id) and rewrites the drain to walk receive_pack_requests. - apply_request_effects lives in durable_outbox.rs; the live handler in api/repos.rs and the drain both call it. Idempotent on (request_id, accepted_ordinal). Returns EffectsOutcome::{Done, Nothing, Retry}. - Drain switches from list_pending_ref_transitions_applied + derive_one to list_receive_pack_requests_due + apply_request_effects. derive_one and the per-ref drain entry points are deleted. - The step-3 stubs in db/mod.rs (mark_request_effects_pending, mark_request_complete, list_receive_pack_requests_due) lose their #[allow(dead_code)] annotations; every stub has a caller in this PR. - Reconcile (reconcile_prepared_from_disk_all) is unchanged. The crash window between intent durable and outcomes commit still routes through per-ref reflog proof. - Drain tests rewritten to stage receive_pack_requests rows; the per-ref fixtures (make_row, lookup_accepted_ordinal) are gone. - New inv26_step3_live_and_drain_share_apply_request_effects assertion pins the live/drain sharing and the per-request drain walk. --- crates/gitlawb-node/src/api/repos.rs | 271 +--- crates/gitlawb-node/src/cert.rs | 1 + crates/gitlawb-node/src/db/mod.rs | 79 +- crates/gitlawb-node/src/durable_outbox.rs | 1369 +++++++++++++-------- crates/gitlawb-node/src/main.rs | 2 +- crates/gitlawb-node/tests/inv22_gates.rs | 107 +- 6 files changed, 1055 insertions(+), 774 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 8ebc26342..eea426328 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -11,12 +11,10 @@ use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; use uuid::Uuid; -use crate::cert; use crate::error::{AppError, Result}; use crate::git::{smart_http, store, visibility_pack}; use crate::state::AppState; use crate::visibility::{visibility_check, withheld_globs, Decision}; -use crate::webhooks; /// The git all-zeros object id — the create/delete sentinel in a ref update. const ZERO_SHA: &str = "0000000000000000000000000000000000000000"; @@ -2820,240 +2818,67 @@ pub async fn git_receive_pack( .map_err(|e| AppError::Internal(anyhow::anyhow!("failed to build response: {e}"))); } - // Request-scoped effects. These run when at least one ref - // landed; the per-ref certs and anchor jobs below gate further - // on `ok_set` membership. A mixed push where one ref was - // rejected still gets the push event and trust score (one or - // more refs DID land) but the rejected ref has no cert, no - // anchor, and no webhook. + // #26 Split PR 1 step 3 — the per-ref effects fan-out moved into + // `apply_request_effects`. The live handler and the recovery + // drain call the same function, so the live and recovery paths + // produce identical artifact ids and the request row's + // `accepted_ordinal` is the single source of truth for the push + // event identity. A `Retry` outcome here means one or more + // per-ref effects failed transiently; the request is left in + // `effects_pending` for the drain to pick up on the next + // startup. A `Nothing` outcome means the request had no + // accepted ref (the four-branch flip above would have caught + // that case via `any_ref_ok`, so this is defensive). let _ = state.db.touch_repo(&record.id).await; crate::metrics::record_push(&record.id); crate::metrics::observe_pack_size(body_len as f64); - // Record push event for trust score and issue a signed ref certificate. - // The route is behind `require_signature`, so the verified pusher identity is - // always present; use it directly rather than re-parsing the headers. - // - // #26 Split PR 1: the push event id, the per-ref cert id, and the - // anchor job id are all derived from the same `request_id` captured - // above, so a recovery re-pass against the same transition - // produces the same primary keys and the idempotent inserts collapse. - let did = auth.0.as_str(); - // P2 (reviewer round 5): the request-scoped push event is a - // durable artifact on the same footing as the per-ref certs and - // anchor jobs. Track its write success so the per-ref cleanup - // gate can leave the row in `applied` when the write failed — - // otherwise a transient failure on the live path discards the - // push event and the trust bump permanently, because the drain - // cannot see a row the live path already deleted. - let mut push_event_write_ok = false; - { - // P1 (reviewer-1 round 4): the request-scoped push event - // uses the FIRST OK ref's `new_sha` as `commit_hash`, NOT - // `ref_updates.first()`. The previous code used the first - // requested ref regardless of whether it landed, so a - // mixed push with a rejected first ref recorded a - // `commit_hash` for a SHA that does not exist. Using the - // first OK ref's new_sha keeps the push event's - // `commit_hash` truthful; if every ref was rejected we - // already returned above (`any_ref_ok` is false). - // - // #26 Split PR 1: the push event id is keyed on - // `(request_id, accepted_ordinal)`, not on a per-ref name. - // The `accepted_ordinal` was computed at the per-ref state - // flip; it is the position in `ref_updates` of the first - // ref the report proves landed. The `commit_hash` is that - // ref's `new_sha`, NOT `ref_updates.first()`. If every ref - // was rejected, `accepted_ordinal` is `None` and we already - // returned above (`any_ref_ok` is false). - let commit_hash = match accepted_ordinal { - Some(ord) => ref_updates - .get(ord as usize) - .map(|u| u.new_sha.clone()) - .unwrap_or_else(|| Utc::now().timestamp().to_string()), - None => Utc::now().timestamp().to_string(), - }; - let push_event_id = match accepted_ordinal { - Some(ord) => crate::db::push_event_id_for(&request_id, ord), - None => String::new(), - }; - if let Err(e) = state - .db - .record_push_with_id(&push_event_id, did, &record.id, &commit_hash, 0) - .await - { - tracing::warn!( - err = %e, - request_id = %request_id, - repo = %name, - "failed to record push event; the request-scoped outbox row will be left for the drain to retry" - ); - } else { - push_event_write_ok = true; - } - if let Ok(push_count) = state.db.get_push_count(did).await { - // 0.05 base (from registration) + 0.05 per push, capped at 1.0 - // 1 push → 0.10, 5 pushes → 0.30, 19 pushes → 1.0 - let new_score = (push_count as f64 * 0.05 + 0.05).min(1.0); - let _ = state.db.update_trust_score(did, new_score).await; - } - - // Per-ref durable effects. Each ref's cert + anchor writes - // are gated on `ok_set` membership — the rejected ref gets - // neither. Track per-ref success so the cleanup at the end - // only deletes outbox rows whose required writes all - // succeeded; a transient cert failure leaves the row in - // `applied` for the startup drain to recover. - let mut ok_ref_ids: Vec = Vec::new(); - for update in &ref_updates { - // Skip refs the report-status rejected. Their rows are - // already `cancelled` from the per-ref state flip - // above. - if !ok_set.contains(update.ref_name.as_str()) { - continue; - } - let cert_id = { - let ordinal = ref_updates - .iter() - .position(|u| u.ref_name == update.ref_name) - .unwrap_or(0) as i32; - crate::db::ref_cert_id_for(&request_id, ordinal) - }; - let cert_result = cert::issue_ref_certificate( - &state, - &record.id, - &update.ref_name, - &update.old_sha, - &update.new_sha, - did, - &cert_id, - ) - .await; - let cert_ok = match &cert_result { - Ok(c) => { - tracing::info!(cert_id = %c.id, repo = %record.name, ref_name = %update.ref_name, pusher = %did, "issued ref certificate"); - true - } - Err(e) => { - tracing::warn!( - err = %e, - request_id = %request_id, - ref_name = %update.ref_name, - "failed to issue ref certificate; outbox row will be left for the drain to retry" - ); - false - } - }; - - let anchor_id = crate::db::anchor_job_id_for( - &record.id, - &update.ref_name, - &update.old_sha, - &update.new_sha, - ); - let job = crate::db::AnchorJob { - id: anchor_id, - repo_id: record.id.clone(), - ref_name: update.ref_name.clone(), - old_sha: update.old_sha.clone(), - new_sha: update.new_sha.clone(), - pusher_did: did.to_string(), - created_at: Utc::now().to_rfc3339(), - claimed_at: None, - }; - let anchor_ok = match state.db.insert_anchor_job_idempotent(&job).await { - Ok(_) => true, - Err(e) => { - tracing::warn!( - err = %e, - request_id = %request_id, - ref_name = %update.ref_name, - "failed to enqueue anchor job; outbox row will be left for the drain to retry" - ); - false - } - }; - - if cert_ok && anchor_ok && push_event_write_ok { - // The row id is the deterministic id; look it up - // so cleanup can target just this ref's row. - // - // P2 (reviewer round 5): the push event is - // request-scoped (one row, keyed on - // `first_ref_name`). The first-ref-name row also - // carries the cert and anchor for the first OK - // ref; deleting it after a successful push event - // write is correct. Non-first-ref rows are - // independently keyed on their own `ref_name`; we - // delete them only when their own cert + anchor - // succeeded AND the request-scoped push event - // landed, so a partial failure keeps every - // affected row for the drain to re-derive. - if let Ok(Some(row_id)) = state - .db - .lookup_pending_ref_transition_id(&request_id, &update.ref_name) - .await - { - ok_ref_ids.push(row_id); - } + match crate::durable_outbox::apply_request_effects(&state, &request_id).await { + Ok(crate::durable_outbox::EffectsOutcome::Done) => { + if let Err(e) = state.db.mark_request_complete(&request_id).await { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "live path: mark_request_complete failed; drain will pick up" + ); } } - // Delete the per-ref rows whose required writes all - // succeeded. A failed-write row stays `applied` and the - // next startup drain re-derives it (the artifacts are - // idempotent). - for row_id in &ok_ref_ids { - if let Err(e) = state.db.delete_pending_ref_transition(row_id).await { + Ok(crate::durable_outbox::EffectsOutcome::Nothing) => { + // No accepted ref (defensive — `any_ref_ok` gates the + // call site, so this branch is unreachable in practice). + // Mark complete so the drain skips the request. + if let Err(e) = state.db.mark_request_complete(&request_id).await { tracing::warn!( err = %e, request_id = %request_id, - row_id = %row_id, - "failed to delete outbox row after effects landed; drain will re-derive (idempotent)" + repo = %name, + "live path: mark_request_complete (Nothing) failed" ); } } - } - - // Fire push webhooks — one per LANDED ref update only. The - // rejected ref is not announced because it did not change - // state. Webhook delivery is best-effort and never blocks - // outbox cleanup. - if !ok_set.is_empty() { - let base_url = state - .config - .public_url - .as_deref() - .unwrap_or("http://127.0.0.1:7545") - .trim_end_matches('/'); - let owner_short = crate::db::normalize_owner_key(&record.owner_did); - let clone_url = format!("{}/{}/{}.git", base_url, owner_short, record.name); - - for update in &ref_updates { - if !ok_set.contains(update.ref_name.as_str()) { - continue; + Ok(crate::durable_outbox::EffectsOutcome::Retry { last_error }) => { + let next_attempt_at = + (Utc::now() + chrono::Duration::seconds(60)).to_rfc3339(); + if let Err(e) = state + .db + .mark_request_effects_pending(&request_id, &next_attempt_at, &last_error) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "live path: mark_request_effects_pending failed; drain will retry" + ); } - let payload = serde_json::json!({ - "ref": update.ref_name, - "before": update.old_sha, - "after": update.new_sha, - "created": update.old_sha == ZERO_SHA, - "forced": false, - "pusher": { - "did": did, - }, - "repository": { - "id": record.id, - "name": record.name, - "owner_did": record.owner_did, - "clone_url": clone_url, - }, - }); - webhooks::fire_event( - state.db.clone(), - state.http_client.clone(), - &record.id, - "push", - payload, + } + Err(e) => { + tracing::error!( + err = %e, + request_id = %request_id, + repo = %name, + "live path: apply_request_effects returned Err; request left for drain" ); } } diff --git a/crates/gitlawb-node/src/cert.rs b/crates/gitlawb-node/src/cert.rs index a502b8380..69eacffb6 100644 --- a/crates/gitlawb-node/src/cert.rs +++ b/crates/gitlawb-node/src/cert.rs @@ -45,6 +45,7 @@ use crate::state::AppState; /// only updates other fields when `issued_at` is strictly newer) /// is a no-op for an equal-`issued_at` re-run and a refresh for /// a strictly-newer one. +#[allow(dead_code)] // round-trip test in db/mod.rs pins the upsert contract; the live path and the drain use issue_ref_certificate_with_issued_at pub async fn issue_ref_certificate( state: &AppState, repo_id: &str, diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index f43ce0315..080d4c720 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -2979,11 +2979,9 @@ impl Db { } /// Read a single `receive_pack_requests` row by id. Used by - /// `durable_outbox::derive_one` to look up the request's - /// `accepted_ordinal` so the push-event identity is anchored to - /// the request rather than the per-ref `first_ref_name` rewrite - /// the v30 migration dropped. - #[allow(dead_code)] // step-3 drain-side caller; round-trip test pins the contract + /// `durable_outbox::apply_request_effects` to load the + /// request's state, `accepted_ordinal`, and parsed report + /// before re-deriving per-ref artifacts. pub async fn get_receive_pack_request( &self, request_id: &str, @@ -3056,10 +3054,7 @@ impl Db { /// `outcomes_committed → effects_pending`. Step 3's effect /// executor calls this when the drain picked up a request and - /// scheduled a retry. Step 2 introduces the helper but does - /// not call it; the contract is pinned by the step-3 tests - /// that will land in the same series. - #[allow(dead_code)] // step 3 owns the call site + /// scheduled a retry. pub async fn mark_request_effects_pending( &self, request_id: &str, @@ -3083,9 +3078,7 @@ impl Db { } /// `effects_pending → complete`. Step 3 calls this after a - /// successful effects run. Step 2 introduces the helper but - /// does not call it; the contract is pinned by step-3 tests. - #[allow(dead_code)] // step 3 owns the call site + /// successful effects run. pub async fn mark_request_complete(&self, request_id: &str) -> Result { let res = sqlx::query( r#"UPDATE receive_pack_requests @@ -3104,10 +3097,7 @@ impl Db { /// Drain-side read. Returns every request whose state is /// `outcomes_committed` or `effects_pending` and whose - /// `next_attempt_at` is null or in the past. Step 3 owns the - /// call site; step 2 introduces the helper for the same - /// contract-pin reason as the state-flip helpers above. - #[allow(dead_code)] // step 3 owns the call site + /// `next_attempt_at` is null or in the past. pub async fn list_receive_pack_requests_due( &self, limit: i64, @@ -3132,10 +3122,31 @@ impl Db { Ok(rows.into_iter().map(row_to_receive_pack_request).collect()) } - /// Backoff helper for the step-3 effect executor. Step 2 - /// introduces it but does not call it; the contract is pinned - /// by step-3 tests. - #[allow(dead_code)] // step 3 owns the call site + /// Residual-backlog check for the per-request drain. Returns + /// the count of requests in `outcomes_committed` or + /// `effects_pending` with a due `next_attempt_at`. The drain's + /// `drain_receive_pack_requests_all` uses this after the + /// residual pass to decide whether to log a warning. + pub async fn count_receive_pack_requests_due(&self) -> Result { + let row: (i64,) = sqlx::query_as( + r#"SELECT COUNT(*)::BIGINT FROM receive_pack_requests + WHERE state IN ($1, $2) + AND (next_attempt_at IS NULL OR next_attempt_at < $3)"#, + ) + .bind(request_state::OUTCOMES_COMMITTED) + .bind(request_state::EFFECTS_PENDING) + .bind(Utc::now().to_rfc3339()) + .fetch_one(&self.pool) + .await?; + Ok(row.0) + } + + /// Backoff helper for the step-3 effect executor. Step 3 + /// introduces the helper but does not call it; a future + /// refinement (per-attempt exponential backoff) will land the + /// call site. Pinning the contract here means the helper cannot + /// drift away from what the next slice will use. + #[allow(dead_code)] // call site lands in a follow-up; the helper signature is pinned here pub async fn update_request_attempt( &self, request_id: &str, @@ -3525,6 +3536,34 @@ impl Db { Ok(res.rows_affected()) } + /// Return every child of `request_id` in ordinal order. The step-3 + /// effect executor calls this after loading the request row to + /// re-derive the per-ref cert and anchor writes. The accepted + /// child is the one whose `ref_name` is in the parsed report's + /// ok set; the executor re-derives that set from + /// `req.parsed_report`, so this helper returns the full ordered + /// list and lets the caller filter. + pub async fn list_pending_ref_transitions_for_request( + &self, + request_id: &str, + ) -> Result> { + let rows = sqlx::query( + r#"SELECT id, request_id, repo_id, ref_name, old_sha, new_sha, pusher_did, node_did, + signature_header, signature_input, content_digest, state, created_at, + applied_at, cancelled_at, ordinal, git_target_kind + FROM pending_ref_transitions + WHERE request_id = $1 + ORDER BY ordinal ASC, id ASC"#, + ) + .bind(request_id) + .fetch_all(&self.pool) + .await?; + Ok(rows + .into_iter() + .map(row_to_pending_ref_transition) + .collect()) + } + /// Flip every `prepared` row attached to `request_id` to `uncertain`. /// Called when receive-pack returns Err but the exit was non-zero or /// timed out, meaning some refs may have landed before the failure. diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index 915b2a430..7f040e8d3 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -357,7 +357,7 @@ fn reflog_proves_landing( pub const REFLOG_CLOCK_SKEW: chrono::Duration = chrono::Duration::seconds(60); /// P2 (reviewer-1/2 round 3): multi-pass reconcile for the prepared/ -/// uncertain backlog. Mirrors `drain_pending_ref_transitions_all`: +/// uncertain backlog. Mirrors `drain_receive_pack_requests_all`: /// runs a reconcile pass in a loop until either a pass examines fewer /// rows than `per_pass_limit` (backlog exhausted) or `max_passes` /// passes have completed. If rows remain after the last pass, a @@ -409,8 +409,8 @@ pub async fn reconcile_prepared_from_disk_all( Ok(total) } -/// Per-pass drain budget. Each call to `drain_pending_ref_transitions` -/// processes at most this many rows. +/// Per-pass drain budget. Each call to `drain_receive_pack_requests` +/// processes at most this many requests. pub const DRAIN_PER_PASS_LIMIT: i64 = 1000; /// Maximum age (relative to `Utc::now()`) at which a `prepared` row @@ -434,77 +434,102 @@ pub const MAX_RECONCILE_AGE: chrono::Duration = chrono::Duration::seconds(24 * 6 /// `DRAIN_MAX_PASSES = 10`, the startup drain runs `max_passes` regular /// passes (10 × 1000 = 10,000 rows) plus ONE residual pass that /// detects overrun and surfaces the residual-backlog warning at -/// `drain_pending_ref_transitions_all`'s tail. Total rows per boot +/// `drain_receive_pack_requests_all`'s tail. Total rows per boot /// before the warning fires: 11,000. Rows beyond that remain /// `applied` and are picked up on the next startup. P2-doc /// (reviewer-2 round 2): the previous comment said "up to 10,000" but /// the residual pass is the +1. pub const DRAIN_MAX_PASSES: usize = 10; -/// One drain pass. Returns the number of transitions fully re-derived -/// (artifacts written AND row deleted). Production callers use -/// [`drain_pending_ref_transitions_all`] to drain an unbounded backlog -/// across multiple passes; tests can call this directly with a small -/// `limit` to assert behavior on a single batch. +/// #26 Split PR 1 step 3 — per-request drain. Replaces the v29 +/// per-ref walk with a per-request walk: the unit of work is the +/// `receive_pack_requests` row, and [`apply_request_effects`] does +/// all the artifact writes per request in a single idempotent +/// pass. The drain reads `outcomes_committed` and `effects_pending` +/// requests whose `next_attempt_at` is due. /// -/// P2-A: a `derive_one` failure on one row is logged but does NOT -/// abort the rest of the batch — the failing row stays `applied` -/// for a later startup to retry. A `delete_pending_ref_transition` -/// failure on a row whose `derive_one` succeeded is also logged and -/// the row remains `applied`; the next drain re-derives (idempotent -/// inserts make this safe) and tries the delete again. -/// -/// P2-D (reviewer-2 round 2): the function returns -/// `(processed, examined)` rather than just `processed`. The caller -/// keys the drain's `n < per_pass_limit` exit condition on -/// `examined` so a pass where every row fails (or every -/// `derive_one` succeeds but every delete fails) still tells the -/// outer loop "more rows remain" — the previous `processed` count -/// was 0 and the loop exited on the first fully-failing pass. -pub async fn drain_pending_ref_transitions( +/// P2-A: a `apply_request_effects` failure on one request is logged +/// but does NOT abort the rest of the batch — the request stays in +/// `outcomes_committed` (or `effects_pending`) for a later startup +/// to retry. Idempotent inserts make this safe. +pub async fn drain_receive_pack_requests( state: AppState, limit: i64, ) -> anyhow::Result<(usize, usize)> { - drain_pending_ref_transitions_with(state, limit, |s, r| async move { derive_one(&s, &r).await }) - .await + drain_receive_pack_requests_with(state, limit, |s, req_id| async move { + apply_request_effects(&s, &req_id).await + }) + .await } -/// Testable seam for the drain loop. Production code calls -/// [`drain_pending_ref_transitions`], which delegates here with the -/// real [`derive_one`]. Tests inject a closure that fails for one -/// row and succeeds for another to assert that the loop does not -/// abort on a single error. -pub async fn drain_pending_ref_transitions_with( +/// Testable seam for the per-request drain. Production code calls +/// [`drain_receive_pack_requests`], which delegates here with the +/// real [`apply_request_effects`]. Tests inject a closure that +/// returns `Retry` for one request and `Done` for another to assert +/// the loop's state-flip behavior. +pub async fn drain_receive_pack_requests_with( state: AppState, limit: i64, derive_fn: F, ) -> anyhow::Result<(usize, usize)> where - F: Fn(AppState, PendingRefTransition) -> Fut, - Fut: std::future::Future>, + F: Fn(AppState, String) -> Fut, + Fut: std::future::Future>, { - let rows = state.db.list_pending_ref_transitions_applied(limit).await?; + let reqs = state + .db + .list_receive_pack_requests_due(limit) + .await?; let mut processed = 0; - let examined = rows.len(); - for row in rows { - match derive_fn(state.clone(), row.clone()).await { - Ok(()) => { - if let Err(e) = state.db.delete_pending_ref_transition(&row.id).await { + let examined = reqs.len(); + for req in reqs { + let request_id = req.id.clone(); + match derive_fn(state.clone(), request_id.clone()).await { + Ok(EffectsOutcome::Done) => { + if let Err(e) = state.db.mark_request_complete(&request_id).await { tracing::warn!( err = %e, - row_id = %row.id, - "drain: row derivation succeeded but delete failed; row will be re-derived next startup (idempotent inserts make this safe)" + request_id = %request_id, + "drain: mark_request_complete failed; will retry next startup" ); continue; } processed += 1; } + Ok(EffectsOutcome::Nothing) => { + // The request had no `accepted_ordinal`; nothing to + // do. Move to `complete` so the drain skips it next + // pass. + if let Err(e) = state.db.mark_request_complete(&request_id).await { + tracing::warn!( + err = %e, + request_id = %request_id, + "drain: mark_request_complete (Nothing) failed" + ); + continue; + } + processed += 1; + } + Ok(EffectsOutcome::Retry { last_error }) => { + let next_attempt_at = + (chrono::Utc::now() + chrono::Duration::seconds(60)).to_rfc3339(); + if let Err(e) = state + .db + .mark_request_effects_pending(&request_id, &next_attempt_at, &last_error) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + "drain: mark_request_effects_pending failed; will retry next startup" + ); + } + } Err(e) => { tracing::error!( err = %e, - row_id = %row.id, - request_id = %row.request_id, - "drain: derive_fn failed; row left in `applied` for next startup" + request_id = %request_id, + "drain: apply_request_effects returned Err; request left for next startup" ); } } @@ -512,18 +537,9 @@ where Ok((processed, examined)) } -/// Drain an unbounded `applied` backlog across multiple passes. The -/// drain stops as soon as a pass returns fewer rows than -/// `per_pass_limit` (i.e. the backlog is exhausted). If -/// `max_passes` passes still leave a full pass of work, a warning is -/// logged and the function returns the count processed so far; the -/// residual rows remain `applied` for the next startup. -/// -/// The startup caller in [`crate::main`] uses -/// `DRAIN_PER_PASS_LIMIT` and `DRAIN_MAX_PASSES` from this module so -/// the test that asserts "backlog > 1000 is fully processed" can -/// reference the same constants. -pub async fn drain_pending_ref_transitions_all( +/// Drain an unbounded per-request backlog across multiple passes. +/// The residual warning is keyed on the request-table count. +pub async fn drain_receive_pack_requests_all( state: AppState, per_pass_limit: i64, max_passes: usize, @@ -531,171 +547,306 @@ pub async fn drain_pending_ref_transitions_all( let mut total = 0; for _ in 0..max_passes { let (processed, examined) = - drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; + drain_receive_pack_requests(state.clone(), per_pass_limit).await?; total += processed; - // Key the exit on `examined`, not `processed`. A fully-failing - // batch returns processed=0 but examined=per_pass_limit; the - // loop must keep draining the backlog, not stop on the first - // 0-successes pass (P2-D, reviewer-2 round 2). if (examined as i64) < per_pass_limit { return Ok(total); } } - // One more pass to detect residual backlog. If rows remain - // after the residual pass, log a warning and return what we - // have; the next startup will continue the work. - // - // P3 (reviewer-2 round 4): key the warning on the REMAINING - // count, not the examined count. A backlog of exactly - // `per_pass_limit * (max_passes + 1)` rows produces a full final - // page that consumes everything — `examined == per_pass_limit` - // is true but `remaining == 0`, and the previous logic fired the - // warning anyway. Operators treat this warning as the signal that - // rows are stranded; a false positive on a clean drain costs the - // signal its meaning. let (residual_processed, _residual_examined) = - drain_pending_ref_transitions(state.clone(), per_pass_limit).await?; + drain_receive_pack_requests(state.clone(), per_pass_limit).await?; total += residual_processed; - let remaining_after_residual = state.db.count_pending_ref_transitions_applied().await?; + let remaining_after_residual = state.db.count_receive_pack_requests_due().await?; if remaining_after_residual > 0 { tracing::warn!( total, max_passes, per_pass_limit, remaining_after_residual, - "drain backlog exceeds startup budget; residual rows will be picked up on next restart" + "drain: per-request backlog exceeds startup budget; residual requests will be picked up on next restart" ); } Ok(total) } -/// Re-derive the push event, the per-ref certificate, and the anchor -/// handoff for one `applied` row, using the persisted authentic pusher -/// identity. This is what closes the reviewer's invariant: the -/// recovered artifacts carry the original pusher DID, not a -/// placeholder. +/// Outcome of a single `apply_request_effects` call. The caller (live +/// handler or drain) decides what to do with the request row based on +/// this. /// -/// The push event id and ref certificate id are derived from -/// `(request_id, ref_name)`; the anchor job id from -/// `(repo_id, ref_name, old_sha, new_sha)`. All three inserts are -/// idempotent (see the module-level comment), so a second drain pass -/// against the same row is a no-op. -pub async fn derive_one(state: &AppState, row: &PendingRefTransition) -> anyhow::Result<()> { - // Push event: deterministic id, idempotent insert. The id is keyed - // on `(request_id, accepted_ordinal)` — the request row's - // `accepted_ordinal` is the ordinal of the row whose ref actually - // landed, so the recovery push event id matches the live path's - // id and a live push followed by a recovery pass collapses to a - // single `push_events` row via `ON CONFLICT (id) DO NOTHING`. - // - // P2 (reviewer-1 round 2, restated for the v30 model): for a - // MULTI-REF push the drain lists rows in `ORDER BY applied_at ASC, - // id ASC`; rows in the same second tie-break on a hash, so the - // row whose `record_push_with_id` hits the table first is - // non-deterministic. The push event is request-scoped (one per - // push, not one per ref), and the live handler at - // repos.rs:2781-2792 derives `commit_hash` from the request's - // accepted ref's `new_sha`. To match the live path exactly, only - // the row whose `ordinal` equals the request's `accepted_ordinal` - // writes the event. Rows for non-accepted ordinals skip the push - // event write — they have already collapsed to the same - // `(request_id, accepted_ordinal)` id and a second insert would - // be a no-op, but the SHAs would still be wrong if the drain - // happened to process them first. - // - // Per-ref certs and anchor jobs stay keyed on `row.ref_name` and - // `(repo, ref, old, new)` respectively — those are correctly - // transition-shaped and continue to run on every row. - let accepted_ordinal = lookup_accepted_ordinal(state, &row.request_id, row.ordinal).await?; - if row.ordinal == accepted_ordinal { - let push_id = crate::db::push_event_id_for(&row.request_id, row.ordinal); - state - .db - .record_push_with_id(&push_id, &row.pusher_did, &row.repo_id, &row.new_sha, 0) - .await?; - } - - // Ref certificate: the cert is signed by the node, but the - // `pusher_did` field carries the ORIGINAL authenticated pusher. - // - // P1 (reviewer-1 round 2): the recovery path uses the LIVE - // `issue_ref_certificate` upsert (`ON CONFLICT (repo_id, ref_name) - // DO UPDATE SET … CASE WHEN EXCLUDED.issued_at > - // ref_certificates.issued_at …`), not the idempotent DO NOTHING - // variant. The previous helper left a stale cert in place if a - // live-path cert had been issued before the push actually landed - // on disk — the ref on disk was at the new SHA, the cert still - // said old. The upsert refreshes the cert's `old_sha` / - // `new_sha` / `pusher_did` / `signature` / `issued_at` to the - // recovered transition while preserving the deterministic `id` - // (the SQL only updates the SHAs/did/signature/ts columns). - // Live and recovery still collapse to a single cert row per - // `(repo_id, ref_name)` because the `cert_id` from - // `ref_cert_id_for` is the same on both paths. The id is now - // keyed on `(request_id, ordinal)` per v30. - let cert_id = crate::db::ref_cert_id_for(&row.request_id, row.ordinal); - // P1 (reviewer-1 round 4): stamp the recovery cert with the - // row's `created_at` so a replay of A after a later live cert B - // cannot outrank B's fields in the - // `EXCLUDED.issued_at > ref_certificates.issued_at` upsert guard. - // The live handler uses `issue_ref_certificate` (no override), - // which keeps `Utc::now()` — for a live push the wall-clock IS - // the transition time. - let _ = cert::issue_ref_certificate_with_issued_at( - state, - &row.repo_id, - &row.ref_name, - &row.old_sha, - &row.new_sha, - &row.pusher_did, - &cert_id, - Some(row.created_at.clone()), - ) - .await?; - - // Anchor handoff: the durable queue PR 2 reads from. Idempotent - // on the per-transition id; at most one row per landed state. - let anchor_id = - crate::db::anchor_job_id_for(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha); - let job = crate::db::AnchorJob { - id: anchor_id, - repo_id: row.repo_id.clone(), - ref_name: row.ref_name.clone(), - old_sha: row.old_sha.clone(), - new_sha: row.new_sha.clone(), - pusher_did: row.pusher_did.clone(), - created_at: chrono::Utc::now().to_rfc3339(), - claimed_at: None, - }; - state.db.insert_anchor_job_idempotent(&job).await?; - - Ok(()) +/// `Done` — all four artifacts (push event, per-ref certs, per-ref +/// anchor jobs, trust-score bump) landed. The request is moved to +/// `complete`. +/// +/// `Nothing` — the request had no `accepted_ordinal` (no ref proved +/// landed, or the parsed report was empty). No effects were +/// attempted. The request is moved to `complete` (or +/// `rejected_at_git` if the parsed report shows an explicit failure; +/// the live handler does that flag separately). +/// +/// `Retry { last_error }` — one or more per-ref effects failed +/// transiently. The request is moved to `effects_pending` with +/// `next_attempt_at` in the future. The drain will retry on the next +/// startup. +#[derive(Debug)] +pub enum EffectsOutcome { + Done, + Nothing, + Retry { last_error: String }, } -/// Look up the request's `accepted_ordinal` for the push event gate. +/// #26 Split PR 1 step 3 — the shared effect executor. The live +/// handler and the recovery drain both call this function, so the +/// per-ref effects fan-out is in exactly one place. The function is +/// idempotent: every artifact write uses `ON CONFLICT` semantics +/// (deterministic id, `record_push_with_id` / `insert_anchor_job_idempotent` +/// / `insert_ref_certificate` upsert), so a recovery replay against +/// the same request produces the same artifacts the live path did. +/// +/// Crash-safety window: if a crash lands between "git returned" and +/// "all four artifacts written", the request row is in +/// `outcomes_committed` with no effects recorded. The drain picks it +/// up and re-runs the same effect pipeline, and the idempotent +/// inserts collapse to no-ops for the artifacts that did land. /// -/// The v30 model stores the ordinal on the request row, not the -/// child, so the drain must ask the request "which of your children -/// landed first?" before writing the push event. A request row that -/// is missing or has no `accepted_ordinal` set is a legacy crash -/// window from before the v30 migration — the drain's per-ref walk -/// still produces a correct push event by treating the first child -/// by `(ordinal, id)` as the accepted one, so the helper falls back -/// to the row's own ordinal in that case. This is the test seam that -/// keeps `drain_re_derives_all_three_artifacts_for_an_applied_row` -/// (a fixture that does NOT pre-stage a `receive_pack_requests` row) -/// passing under the new model. -async fn lookup_accepted_ordinal( +/// If a crash lands between "all artifacts written" and "request +/// moved to `complete`", the same drain pass completes the state +/// transition. The artifacts are already in place; the +/// `mark_request_complete` call is a single SQL UPDATE. +pub async fn apply_request_effects( state: &AppState, request_id: &str, - fallback: i32, -) -> anyhow::Result { - let row: Option<(Option,)> = - sqlx::query_as("SELECT accepted_ordinal FROM receive_pack_requests WHERE id = $1") - .bind(request_id) - .fetch_optional(state.db.pool()) - .await?; - Ok(row.and_then(|(o,)| o).unwrap_or(fallback)) +) -> anyhow::Result { + // 1. Load the request row. + let req = state + .db + .get_receive_pack_request(request_id) + .await? + .ok_or_else(|| anyhow::anyhow!("request row missing for {request_id}"))?; + + // 2. State gate: only `outcomes_committed` and `effects_pending` are + // eligible. Terminal states (`complete`, `rejected_at_git`) are + // skipped. + if !matches!( + req.state.as_str(), + crate::db::request_state::OUTCOMES_COMMITTED | crate::db::request_state::EFFECTS_PENDING + ) { + return Ok(EffectsOutcome::Nothing); + } + + // 3. No accepted ordinal means no ref proved landed. The request is + // eligible for `complete` (or `rejected_at_git` if the parsed + // report shows an explicit failure, but that flag is set by the + // handler's four-branch flip, not here). + let accepted_ordinal = match req.accepted_ordinal { + Some(o) => o, + None => return Ok(EffectsOutcome::Nothing), + }; + + // 4. Load the request's children. Certs and anchor jobs run for + // every child whose `ref_name` is in the parsed report's ok + // set; the request row's `parsed_report` is the durable + // record of that set. + let children = state + .db + .list_pending_ref_transitions_for_request(request_id) + .await?; + let ok_ref_names: std::collections::HashSet = req + .parsed_report + .as_ref() + .and_then(|v| v.get("ref_results")) + .and_then(|v| v.as_array()) + .map(|arr| { + arr.iter() + .filter_map(|r| { + let ok = r.get("ok").and_then(|o| o.as_bool()).unwrap_or(false); + let name = r + .get("ref_name") + .and_then(|n| n.as_str()) + .map(|s| s.to_string()); + if ok { + name + } else { + None + } + }) + .collect() + }) + .unwrap_or_default(); + let accepted_children: Vec<&PendingRefTransition> = children + .iter() + .filter(|c| ok_ref_names.contains(&c.ref_name)) + .collect(); + + // 5. Look up the repo for cert/webhook payload construction. If + // the row is missing (deleted under us), bail with Retry so + // the drain re-runs later when the cache is warm again. + let repo = match state.db.get_repo_by_id(&req.repo_id).await? { + Some(r) => r, + None => { + return Ok(EffectsOutcome::Retry { + last_error: format!("repo {} not found", req.repo_id), + }); + } + }; + + // 6. Push event — written once, for the request. The live and + // recovery paths produce the same id because both key on + // `(request_id, accepted_ordinal)`. + let push_event_id = + crate::db::push_event_id_for(&req.id, accepted_ordinal); + let accepted_ref = children + .iter() + .find(|c| c.ordinal == accepted_ordinal); + let commit_hash = accepted_ref + .map(|c| c.new_sha.clone()) + .unwrap_or_else(|| chrono::Utc::now().timestamp().to_string()); + if let Err(e) = state + .db + .record_push_with_id( + &push_event_id, + &req.pusher_did, + &req.repo_id, + &commit_hash, + 0, + ) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + "apply_request_effects: push event insert failed; request left for drain retry" + ); + return Ok(EffectsOutcome::Retry { + last_error: format!("push event: {e}"), + }); + } + + // 7. Trust score bump — best-effort, like the inline handler. A + // failure here does NOT retry the request; the bump is + // informational and the next push will catch up. + if let Ok(push_count) = state.db.get_push_count(&req.pusher_did).await { + // 0.05 base (from registration) + 0.05 per push, capped at 1.0 + let new_score = (push_count as f64 * 0.05 + 0.05).min(1.0); + let _ = state + .db + .update_trust_score(&req.pusher_did, new_score) + .await; + } + + // 8. Per-ref certs and anchor jobs. Each accepted child gets one + // of each. Failures are accumulated; the first one is + // returned as the Retry reason. + let mut first_error: Option = None; + for child in &accepted_children { + let cert_id = crate::db::ref_cert_id_for(&req.id, child.ordinal); + if let Err(e) = cert::issue_ref_certificate_with_issued_at( + state, + &req.repo_id, + &child.ref_name, + &child.old_sha, + &child.new_sha, + &req.pusher_did, + &cert_id, + Some(child.created_at.clone()), + ) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + ref_name = %child.ref_name, + "apply_request_effects: cert insert failed; child left for drain retry" + ); + first_error.get_or_insert_with(|| format!("cert {}: {e}", child.ref_name)); + continue; + } + + let anchor_id = crate::db::anchor_job_id_for( + &req.repo_id, + &child.ref_name, + &child.old_sha, + &child.new_sha, + ); + let job = crate::db::AnchorJob { + id: anchor_id, + repo_id: req.repo_id.clone(), + ref_name: child.ref_name.clone(), + old_sha: child.old_sha.clone(), + new_sha: child.new_sha.clone(), + pusher_did: req.pusher_did.clone(), + created_at: chrono::Utc::now().to_rfc3339(), + claimed_at: None, + }; + if let Err(e) = state.db.insert_anchor_job_idempotent(&job).await { + tracing::warn!( + err = %e, + request_id = %request_id, + ref_name = %child.ref_name, + "apply_request_effects: anchor insert failed; child left for drain retry" + ); + first_error.get_or_insert_with(|| format!("anchor {}: {e}", child.ref_name)); + } + } + + if let Some(err) = first_error { + return Ok(EffectsOutcome::Retry { last_error: err }); + } + + // 9. All artifacts landed — clean up the children and let the + // caller move the request to `complete`. + if let Err(e) = state + .db + .delete_pending_ref_transitions_by_request_id(request_id) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + "apply_request_effects: child cleanup failed; idempotent retry will pick them up on next pass" + ); + // Don't fail the request — the artifacts are in place and a + // future pass is harmless. + } + + // 10. Webhooks — best-effort, per landed ref. Same shape as the + // inline handler's webhook block. + if !ok_ref_names.is_empty() { + let base_url = state + .config + .public_url + .as_deref() + .unwrap_or("http://127.0.0.1:7545") + .trim_end_matches('/'); + let owner_short = crate::db::normalize_owner_key(&repo.owner_did); + let clone_url = format!("{}/{}/{}.git", base_url, owner_short, repo.name); + for child in &accepted_children { + let payload = serde_json::json!({ + "ref": child.ref_name, + "before": child.old_sha, + "after": child.new_sha, + "created": child.old_sha == "0000000000000000000000000000000000000000", + "forced": false, + "pusher": { + "did": req.pusher_did, + }, + "repository": { + "id": repo.id, + "name": repo.name, + "owner_did": repo.owner_did, + "clone_url": clone_url, + }, + }); + crate::webhooks::fire_event( + state.db.clone(), + state.http_client.clone(), + &repo.id, + "push", + payload, + ); + } + } + + Ok(EffectsOutcome::Done) } #[cfg(test)] @@ -765,15 +916,127 @@ mod drain_tests { } } - /// The reviewer's proof at the durable-outbox layer. Insert a row - /// in `applied` state (the crash window), drain, and assert - /// exactly one push event, one cert with the original pusher, - /// and one anchor job. + /// Stage a `receive_pack_requests` row in `outcomes_committed` + /// alongside the per-ref children that landed under it. The + /// `parsed_report` is the durable record the effect executor + /// reads to decide which children are `ok`. Each child is + /// inserted via `insert_pending_ref_transition_for_test`, so + /// the deterministic PKs match what the production handler + /// would write. The repo row is also seeded so `apply_request_effects`'s + /// `get_repo_by_id` lookup succeeds (the live handler always + /// has the repo in cache before the effect executor is called). + async fn stage_request_with_children( + db: &Db, + request_id: &str, + repo_id: &str, + accepted_ordinal: Option, + children: &[PendingRefTransition], + parsed_report: serde_json::Value, + ) { + stage_request_with_pusher( + db, + request_id, + repo_id, + "did:key:z6pusher", + accepted_ordinal, + children, + parsed_report, + ) + .await; + } + + /// Like [`stage_request_with_children`] but lets the caller pick + /// the request row's `pusher_did`. Used by the cert-refresh + /// tests where the recovery's pusher DID must NOT match the + /// helper's default. + async fn stage_request_with_pusher( + db: &Db, + request_id: &str, + repo_id: &str, + pusher_did: &str, + accepted_ordinal: Option, + children: &[PendingRefTransition], + parsed_report: serde_json::Value, + ) { + // Seed a minimal repo row so the effect executor's + // `get_repo_by_id` lookup succeeds. `ON CONFLICT DO NOTHING` + // means tests that already seeded a repo (e.g. cert-refresh + // tests that need a specific `owner_did`) are unaffected. + sqlx::query( + r#"INSERT INTO repos (id, name, owner_did, description, is_public, default_branch, + created_at, updated_at, disk_path, forked_from, machine_id) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) + ON CONFLICT (id) DO NOTHING"#, + ) + .bind(repo_id) + .bind(repo_id) + .bind(pusher_did) + .bind(Option::::None) + .bind(true) + .bind("main") + .bind(chrono::Utc::now().to_rfc3339()) + .bind(chrono::Utc::now().to_rfc3339()) + .bind(format!("/tmp/{repo_id}")) + .bind(Option::::None) + .bind(Option::::None) + .execute(db.pool()) + .await + .expect("seed repo row"); + + sqlx::query( + r#"INSERT INTO receive_pack_requests + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(request_id) + .bind(repo_id) + .bind(pusher_did) + .bind("did:key:z6node") + .bind(Vec::::new()) + .bind([0u8; 32].to_vec()) + .bind(crate::db::request_state::OUTCOMES_COMMITTED) + .bind(Some(true)) + .bind(&parsed_report) + .bind(accepted_ordinal) + .bind(0_i32) + .bind(Option::::None) + .bind(Option::::None) + .bind(Utc::now().to_rfc3339()) + .bind(Option::::None) + .execute(db.pool()) + .await + .unwrap(); + + for child in children { + db.insert_pending_ref_transition_for_test(child).await.unwrap(); + } + } + + /// Build the `parsed_report` JSON the drain reads. The + /// `apply_request_effects` effect-executor uses the `ok` field + /// per `ref_name` to decide which children get certs and + /// anchors; the `accepted_ordinal` field on the request row + /// picks the row whose `new_sha` carries the push event. + fn parsed_report_ok(refs: &[(&str, bool)]) -> serde_json::Value { + serde_json::json!({ + "unpack_ok": true, + "ref_results": refs.iter().map(|(name, ok)| serde_json::json!({ + "ref_name": name, + "ok": ok, + })).collect::>(), + }) + } + + /// The reviewer's proof at the durable-outbox layer. Stage a + /// `receive_pack_requests` row in `outcomes_committed` with one + /// landed child (the crash window — receive_pack returned Ok and + /// git accepted, only the effects fan-out didn't run), drain, and + /// assert exactly one push event, one cert with the original + /// pusher, one anchor job, and the request moved to `complete`. #[sqlx::test] async fn drain_re_derives_all_three_artifacts_for_an_applied_row(pool: sqlx::PgPool) { - // Pre-create the repo so the FK-ish usage in tests doesn't blow up. - // The drain itself does not require a repo row to exist; the test - // only checks the derived artifacts. let state = crate::test_support::test_state(pool).await; let repo_id = "repo-failure-injection"; @@ -781,17 +1044,23 @@ mod drain_tests { let old = "a".repeat(40); let new = "b".repeat(40); let row = make_row(repo_id, ref_name, &old, &new); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + let request_id = row.request_id.clone(); + let parsed_report = parsed_report_ok(&[(ref_name, true)]); + stage_request_with_children( + &state.db, + &request_id, + repo_id, + Some(row.ordinal), + std::slice::from_ref(&row), + parsed_report, + ) + .await; - let (n, examined) = drain_pending_ref_transitions(state.clone(), 100) + let (n, examined) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); - assert_eq!(n, 1, "exactly one transition re-derived"); - assert_eq!(examined, 1, "the loop examined the single row"); + assert_eq!(n, 1, "exactly one request re-derived"); + assert_eq!(examined, 1, "the loop examined the single request"); // Push event: exactly one row, keyed on the deterministic id. let _push_id = crate::db::push_event_id_for(&row.request_id, row.ordinal); @@ -832,126 +1101,158 @@ mod drain_tests { .unwrap(); assert_eq!(anchor_count, 1, "exactly one anchor job per transition"); - // The drain deleted the row. - let after = state + // The request row moved to `complete`. + let after = state.db.get_receive_pack_request(&request_id).await.unwrap(); + assert_eq!( + after.expect("request row exists").state, + crate::db::request_state::COMPLETE, + "drain moves the request to complete" + ); + // Children are cleaned up. + let still_applied = state .db .list_pending_ref_transitions_applied(100) .await .unwrap(); assert!( - after.is_empty(), - "drain deletes the row after the work lands" + still_applied.is_empty(), + "drain deletes the children after the work lands" ); // A second drain pass is a no-op. - let (n2, examined2) = drain_pending_ref_transitions(state.clone(), 100) + let (n2, examined2) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); assert_eq!(n2, 0, "a second drain pass has nothing to do"); - assert_eq!(examined2, 0, "no rows to examine on a second pass"); + assert_eq!(examined2, 0, "no requests to examine on a second pass"); } - /// The reviewer's second proof, end-to-end. A `cancelled` row is - /// NEVER promoted by the drain. The drain only re-derives - /// artifacts for `applied` rows; a row that was `cancelled` - /// because receive_pack returned Err stays cancelled, and no - /// push event, cert, or anchor is created. + /// The reviewer's second proof, end-to-end. A request that git + /// rejected (no `accepted_ordinal`) never produces a push event, + /// cert, or anchor. The drain still picks up the request + /// (because it's in `outcomes_committed` — the live handler + /// always lands here after git returns), `apply_request_effects` + /// returns `Nothing` because there is no accepted ref, and the + /// drain moves the request to `complete` without writing any + /// artifacts. #[sqlx::test] - async fn cancelled_row_produces_no_artifacts(pool: sqlx::PgPool) { + async fn rejected_at_git_request_produces_no_artifacts(pool: sqlx::PgPool) { let state = crate::test_support::test_state(pool).await; - let mut row = make_row( - "repo-cancel", - "refs/heads/main", - &"a".repeat(40), - &"b".repeat(40), - ); - row.state = pending_state::CANCELLED.to_string(); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + // Stage a request in `outcomes_committed` with NO + // `accepted_ordinal` (git rejected all refs). The drain + // picks it up, `apply_request_effects` short-circuits at the + // `accepted_ordinal.is_none()` gate, and the drain calls + // `mark_request_complete` for `Nothing`. + let request_id = "req-rejected"; + let repo_id = "repo-rejected"; + let parsed_report = serde_json::json!({ + "unpack_ok": false, + "ref_results": [{ + "ref_name": "refs/heads/main", + "ok": false, + "message": "deny non-fast-forward", + }], + }); + stage_request_with_children( + &state.db, + request_id, + repo_id, + None, + &[], + parsed_report, + ) + .await; - let (n, _examined) = drain_pending_ref_transitions(state.clone(), 100) + let (n, examined) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); - assert_eq!(n, 0, "the drain must not promote a cancelled row"); + assert_eq!(n, 1, "drain processed the no-effect request"); + assert_eq!(examined, 1, "the loop examined the request"); - // No push event, no cert, no anchor. - let push_count = state - .db - .count_push_events(&row.repo_id, &row.new_sha, &row.pusher_did) - .await - .unwrap(); - assert_eq!(push_count, 0); - let certs = state - .db - .list_ref_certificates(&row.repo_id, 10) - .await - .unwrap(); - assert!(certs.is_empty()); - let anchor_count = state - .db - .count_anchor_jobs(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha) - .await - .unwrap(); - assert_eq!(anchor_count, 0); + // The request is now `complete` — Nothing outcome moves it. + let after = state.db.get_receive_pack_request(request_id).await.unwrap(); + assert_eq!( + after.expect("request row exists").state, + crate::db::request_state::COMPLETE, + "Nothing outcome moves the request to complete" + ); - // The cancelled row is also left untouched. - let still = state + // No push event, no cert, no anchor. + let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); + assert_eq!( + push_count, 0, + "no push event for a request with no accepted ref" + ); + let certs = state.db.list_ref_certificates(repo_id, 10).await.unwrap(); + assert!(certs.is_empty(), "no certs for a no-effect request"); + let still_applied = state .db .list_pending_ref_transitions_applied(100) .await .unwrap(); - assert!(still.is_empty(), "drain reads only applied rows"); + assert!( + still_applied.is_empty(), + "no children exist for this no-effect request" + ); } - /// A `prepared` row that the handler never reached the post-Ok - /// branch for (e.g. process crash between insert_prepared and - /// mark_applied) is also never promoted. The drain reads only - /// `applied` rows, so a `prepared` row stays in `prepared` and - /// is invisible to the drain. + /// A request the handler has not yet finished (state = + /// `received`, git has not yet returned) is invisible to the + /// per-request drain. The drain only reads `outcomes_committed` + /// and `effects_pending`, so a `received` row stays where the + /// handler left it and no effects are attempted. #[sqlx::test] - async fn prepared_row_produces_no_artifacts(pool: sqlx::PgPool) { + async fn received_request_produces_no_artifacts(pool: sqlx::PgPool) { let state = crate::test_support::test_state(pool).await; - let mut row = make_row( - "repo-prep", - "refs/heads/main", - &"a".repeat(40), - &"b".repeat(40), - ); - row.state = pending_state::PREPARED.to_string(); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + // Stage the request row directly in `received` (the state + // the handler writes before git returns). + let request_id = "req-received"; + let repo_id = "repo-received"; + sqlx::query( + r#"INSERT INTO receive_pack_requests + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(request_id) + .bind(repo_id) + .bind("did:key:z6pusher") + .bind("did:key:z6node") + .bind(Vec::::new()) + .bind([0u8; 32].to_vec()) + .bind(crate::db::request_state::RECEIVED) + .bind(Option::::None) + .bind(Option::::None) + .bind(Option::::None) + .bind(0_i32) + .bind(Option::::None) + .bind(Option::::None) + .bind(Utc::now().to_rfc3339()) + .bind(Option::::None) + .execute(state.db.pool()) + .await + .unwrap(); - let (n, _examined) = drain_pending_ref_transitions(state.clone(), 100) + // The drain must not pick up a `received` row. + let (n, examined) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); - assert_eq!(n, 0, "the drain must not promote a prepared row"); + assert_eq!(n, 0, "the drain must not touch a `received` request"); + assert_eq!(examined, 0, "the drain's WHERE excludes `received`"); - let push_count = state - .db - .count_push_events(&row.repo_id, &row.new_sha, &row.pusher_did) - .await - .unwrap(); - assert_eq!(push_count, 0); - let certs = state - .db - .list_ref_certificates(&row.repo_id, 10) - .await - .unwrap(); - assert!(certs.is_empty()); - let anchor_count = state - .db - .count_anchor_jobs(&row.repo_id, &row.ref_name, &row.old_sha, &row.new_sha) - .await - .unwrap(); - assert_eq!(anchor_count, 0); + // The request is unchanged. + let after = state.db.get_receive_pack_request(request_id).await.unwrap(); + assert_eq!( + after.expect("request row exists").state, + crate::db::request_state::RECEIVED, + "received requests are left to the handler" + ); + + let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); + assert_eq!(push_count, 0, "no push event for an unstarted request"); } // ----- P1-A reconcile tests ----- @@ -1727,23 +2028,21 @@ mod drain_tests { // ----- P2-A drain resilience tests ----- // // These tests cover the "drain must not abort on first failure" - // and "drain must not cap at 1000 rows per startup" findings. - // Backlog processing uses the production `drain_pending_ref_transitions_all` - // (the `DRAIN_PER_PASS_LIMIT` / `DRAIN_MAX_PASSES` constants from - // this module) so the test exercises the same wrapper the - // startup calls. Failure isolation uses - // `drain_pending_ref_transitions_with` to inject a closure that - // errors for one row and succeeds for the next. + // and "drain must not cap at 1000 requests per startup" findings. + // Backlog processing uses the production + // `drain_receive_pack_requests_all` (the `DRAIN_PER_PASS_LIMIT` / + // `DRAIN_MAX_PASSES` constants from this module) so the test + // exercises the same wrapper the startup calls. Failure isolation + // uses `drain_receive_pack_requests_with` to inject a closure + // that returns `Retry` for one request and `Done` for the next. #[sqlx::test] async fn drain_processes_backlog_larger_than_one_pass(pool: sqlx::PgPool) { let state = crate::test_support::test_state(pool).await; - // Seed 1500 distinct `applied` rows. Each row needs a unique - // `request_id` so the deterministic `pending_ref_transition` - // PKs (which hash `request_id`) don't collide, and the - // push-event / cert / anchor PKs (which also hash - // `request_id`) don't collide either. + // Seed 1500 distinct request rows. Each request is its own + // `receive_pack_requests.id`; per-ref PKs hash the request + // id, and the certs / anchor jobs hash the request id too. const N: usize = 1500; for i in 0..N { let mut row = make_row( @@ -1752,9 +2051,6 @@ mod drain_tests { &"0".repeat(40), &format!("{:040x}", i as u64), ); - // Override `request_id` to a unique value per row. The - // `id` is derived from this in `make_row`, so the unique - // `request_id` also gives a unique row PK. row.request_id = format!("req-{i}"); row.id = crate::db::deterministic_id(&[ "pending_ref_transition", @@ -1764,17 +2060,22 @@ mod drain_tests { &row.old_sha, &row.new_sha, ]); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + let parsed_report = parsed_report_ok(&[("refs/heads/main", true)]); + stage_request_with_children( + &state.db, + &row.request_id, + "repo-backlog", + Some(row.ordinal), + std::slice::from_ref(&row), + parsed_report, + ) + .await; } // Drain with the production limits. Two passes of 1000 each - // cover all 1500 rows; the third pass would be empty and + // cover all 1500 requests; the third pass would be empty and // exits the loop early on the `n < per_pass_limit` check. - let total = drain_pending_ref_transitions_all( + let total = drain_receive_pack_requests_all( state.clone(), DRAIN_PER_PASS_LIMIT, DRAIN_MAX_PASSES, @@ -1783,26 +2084,32 @@ mod drain_tests { .unwrap(); assert_eq!(total, N, "drain processed the full backlog"); - // No `applied` rows remain. + // No `outcomes_committed` requests remain. let after = state .db - .list_pending_ref_transitions_applied(10_000) + .count_receive_pack_requests_due() .await .unwrap(); - assert!( - after.is_empty(), - "every applied row was processed and deleted" + assert_eq!( + after, 0, + "every request row was processed and moved to complete" ); + // No per-ref children remain either. + let still = state + .db + .list_pending_ref_transitions_applied(10_000) + .await + .unwrap(); + assert!(still.is_empty(), "every child was cleaned up"); } #[sqlx::test] async fn drain_continues_past_a_failing_row(pool: sqlx::PgPool) { let state = crate::test_support::test_state(pool).await; - // Seed two `applied` rows, A first so the `ORDER BY applied_at - // ASC NULLS LAST, id ASC` query hits A before B. They have - // distinct `request_id`s so the deterministic PKs don't - // collide. + // Seed two requests (A first so the `ORDER BY created_at ASC, + // id ASC` query hits A before B). Each request owns a single + // child row at ordinal 0. let mut row_a = make_row( "repo-fail-then-pass", "refs/heads/main", @@ -1833,63 +2140,86 @@ mod drain_tests { &row_b.old_sha, &row_b.new_sha, ]); - state - .db - .insert_pending_ref_transition_for_test(&row_a) - .await - .unwrap(); - state - .db - .insert_pending_ref_transition_for_test(&row_b) - .await - .unwrap(); + let parsed_report = parsed_report_ok(&[("refs/heads/main", true)]); + stage_request_with_children( + &state.db, + "req-A", + "repo-fail-then-pass", + Some(0), + std::slice::from_ref(&row_a), + parsed_report.clone(), + ) + .await; + stage_request_with_children( + &state.db, + "req-B", + "repo-fail-then-pass", + Some(0), + std::slice::from_ref(&row_b), + parsed_report, + ) + .await; - // Inject a closure that fails for row A and delegates to - // `derive_one` for everything else. Row B is processed - // normally; row A's failure is logged and the row stays - // `applied` for a future retry. The `Fn` bound on the seam - // forbids moving the id into the closure, so the closure - // clones the id from the outer `row_a_id` local on every - // iteration. - let row_a_id = row_a.id.clone(); + // Inject a closure that returns Retry for request A and + // delegates to the real `apply_request_effects` for B. + // Request A is moved to `effects_pending` for a future + // retry; request B is fully processed. + let state_for_closure = state.clone(); let (processed, examined) = - drain_pending_ref_transitions_with(state.clone(), 100, |s, r| { - let target = row_a_id.clone(); + drain_receive_pack_requests_with(state.clone(), 100, |_s, req_id| { + let target = String::from("req-A"); + let st = state_for_closure.clone(); async move { - if r.id == target { - Err(anyhow::anyhow!("injected derive failure")) + if req_id == target { + Ok(EffectsOutcome::Retry { + last_error: "injected derive failure".to_string(), + }) } else { - derive_one(&s, &r).await + apply_request_effects(&st, &req_id).await } } }) .await .unwrap(); - assert_eq!(processed, 1, "only row B is fully processed and deleted"); + assert_eq!(processed, 1, "only request B is fully processed"); assert_eq!( examined, 2, - "the loop examined both rows; processed/derivation is independent of pagination" + "the loop examined both requests; processed/derivation is independent of pagination" ); - // Row A is still in `applied` (NOT deleted, NOT re-derivable - // yet by a future pass that just calls `derive_one` — the - // inserted artifacts were never created). - let after = state + // Request A is in `effects_pending` (Retry moved it there). + let a_req = state .db - .list_pending_ref_transitions_applied(100) + .get_receive_pack_request("req-A") .await - .unwrap(); - assert_eq!(after.len(), 1, "row A is still in `applied`"); - assert_eq!(after[0].id, row_a_id); + .unwrap() + .expect("req-A exists"); + assert_eq!( + a_req.state, + crate::db::request_state::EFFECTS_PENDING, + "Retry outcome moves A to effects_pending" + ); + // Request B is in `complete`. + let b_req = state + .db + .get_receive_pack_request("req-B") + .await + .unwrap() + .expect("req-B exists"); + assert_eq!( + b_req.state, + crate::db::request_state::COMPLETE, + "Done outcome moves B to complete" + ); - // Row A's artifacts were not created (the closure errored - // before any insert ran). + // Request A's artifacts were not created (the closure + // returned Retry before any insert ran). let a_push = state .db .count_push_events(&row_a.repo_id, &row_a.new_sha, &row_a.pusher_did) .await .unwrap(); - assert_eq!(a_push, 0, "row A's push event was not created"); + assert_eq!(a_push, 0, "request A's push event was not created"); let a_anchors = state .db .count_anchor_jobs( @@ -1900,38 +2230,22 @@ mod drain_tests { ) .await .unwrap(); - assert_eq!(a_anchors, 0, "row A's anchor job was not created"); - let a_certs = state - .db - .list_ref_certificates(&row_a.repo_id, 10) - .await - .unwrap(); - // The cert table is per-(repo, ref) with one row. Row B's - // drain wrote a cert for this `(repo, ref)`. We need to - // check row A's specific cert by its deterministic id — - // the row A's `derive_one` never ran, so the row A cert id - // must not exist in the table. + assert_eq!(a_anchors, 0, "request A's anchor job was not created"); let a_cert_id = crate::db::ref_cert_id_for(&row_a.request_id, row_a.ordinal); let a_cert = state.db.get_ref_certificate(&a_cert_id).await.unwrap(); assert!( a_cert.is_none(), - "row A's specific cert was not created (got {:?})", + "request A's cert id must not exist (got {:?})", a_cert.map(|c| c.id) ); - // The single cert for this `(repo, ref)` is row B's. - assert_eq!(a_certs.len(), 1, "row B's cert exists in the table"); - let b_cert_id = crate::db::ref_cert_id_for(&row_b.request_id, row_b.ordinal); - assert_eq!(a_certs[0].id, b_cert_id, "the only cert is row B's"); - // Row B's other artifacts WERE created (the closure called - // the real `derive_one`, which writes the push event and - // anchor job for row B). + // Request B's artifacts WERE created. let b_push = state .db .count_push_events(&row_b.repo_id, &row_b.new_sha, &row_b.pusher_did) .await .unwrap(); - assert_eq!(b_push, 1, "row B's push event was created"); + assert_eq!(b_push, 1, "request B's push event was created"); let b_anchors = state .db .count_anchor_jobs( @@ -1942,7 +2256,10 @@ mod drain_tests { ) .await .unwrap(); - assert_eq!(b_anchors, 1, "row B's anchor job was created"); + assert_eq!(b_anchors, 1, "request B's anchor job was created"); + let b_cert_id = crate::db::ref_cert_id_for(&row_b.request_id, row_b.ordinal); + let b_cert = state.db.get_ref_certificate(&b_cert_id).await.unwrap(); + assert!(b_cert.is_some(), "request B's cert was created"); } // ----- P2-B multi-ref push event cardinality test ----- @@ -1964,43 +2281,18 @@ mod drain_tests { ) { let state = crate::test_support::test_state(pool).await; - // Stage the request row with `accepted_ordinal = 0` so the - // drain's gate (`row.ordinal == accepted_ordinal`) only fires - // for the first child. The v30 model carries the - // accepted-ordinal on the request row, so the test fixture - // must include one — this is the production shape the drain - // sees on every replay. - sqlx::query( - "INSERT INTO receive_pack_requests \ - (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, \ - state, created_at, accepted_ordinal) \ - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", - ) - .bind("req-multi") - .bind("repo-multi") - .bind("did:key:z6pusher") - .bind("did:key:z6node") - .bind(Vec::::new()) - .bind([0u8; 32].to_vec()) - .bind("outcomes_committed") - .bind(Utc::now().to_rfc3339()) - .bind(Some(0_i32)) - .execute(state.db.pool()) - .await - .unwrap(); - - // Three rows for the SAME `request_id`, distinct `ref_name`s, - // distinct ordinals 0/1/2. The `new_sha` is the same across - // all three because this models a push that advanced a tip - // commit onto three refs at once (a common case for - // `git push --all` or for a single-commit push to multiple - // branches). + // Three children for the SAME `request_id`, distinct + // `ref_name`s, distinct ordinals 0/1/2. The `new_sha` is + // shared across all three because this models a push that + // advanced a tip commit onto three refs at once (the + // ordinary shape of `git push --all`). let shared_new_sha = "c".repeat(40); let ref_names = [ "refs/heads/main", "refs/heads/feature-a", "refs/heads/feature-b", ]; + let mut children = Vec::new(); for (i, ref_name) in ref_names.iter().enumerate() { let mut row = make_row("repo-multi", ref_name, &"0".repeat(40), &shared_new_sha); row.request_id = "req-multi".to_string(); @@ -2016,25 +2308,36 @@ mod drain_tests { &row.old_sha, &row.new_sha, ]); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + children.push(row); } + // Stage the request with `accepted_ordinal = Some(0)` so the + // first child is the one whose `new_sha` becomes the push + // event's commit_hash. All three refs are in the parsed + // report's ok set. + let parsed_report = parsed_report_ok(&[ + ("refs/heads/main", true), + ("refs/heads/feature-a", true), + ("refs/heads/feature-b", true), + ]); + stage_request_with_children( + &state.db, + "req-multi", + "repo-multi", + Some(0), + &children, + parsed_report, + ) + .await; - // Drain all three rows. - let (n, examined) = drain_pending_ref_transitions(state.clone(), 100) + // Drain the request. + let (n, examined) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); - assert_eq!(n, 3, "all three rows re-derived"); - assert_eq!(examined, 3, "the loop examined all three rows"); + assert_eq!(n, 1, "the request was processed"); + assert_eq!(examined, 1, "the loop examined the single request"); // Exactly one push event row, keyed on the deterministic - // (request_id, accepted_ordinal) id. Only the first child - // (ordinal 0) wrote the event, so the others' attempted - // writes either no-op'd (if their id collided with a row the - // request didn't accept) or never ran. + // (request_id, accepted_ordinal) id. let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); assert_eq!( push_count, 1, @@ -2050,9 +2353,6 @@ mod drain_tests { // The deterministic id is the one the live path would have // written. let expected_id = crate::db::push_event_id_for("req-multi", 0); - // We don't have a direct "select by id" for push_events; the - // count of 1 already proves the cardinality. Assert the id - // is stable for completeness. assert_eq!( expected_id, crate::db::push_event_id_for("req-multi", 0), @@ -2060,7 +2360,7 @@ mod drain_tests { ); // Certs: one per ref (the cert contract is per-ref, NOT - // collapsed by ordinal). Three rows → three certs. + // collapsed by ordinal). Three children → three certs. let certs = state .db .list_ref_certificates("repo-multi", 10) @@ -2073,7 +2373,7 @@ mod drain_tests { ); // Anchor jobs: one per `(repo, ref, old, new)` transition. - // Three rows → three anchor jobs. + // Three children → three anchor jobs. for (i, ref_name) in ref_names.iter().enumerate() { let n = state .db @@ -2105,27 +2405,6 @@ mod drain_tests { async fn multi_ref_recovery_uses_first_refs_new_sha_for_push_event(pool: sqlx::PgPool) { let state = crate::test_support::test_state(pool).await; - // Stage the request row with `accepted_ordinal = 0` so the - // first child (ordinal 0) is the only row whose gate fires. - sqlx::query( - "INSERT INTO receive_pack_requests \ - (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, \ - state, created_at, accepted_ordinal) \ - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", - ) - .bind("req-multi-distinct") - .bind("repo-multi-distinct") - .bind("did:key:z6pusher") - .bind("did:key:z6node") - .bind(Vec::::new()) - .bind([0u8; 32].to_vec()) - .bind("outcomes_committed") - .bind(Utc::now().to_rfc3339()) - .bind(Some(0_i32)) - .execute(state.db.pool()) - .await - .unwrap(); - // Three refs, each with a distinct `new_sha` modelling a // multi-branch push where each ref advanced to a different // tip. The first ref's new_sha is the one the live handler @@ -2139,6 +2418,7 @@ mod drain_tests { "refs/heads/feature-b", ]; let new_shas = [&first_new_sha, &second_new_sha, &third_new_sha]; + let mut children = Vec::new(); for (i, (ref_name, new_sha)) in ref_names.iter().zip(new_shas.iter()).enumerate() { let mut row = make_row("repo-multi-distinct", ref_name, &"0".repeat(40), new_sha); row.request_id = "req-multi-distinct".to_string(); @@ -2155,26 +2435,34 @@ mod drain_tests { &row.old_sha, &row.new_sha, ]); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + children.push(row); } + let parsed_report = parsed_report_ok(&[ + ("refs/heads/main", true), + ("refs/heads/feature-a", true), + ("refs/heads/feature-b", true), + ]); + stage_request_with_children( + &state.db, + "req-multi-distinct", + "repo-multi-distinct", + Some(0), + &children, + parsed_report, + ) + .await; - // Drain all three rows. - let (n, examined) = drain_pending_ref_transitions(state.clone(), 100) + // Drain the request. + let (n, examined) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); - assert_eq!(n, 3, "all three rows re-derived"); - assert_eq!(examined, 3, "the loop examined all three rows"); + assert_eq!(n, 1, "the request was processed"); + assert_eq!(examined, 1, "the loop examined the single request"); // Exactly one push event row, keyed on the deterministic - // (request_id, accepted_ordinal) id. Only the row whose - // ordinal matches the request's `accepted_ordinal` ran - // `record_push_with_id`, so the persisted `commit_hash` is - // the FIRST ref's `new_sha` — the same value the live path - // would have written. + // (request_id, accepted_ordinal) id. The persisted + // `commit_hash` is the FIRST ref's `new_sha` — the same + // value the live path would have written. let push_count = state.db.get_push_count("did:key:z6pusher").await.unwrap(); assert_eq!(push_count, 1, "exactly one push event row"); let first_event = state @@ -2200,8 +2488,7 @@ mod drain_tests { ); } - // Certs and anchors stay per-ref and per-transition - // (unchanged from the prior round). + // Certs stay per-ref (three refs → three certs). let certs = state .db .list_ref_certificates("repo-multi-distinct", 10) @@ -2214,18 +2501,20 @@ mod drain_tests { // // The previous loop's exit condition was `(n as i64) < per_pass_limit` // where `n` was rows *fully processed* (derive + delete). A pass - // where every `derive_one` returns Err logs each failure but - // increments `count = 0`; the outer loop sees `0 < per_pass_limit` - // and returns. Remaining `applied` rows are never attempted that - // boot. The fix returns `(processed, examined)` and keys the exit - // on `examined`. This test seeds `per_pass_limit` rows with a - // closure that fails for every one, then asserts the drain ran - // every row (processed=0, examined=per_pass_limit) so the outer - // loop continues to the next pass. + // where every `apply_request_effects` returns Retry logs each + // failure but increments `processed = 0`; the outer loop sees + // `0 < per_pass_limit` and returns. Remaining requests are never + // attempted that boot. The fix returns `(processed, examined)` + // and keys the exit on `examined`. This test seeds + // `per_pass_limit` requests with a closure that retries every + // one, then asserts the drain ran every request (processed=0, + // examined=per_pass_limit) so the outer loop continues to the + // next pass. #[sqlx::test] async fn drain_does_not_exit_early_when_every_row_fails(pool: sqlx::PgPool) { let state = crate::test_support::test_state(pool).await; const N: usize = 5; + let parsed_report = parsed_report_ok(&[("refs/heads/main", true)]); for i in 0..N { let mut row = make_row( "repo-all-fail", @@ -2242,36 +2531,53 @@ mod drain_tests { &row.old_sha, &row.new_sha, ]); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + stage_request_with_children( + &state.db, + &row.request_id, + "repo-all-fail", + Some(row.ordinal), + std::slice::from_ref(&row), + parsed_report.clone(), + ) + .await; } let (processed, examined) = - drain_pending_ref_transitions_with(state.clone(), N as i64, |_s, _r| async move { - Err(anyhow::anyhow!("injected: every row fails")) + drain_receive_pack_requests_with(state.clone(), N as i64, |_s, _req_id| async move { + Ok(EffectsOutcome::Retry { + last_error: "injected: every request fails".to_string(), + }) }) .await .unwrap(); - assert_eq!(processed, 0, "no row was fully processed"); + assert_eq!(processed, 0, "no request was fully processed"); assert_eq!( examined, N, - "the loop examined every row even though every derive failed" + "the loop examined every request even though every derive failed" ); - // The all-fail rows are still `applied` for a future retry: - // the loop never deletes a row whose derive returned Err. - let after = state + // Every request is in `effects_pending` for a future retry. + let due = state .db - .list_pending_ref_transitions_applied(100) + .count_receive_pack_requests_due() .await .unwrap(); + // The Retry path sets `next_attempt_at` 60s in the future, + // so the due count is 0 — but the requests still exist. + assert_eq!(due, 0, "Retry schedules the requests 60s out"); + // And there are N total outcomes_committed/effects_pending. + let total: i64 = sqlx::query_scalar( + r#"SELECT COUNT(*)::BIGINT FROM receive_pack_requests + WHERE state IN ($1, $2)"#, + ) + .bind(crate::db::request_state::OUTCOMES_COMMITTED) + .bind(crate::db::request_state::EFFECTS_PENDING) + .fetch_one(state.db.pool()) + .await + .unwrap(); assert_eq!( - after.len(), - N, - "failed rows stay `applied` for the next startup" + total, N as i64, + "all N requests are still pending for the next startup" ); } @@ -2337,10 +2643,10 @@ mod drain_tests { .await .unwrap(); - // Seed the durable row with the LANDED transition (what the - // push actually applied to disk): a different old_sha and - // new_sha, the genuine pusher DID. The drain must refresh - // the stale cert to this transition. + // Seed the durable child with the LANDED transition (what + // the push actually applied to disk): a different old_sha + // and new_sha, the genuine pusher DID. The drain must + // refresh the stale cert to this transition. let landed_old = "2".repeat(40); let landed_new = "3".repeat(40); let landed_pusher = "did:key:zLandedPusher"; @@ -2356,19 +2662,25 @@ mod drain_tests { &row.old_sha, &row.new_sha, ]); - state - .db - .insert_pending_ref_transition_for_test(&row) - .await - .unwrap(); + let parsed_report = parsed_report_ok(&[("refs/heads/main", true)]); + stage_request_with_pusher( + &state.db, + "req-stale", + &rec.id, + landed_pusher, + Some(0), + std::slice::from_ref(&row), + parsed_report, + ) + .await; // Drain. The recovery upsert must overwrite the stale cert // with the landed transition's SHAs / pusher / signature. - let (processed, examined) = drain_pending_ref_transitions(state.clone(), 100) + let (processed, examined) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); - assert_eq!(processed, 1, "the row was drained"); - assert_eq!(examined, 1, "the loop examined the single row"); + assert_eq!(processed, 1, "the request was drained"); + assert_eq!(examined, 1, "the loop examined the single request"); let certs = state.db.list_ref_certificates(&rec.id, 10).await.unwrap(); assert_eq!(certs.len(), 1, "exactly one cert row, the same id"); @@ -2429,7 +2741,7 @@ mod drain_tests { }; state.db.create_repo(&rec).await.unwrap(); - // A: original push, recovery row still `applied`. + // A: original push, request row still in `outcomes_committed`. let a_old = "0".repeat(40); let a_new = "1".repeat(40); let a_pusher = "did:key:zA"; @@ -2449,16 +2761,22 @@ mod drain_tests { // Backdate A's created_at by 5 minutes so the replay's // stamped `issued_at` is provably older than B's live one. a_row.created_at = (chrono::Utc::now() - chrono::Duration::minutes(5)).to_rfc3339(); - state - .db - .insert_pending_ref_transition_for_test(&a_row) - .await - .unwrap(); + let parsed_report = parsed_report_ok(&[("refs/heads/main", true)]); + stage_request_with_pusher( + &state.db, + a_request, + &rec.id, + a_pusher, + Some(0), + std::slice::from_ref(&a_row), + parsed_report, + ) + .await; // A's cert was written live (or never — we test the case - // where the row was left `applied` and the cert was NOT - // yet written, then B's live push arrives first and writes - // its cert, then A's drain replays). + // where the row was left pending and the cert was NOT yet + // written, then B's live push arrives first and writes its + // cert, then A's drain replays). // // Simulate: the live cert B has been written by a later // push. @@ -2466,10 +2784,11 @@ mod drain_tests { let b_new = "2".repeat(40); let b_pusher = "did:key:zB"; // B is a stand-in for "a later live push already wrote its - // cert". The cert id is arbitrary — what matters is the row - // collides with A's recovery on the `(repo_id, ref_name)` - // unique index. Use B's request-scoped id at ordinal 0 so the - // id is a real `(request_id, ordinal)` shape. + // cert". The cert id is arbitrary — what matters is the + // row collides with A's recovery on the + // `(repo_id, ref_name)` unique index. Use B's + // request-scoped id at ordinal 0 so the id is a real + // `(request_id, ordinal)` shape. let b_cert_id = crate::db::ref_cert_id_for("req-B", 0); state .db @@ -2491,11 +2810,11 @@ mod drain_tests { // created_at = now-5min) is OLDER than B's cert (now), so // the per-column CASE WHEN guards must NOT update B's // fields. - let (processed, examined) = drain_pending_ref_transitions(state.clone(), 100) + let (processed, examined) = drain_receive_pack_requests(state.clone(), 100) .await .unwrap(); - assert_eq!(processed, 1, "A's row was drained"); - assert_eq!(examined, 1, "the loop examined A's row"); + assert_eq!(processed, 1, "A's request was drained"); + assert_eq!(examined, 1, "the loop examined A's request"); let certs = state.db.list_ref_certificates(&rec.id, 10).await.unwrap(); assert_eq!(certs.len(), 1, "exactly one cert row remains"); diff --git a/crates/gitlawb-node/src/main.rs b/crates/gitlawb-node/src/main.rs index 8944ea5e4..72ea6fad3 100644 --- a/crates/gitlawb-node/src/main.rs +++ b/crates/gitlawb-node/src/main.rs @@ -714,7 +714,7 @@ async fn main() -> Result<()> { "pending ref transition reconcile failed at startup (non-fatal; will retry on next start)" ), } - match durable_outbox::drain_pending_ref_transitions_all( + match durable_outbox::drain_receive_pack_requests_all( state.clone(), durable_outbox::DRAIN_PER_PASS_LIMIT, durable_outbox::DRAIN_MAX_PASSES, diff --git a/crates/gitlawb-node/tests/inv22_gates.rs b/crates/gitlawb-node/tests/inv22_gates.rs index 4c2098fb8..87fc2af0e 100644 --- a/crates/gitlawb-node/tests/inv22_gates.rs +++ b/crates/gitlawb-node/tests/inv22_gates.rs @@ -547,9 +547,15 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() { let touch = production .find("state.db.touch_repo(") .expect("U5 gate stale: git_receive_pack no longer calls touch_repo"); - let webhook = production - .find("webhooks::fire_event(") - .expect("U5 gate stale: git_receive_pack no longer fires push webhooks"); + // #26 Split PR 1 step 3 — the webhook fan-out moved into + // `durable_outbox::apply_request_effects`, which the live + // handler calls inline (the recovery drain calls the same + // function on the next startup). The gate now pins that the + // handler is wired to the executor, not to a per-ref inline + // webhook call. + let effects_executor = production + .find("apply_request_effects(&state, &request_id)") + .expect("U5 gate stale: git_receive_pack no longer calls apply_request_effects inline"); assert!( success_flag < gate_open && gate_open < spawn, @@ -563,9 +569,100 @@ fn inv22_replication_tail_spawns_at_the_durability_boundary() { rejected push now spawns a tail" ); assert!( - spawn < release && spawn < touch && spawn < webhook, + spawn < release && spawn < touch && spawn < effects_executor, "U5 gate bypassed: the tail must be spawned BEFORE guard.release, touch_repo \ - and the webhook fan-out, so a disconnect in any of those windows cannot drop \ + and the effect executor, so a disconnect in any of those windows cannot drop \ this push's pins, recovery copy, and announcements" ); } + +/// #26 Split PR 1 step 3 — drain + handler share a single effect +/// executor. The live handler (api/repos.rs) and the recovery +/// drain (`durable_outbox::drain_receive_pack_requests_with`) both +/// call `apply_request_effects`, so the per-ref effects fan-out +/// lives in exactly one place. The v29 per-ref walk +/// (`derive_one`, `drain_pending_ref_transitions_all`, +/// `lookup_accepted_ordinal`) is dead code: any caller reintroduced +/// would be the per-ref walk the step-3 PR removed. This gate +/// fails if a call site slips back in or the new seam is bypassed. +#[test] +fn inv26_step3_live_and_drain_share_apply_request_effects() { + let repos = src("api/repos.rs"); + let outbox = src("durable_outbox.rs"); + + // The live handler calls `apply_request_effects`. Split the + // file at the test attribute so test code can't satisfy the + // gate by itself. + let production_repos = repos + .split("\nmod tests {") + .next() + .expect("split always yields a first chunk"); + + assert!( + production_repos.contains("apply_request_effects(&state, &request_id)"), + "live handler must call `apply_request_effects(&state, &request_id)`; \ + reverting to a per-ref inline fan-out splits live and recovery" + ); + + // The drain's per-request seam calls the same executor. Test + // code lives below `mod drain_tests`, so split there too. + let production_outbox = outbox + .split("\nmod drain_tests {") + .next() + .expect("split always yields a first chunk"); + + assert!( + production_outbox.contains("drain_receive_pack_requests_with"), + "drain seam `drain_receive_pack_requests_with` must exist; \ + removing it forces a per-ref walk back into the drain" + ); + assert!( + production_outbox.contains("apply_request_effects"), + "durable_outbox production code must define `apply_request_effects`; \ + removing it splits the executor between live and recovery" + ); + + // The drain's per-request walker is wired to the executor. + // The closure body of `drain_receive_pack_requests_with` is the + // only call site — if a future change calls `derive_one` + // instead, this assertion fires. + let drain_seam_open = production_outbox + .find("pub async fn drain_receive_pack_requests_with") + .expect("drain seam must be defined in the production half"); + let drain_seam_close = production_outbox[drain_seam_open..] + .find("\n}\n") + .expect("drain seam body must close"); + let drain_seam_body = &production_outbox[drain_seam_open..drain_seam_open + drain_seam_close]; + assert!( + drain_seam_body.contains("apply_request_effects"), + "drain_receive_pack_requests_with must call apply_request_effects; \ + wiring it to a per-ref helper reintroduces the v29 walk" + ); + + // The deleted per-ref drain must have zero call sites in the + // production half. A regression that re-adds a caller would + // bring back the per-ref fan-out. + assert!( + !production_outbox.contains("drain_pending_ref_transitions_all("), + "deleted `drain_pending_ref_transitions_all` must have zero production call sites; \ + a per-ref walk is reintroduced" + ); + assert!( + !production_outbox.contains("derive_one("), + "deleted `derive_one` must have zero production call sites; \ + the per-ref fan-out is reintroduced" + ); + assert!( + !production_outbox.contains("lookup_accepted_ordinal("), + "deleted `lookup_accepted_ordinal` must have zero production call sites; \ + the per-ref ordinal lookup is reintroduced" + ); + assert!( + !production_repos.contains("derive_one("), + "deleted `derive_one` must have zero live-handler call sites" + ); + assert!( + !production_repos.contains("drain_pending_ref_transitions"), + "deleted per-ref drain functions must have zero live-handler call sites" + ); +} From a014d8b66973cd6b41c0c0e3ae244a13dc5e5e35 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Thu, 3 Sep 2026 11:53:00 +0600 Subject: [PATCH 21/22] fix(node): bounded retirement purge for terminal request rows (#26 split 1/4 step 4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v30 partial index idx_receive_pack_requests_completed_at exists but no code reads it. Step 4 wires a periodic purge task that deletes terminal `complete` and `rejected_at_git` rows older than the retention window, along with their per-ref children. `quarantined` (a step-5 state) is never purged by the timer. - `purge_completed_receive_pack_requests(older_than, limit)` — deletes parent requests using the v30 partial index. - `purge_completed_pending_ref_transitions(older_than, limit)` — deletes children of purged parents, gated on the child's own applied_at / cancelled_at. - `purge_request_queue(db, retention_days, limit)` — orchestrator in durable_outbox.rs that calls both and returns the totals. The deletion order (parents first, then children) is the contract; a crash mid-purge leaves orphaned children that the next pass will pick up. - `spawn_queue_lifecycle_sweep` in main.rs — periodic task on the same detached pattern as `spawn_legacy_cid_sweep`, 24-hour interval, shutdown-aware. The interval matches the spec's "one per cluster per day" target. - Config knobs `queue_retention_days` (default 7, range 1..=365) and `queue_purge_batch` (default 1000, matches DRAIN_PER_PASS_LIMIT). - 3 new `durable_outbox::drain_tests::purge_*` tests pin the contract: only old terminal rows are deleted, the second pass is a no-op, and a row inside the retention window survives. - New `inv26_step4_queue_lifecycle_purge_is_wired` gate asserts the wiring (main.rs calls the purge, the DB helpers exist, the Config knobs exist). --- crates/gitlawb-node/src/api/repos.rs | 3 +- crates/gitlawb-node/src/config.rs | 51 +++++ crates/gitlawb-node/src/db/mod.rs | 73 ++++++ crates/gitlawb-node/src/durable_outbox.rs | 259 ++++++++++++++++++---- crates/gitlawb-node/src/main.rs | 38 ++++ crates/gitlawb-node/tests/inv22_gates.rs | 46 ++++ 6 files changed, 431 insertions(+), 39 deletions(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index eea426328..0e96422c4 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2858,8 +2858,7 @@ pub async fn git_receive_pack( } } Ok(crate::durable_outbox::EffectsOutcome::Retry { last_error }) => { - let next_attempt_at = - (Utc::now() + chrono::Duration::seconds(60)).to_rfc3339(); + let next_attempt_at = (Utc::now() + chrono::Duration::seconds(60)).to_rfc3339(); if let Err(e) = state .db .mark_request_effects_pending(&request_id, &next_attempt_at, &last_error) diff --git a/crates/gitlawb-node/src/config.rs b/crates/gitlawb-node/src/config.rs index 1fbf4376e..6395ac6d7 100644 --- a/crates/gitlawb-node/src/config.rs +++ b/crates/gitlawb-node/src/config.rs @@ -702,6 +702,33 @@ pub struct Config { value_parser = clap::builder::RangedU64ValueParser::::new().range(0..=86_400) )] pub pin_repair_sweep_delay_secs: u64, + + /// #26 Split PR 1 step 4 — receive-pack queue retention window. + /// Terminal `complete` and `rejected_at_git` rows older than + /// this are eligible for the periodic purge. `quarantined` + /// rows are never purged on a timer. The v30 partial index + /// `idx_receive_pack_requests_completed_at` keeps the scan + /// cheap regardless of the value. + #[arg( + long, + env = "GITLAWB_QUEUE_RETENTION_DAYS", + default_value_t = 7, + value_parser = clap::builder::RangedU64ValueParser::::new().range(1..=365) + )] + pub queue_retention_days: i64, + + /// #26 Split PR 1 step 4 — receive-pack queue purge batch size. + /// Each periodic purge pass deletes at most this many terminal + /// rows per batch. The drain and the purge share the same + /// `DRAIN_PER_PASS_LIMIT` budget; see the spawn function in + /// `main.rs` for the wiring. + #[arg( + long, + env = "GITLAWB_QUEUE_PURGE_BATCH", + default_value_t = 1000, + value_parser = clap::builder::RangedU64ValueParser::::new().range(1..=100_000) + )] + pub queue_purge_batch: i64, } impl Config { @@ -963,6 +990,30 @@ mod tests { ); } + #[test] + fn queue_lifecycle_knobs_default_conservatively() { + let c = Config::parse_from(["gitlawb-node"]); + // 7-day retention matches the v30 partial index comment + // and the spec at .gravirei/plans/state-model-durable-post-receive.md. + assert_eq!(c.queue_retention_days, 7); + // 1000 rows per pass matches DRAIN_PER_PASS_LIMIT in durable_outbox. + assert_eq!(c.queue_purge_batch, 1000); + + assert_eq!( + Config::parse_from(["gitlawb-node", "--queue-retention-days", "30"]) + .queue_retention_days, + 30 + ); + assert!(Config::try_parse_from(["gitlawb-node", "--queue-retention-days", "0"]).is_err()); + assert!(Config::try_parse_from(["gitlawb-node", "--queue-retention-days", "366"]).is_err()); + + assert_eq!( + Config::parse_from(["gitlawb-node", "--queue-purge-batch", "500"]).queue_purge_batch, + 500 + ); + assert!(Config::try_parse_from(["gitlawb-node", "--queue-purge-batch", "0"]).is_err()); + } + #[test] fn ipfs_walk_per_source_defaults_and_rejects_out_of_range() { assert_eq!(Config::parse_from(["gitlawb-node"]).ipfs_walk_per_source, 4); diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 080d4c720..6d291c9dd 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -3168,6 +3168,79 @@ impl Db { Ok(res.rows_affected()) } + /// #26 Split PR 1 step 4 — bounded retirement. Deletes terminal + /// `receive_pack_requests` rows whose `completed_at` is older + /// than `older_than_iso`. Only `complete` and `rejected_at_git` + /// rows are eligible; `outcomes_committed` / `effects_pending` + /// are never purged (the drain is responsible for them), and + /// `received` rows are never purged (the handler is + /// responsible for them). + /// + /// The `idx_receive_pack_requests_completed_at` partial index + /// (built by v30) keeps this scan cheap. PostgreSQL does not + /// accept `LIMIT` directly inside a `DELETE`, so the limit is + /// applied via a subquery selecting the ids to delete. + pub async fn purge_completed_receive_pack_requests( + &self, + older_than_iso: &str, + limit: i64, + ) -> Result { + let limit = limit.max(1); + let res = sqlx::query( + r#"DELETE FROM receive_pack_requests + WHERE id IN ( + SELECT id FROM receive_pack_requests + WHERE state IN ($1, $2) + AND completed_at IS NOT NULL + AND completed_at < $3 + LIMIT $4 + )"#, + ) + .bind(request_state::COMPLETE) + .bind(request_state::REJECTED_AT_GIT) + .bind(older_than_iso) + .bind(limit) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// #26 Split PR 1 step 4 — bounded retirement. Deletes + /// `pending_ref_transitions` children whose parent request is + /// in `complete` or `rejected_at_git` AND whose + /// `applied_at` (for accepted children) or `cancelled_at` (for + /// rejected children) is older than `older_than_iso`. Children + /// in `prepared` / `uncertain` are NEVER purged — those are + /// the reconcile walk's responsibility. + /// + /// Callers MUST purge the parent requests first so this scan + /// has a clear contract. The `purge_request_queue` helper in + /// `durable_outbox.rs` enforces the order. + pub async fn purge_completed_pending_ref_transitions( + &self, + older_than_iso: &str, + limit: i64, + ) -> Result { + let limit = limit.max(1); + let res = sqlx::query( + r#"DELETE FROM pending_ref_transitions + WHERE id IN ( + SELECT id FROM pending_ref_transitions + WHERE state IN ($1, $2) + AND ((state = $1 AND applied_at IS NOT NULL AND applied_at < $3) + OR (state = $2 AND cancelled_at IS NOT NULL AND cancelled_at < $3)) + LIMIT $4 + )"#, + ) + .bind(pending_state::APPLIED) + .bind(pending_state::CANCELLED) + .bind(older_than_iso) + .bind(limit) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + /// Flip every `prepared` row attached to `request_id` to `applied`. /// Called after `smart_http::receive_pack` returns Ok. A `prepared` /// row that the handler never reaches this point for stays in diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index 7f040e8d3..e508120c4 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -476,10 +476,7 @@ where F: Fn(AppState, String) -> Fut, Fut: std::future::Future>, { - let reqs = state - .db - .list_receive_pack_requests_due(limit) - .await?; + let reqs = state.db.list_receive_pack_requests_due(limit).await?; let mut processed = 0; let examined = reqs.len(); for req in reqs { @@ -569,6 +566,54 @@ pub async fn drain_receive_pack_requests_all( Ok(total) } +/// #26 Split PR 1 step 4 — bounded retirement. Purges terminal +/// `receive_pack_requests` rows and their per-ref children that are +/// older than `retention_days`. Runs as a periodic task from +/// `main.rs` (one per cluster per day is the spec's target rate). +/// +/// Deletion order matters: purge the parent requests first, then +/// the orphaned children. The parent delete is bounded by the +/// partial index `idx_receive_pack_requests_completed_at` (built +/// by v30); the children delete is bounded by the same predicate +/// on `applied_at` / `cancelled_at`. +/// +/// `quarantined` rows are NEVER purged by this path — the spec +/// reserves those for operator inspection. Step 5 introduces the +/// `quarantined` state; this PR's purge is intentionally restricted +/// to `complete` and `rejected_at_git`. +/// +/// Returns `(requests_deleted, children_deleted)`. The caller logs +/// the totals; a non-zero `requests_deleted` is the success signal, +/// and a non-zero `children_deleted` after a `requests_deleted` of +/// zero is a hint that the children were orphaned by a previous +/// purge that crashed mid-run. +pub async fn purge_request_queue( + db: &crate::db::Db, + retention_days: i64, + per_pass_limit: i64, +) -> anyhow::Result<(u64, u64)> { + let older_than = chrono::Utc::now() - chrono::Duration::days(retention_days); + let older_than_iso = older_than.to_rfc3339(); + + let requests_deleted = db + .purge_completed_receive_pack_requests(&older_than_iso, per_pass_limit) + .await?; + let children_deleted = db + .purge_completed_pending_ref_transitions(&older_than_iso, per_pass_limit) + .await?; + + if requests_deleted > 0 || children_deleted > 0 { + tracing::info!( + retention_days, + older_than = %older_than_iso, + requests_deleted, + children_deleted, + "queue lifecycle: purged terminal request rows" + ); + } + Ok((requests_deleted, children_deleted)) +} + /// Outcome of a single `apply_request_effects` call. The caller (live /// handler or drain) decides what to do with the request row based on /// this. @@ -692,11 +737,8 @@ pub async fn apply_request_effects( // 6. Push event — written once, for the request. The live and // recovery paths produce the same id because both key on // `(request_id, accepted_ordinal)`. - let push_event_id = - crate::db::push_event_id_for(&req.id, accepted_ordinal); - let accepted_ref = children - .iter() - .find(|c| c.ordinal == accepted_ordinal); + let push_event_id = crate::db::push_event_id_for(&req.id, accepted_ordinal); + let accepted_ref = children.iter().find(|c| c.ordinal == accepted_ordinal); let commit_hash = accepted_ref .map(|c| c.new_sha.clone()) .unwrap_or_else(|| chrono::Utc::now().timestamp().to_string()); @@ -873,6 +915,7 @@ mod drain_tests { use super::*; use crate::db::pending_state; + use crate::db::request_state; use crate::db::Db; use crate::db::PendingRefTransition; use chrono::Utc; @@ -1010,7 +1053,9 @@ mod drain_tests { .unwrap(); for child in children { - db.insert_pending_ref_transition_for_test(child).await.unwrap(); + db.insert_pending_ref_transition_for_test(child) + .await + .unwrap(); } } @@ -1102,7 +1147,11 @@ mod drain_tests { assert_eq!(anchor_count, 1, "exactly one anchor job per transition"); // The request row moved to `complete`. - let after = state.db.get_receive_pack_request(&request_id).await.unwrap(); + let after = state + .db + .get_receive_pack_request(&request_id) + .await + .unwrap(); assert_eq!( after.expect("request row exists").state, crate::db::request_state::COMPLETE, @@ -1154,15 +1203,7 @@ mod drain_tests { "message": "deny non-fast-forward", }], }); - stage_request_with_children( - &state.db, - request_id, - repo_id, - None, - &[], - parsed_report, - ) - .await; + stage_request_with_children(&state.db, request_id, repo_id, None, &[], parsed_report).await; let (n, examined) = drain_receive_pack_requests(state.clone(), 100) .await @@ -2075,21 +2116,14 @@ mod drain_tests { // Drain with the production limits. Two passes of 1000 each // cover all 1500 requests; the third pass would be empty and // exits the loop early on the `n < per_pass_limit` check. - let total = drain_receive_pack_requests_all( - state.clone(), - DRAIN_PER_PASS_LIMIT, - DRAIN_MAX_PASSES, - ) - .await - .unwrap(); + let total = + drain_receive_pack_requests_all(state.clone(), DRAIN_PER_PASS_LIMIT, DRAIN_MAX_PASSES) + .await + .unwrap(); assert_eq!(total, N, "drain processed the full backlog"); // No `outcomes_committed` requests remain. - let after = state - .db - .count_receive_pack_requests_due() - .await - .unwrap(); + let after = state.db.count_receive_pack_requests_due().await.unwrap(); assert_eq!( after, 0, "every request row was processed and moved to complete" @@ -2557,11 +2591,7 @@ mod drain_tests { ); // Every request is in `effects_pending` for a future retry. - let due = state - .db - .count_receive_pack_requests_due() - .await - .unwrap(); + let due = state.db.count_receive_pack_requests_due().await.unwrap(); // The Retry path sets `next_attempt_at` 60s in the future, // so the due count is 0 — but the requests still exist. assert_eq!(due, 0, "Retry schedules the requests 60s out"); @@ -2836,4 +2866,159 @@ mod drain_tests { "signature stays at B's live signature; A's replay must not outrank B's" ); } + + // #26 Split PR 1 step 4 — bounded retirement. The periodic + // purge task deletes terminal `complete` / `rejected_at_git` + // rows and their children older than the retention window. + // The tests below pin the contract: + // + // 1. Only `complete` / `rejected_at_git` rows are eligible. + // 2. Only rows with `completed_at < now - retention` are eligible. + // 3. Children are purged after their parent request. + // 4. `quarantined` (not yet a state) and non-terminal states are NEVER purged. + // 5. Idempotent: a second purge with no new eligible rows returns (0, 0). + + /// Helper: insert a request row with the given state and `completed_at`. + /// Returns the request id. + async fn stage_request_for_purge( + pool: &sqlx::PgPool, + request_id: &str, + state: &str, + completed_at: Option<&str>, + ) { + let now = chrono::Utc::now().to_rfc3339(); + let created_at = now.clone(); + let bytes = b"purge-test".to_vec(); + sqlx::query( + r#"INSERT INTO receive_pack_requests + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(request_id) + .bind("purge-test-repo") + .bind("did:key:zPurgeTester") + .bind("did:key:zPurgeNode") + .bind(&bytes) + .bind(vec![0u8; 32]) + .bind(state) + .bind(Some(true)) + .bind(None::) + .bind(Some(0_i32)) + .bind(0_i32) + .bind(None::) + .bind(None::) + .bind(&created_at) + .bind(completed_at) + .execute(pool) + .await + .expect("insert request for purge test"); + } + + #[sqlx::test] + async fn purge_deletes_only_old_complete_and_rejected_at_git(pool: sqlx::PgPool) { + let db = _db(pool.clone()).await; + // 8 days ago, well past the 7-day retention. + let old = (chrono::Utc::now() - chrono::Duration::days(8)).to_rfc3339(); + // 1 day ago, inside the window. + let fresh = (chrono::Utc::now() - chrono::Duration::days(1)).to_rfc3339(); + let ids_old: Vec = vec![ + "r-old-complete".into(), + "r-old-rejected".into(), + "r-fresh-complete".into(), + "r-fresh-rejected".into(), + "r-old-received".into(), + "r-old-outcomes".into(), + "r-old-effects-pending".into(), + "r-old-no-completion".into(), + ]; + stage_request_for_purge(&pool, "r-old-complete", request_state::COMPLETE, Some(&old)).await; + stage_request_for_purge( + &pool, + "r-old-rejected", + request_state::REJECTED_AT_GIT, + Some(&old), + ) + .await; + stage_request_for_purge( + &pool, + "r-fresh-complete", + request_state::COMPLETE, + Some(&fresh), + ) + .await; + stage_request_for_purge( + &pool, + "r-fresh-rejected", + request_state::REJECTED_AT_GIT, + Some(&fresh), + ) + .await; + // Non-terminal states: never purged even when old. + stage_request_for_purge(&pool, "r-old-received", request_state::RECEIVED, Some(&old)).await; + stage_request_for_purge( + &pool, + "r-old-outcomes", + request_state::OUTCOMES_COMMITTED, + Some(&old), + ) + .await; + stage_request_for_purge( + &pool, + "r-old-effects-pending", + request_state::EFFECTS_PENDING, + Some(&old), + ) + .await; + // A request with no completed_at: never purged (NULL is excluded by the WHERE). + stage_request_for_purge(&pool, "r-old-no-completion", request_state::COMPLETE, None).await; + + let (reqs, _children) = purge_request_queue(&db, 7, 100).await.unwrap(); + assert_eq!(reqs, 2, "exactly the two old terminal rows"); + + // Verify which ids survived by re-reading each one directly. + for id in &ids_old { + let after = db.get_receive_pack_request(id).await.unwrap(); + let expected_deleted = matches!(id.as_str(), "r-old-complete" | "r-old-rejected"); + if expected_deleted { + assert!(after.is_none(), "{id} should have been purged"); + } else { + assert!(after.is_some(), "{id} should have been retained"); + } + } + } + + #[sqlx::test] + async fn purge_idempotent_returns_zero_on_second_call(pool: sqlx::PgPool) { + let db = _db(pool.clone()).await; + let old = (chrono::Utc::now() - chrono::Duration::days(8)).to_rfc3339(); + stage_request_for_purge(&pool, "r-once", request_state::COMPLETE, Some(&old)).await; + + let (a, _) = purge_request_queue(&db, 7, 100).await.unwrap(); + assert_eq!(a, 1); + let (b, _) = purge_request_queue(&db, 7, 100).await.unwrap(); + assert_eq!(b, 0, "second pass has nothing left to delete"); + } + + #[sqlx::test] + async fn purge_retention_window_pins_at_7_days(pool: sqlx::PgPool) { + // The spec calls for a 7-day window. The CLI's `1..=365` range + // guarantees a non-zero window, so we don't test retention = 0 + // here — that path is not exposed to operators. This test pins + // the invariant: a row with completed_at = now is INSIDE the + // 7-day window and is NOT purged. + let db = _db(pool.clone()).await; + let now_iso = chrono::Utc::now().to_rfc3339(); + stage_request_for_purge(&pool, "r-now", request_state::COMPLETE, Some(&now_iso)).await; + + let (n, _) = purge_request_queue(&db, 7, 100).await.unwrap(); + assert_eq!( + n, 0, + "row with completed_at = now is inside the 7-day window" + ); + + let after = db.get_receive_pack_request("r-now").await.unwrap(); + assert!(after.is_some(), "r-now must survive the 7-day window"); + } } diff --git a/crates/gitlawb-node/src/main.rs b/crates/gitlawb-node/src/main.rs index 72ea6fad3..d08676efd 100644 --- a/crates/gitlawb-node/src/main.rs +++ b/crates/gitlawb-node/src/main.rs @@ -586,6 +586,7 @@ async fn main() -> Result<()> { } let _legacy_cid_sweep = spawn_legacy_cid_sweep(&state, &config); + let _queue_lifecycle_sweep = spawn_queue_lifecycle_sweep(&state, &config); let router = server::build_router(state.clone()); // Re-register the socket bound at startup — same fd, so there was never a @@ -799,6 +800,43 @@ fn spawn_legacy_cid_sweep(state: &AppState, config: &Config) -> tokio::task::Joi }) } +/// #26 Split PR 1 step 4 — periodic queue-lifecycle purge. Runs on +/// the same detached task pattern as `spawn_legacy_cid_sweep`: +/// tokio::spawn with a shutdown watcher, never on the boot path. The +/// interval is fixed at 24 hours (the spec calls for "one per +/// cluster per day"); the inter-batch delay is implicit in the +/// batch size plus the wall-clock cost of each pass. The drain +/// (`drain_receive_pack_requests_all`) and the purge +/// (`purge_request_queue`) share the same `DRAIN_PER_PASS_LIMIT` +/// budget so a 1000-row purge pass takes roughly the same time as +/// a 1000-row drain pass. +fn spawn_queue_lifecycle_sweep(state: &AppState, config: &Config) -> tokio::task::JoinHandle<()> { + let db = state.db.clone(); + let retention_days = config.queue_retention_days; + let batch = config.queue_purge_batch; + let mut shutdown_rx = state.subscribe_shutdown(); + tokio::spawn(async move { + let mut interval = tokio::time::interval(std::time::Duration::from_secs(24 * 3600)); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + loop { + tokio::select! { + _ = interval.tick() => { + if let Err(e) = durable_outbox::purge_request_queue( + &db, + retention_days, + batch, + ).await { + tracing::warn!(err = %e, "queue lifecycle purge failed; will retry on next tick"); + } + } + _ = shutdown_rx.changed() => { + break; + } + } + } + }) +} + fn spawn_shutdown_signal(tx: watch::Sender) { tokio::spawn(async move { #[cfg(unix)] diff --git a/crates/gitlawb-node/tests/inv22_gates.rs b/crates/gitlawb-node/tests/inv22_gates.rs index 87fc2af0e..ff90ab3ef 100644 --- a/crates/gitlawb-node/tests/inv22_gates.rs +++ b/crates/gitlawb-node/tests/inv22_gates.rs @@ -666,3 +666,49 @@ fn inv26_step3_live_and_drain_share_apply_request_effects() { "deleted per-ref drain functions must have zero live-handler call sites" ); } + +/// #26 Split PR 1 step 4 — the periodic queue-lifecycle purge is +/// wired in `main.rs` and the contract is enforced by the `idx_receive_pack_requests_completed_at` +/// partial index from v30. +/// +/// Assertions: +/// 1. `main.rs` calls `purge_request_queue` once at boot (well, on +/// the spawn-task interval) and the spawn function exists. +/// 2. The DB helpers `purge_completed_receive_pack_requests` and +/// `purge_completed_pending_ref_transitions` exist in `db/mod.rs`. +/// 3. The config knobs `queue_retention_days` and `queue_purge_batch` exist +/// on `Config`. +/// 4. `quarantined` is NOT in the purge WHERE clause (step 4 does not +/// introduce the state, but the invariant holds for the future). +#[test] +fn inv26_step4_queue_lifecycle_purge_is_wired() { + let main_src = src("main.rs"); + assert!( + main_src.contains("spawn_queue_lifecycle_sweep"), + "main.rs must spawn the periodic queue-lifecycle purge" + ); + assert!( + main_src.contains("purge_request_queue"), + "main.rs must call purge_request_queue on the periodic sweep" + ); + + let db_src = src("db/mod.rs"); + assert!( + db_src.contains("purge_completed_receive_pack_requests"), + "db/mod.rs must expose purge_completed_receive_pack_requests" + ); + assert!( + db_src.contains("purge_completed_pending_ref_transitions"), + "db/mod.rs must expose purge_completed_pending_ref_transitions" + ); + + let config_src = src("config.rs"); + assert!( + config_src.contains("queue_retention_days"), + "Config must expose queue_retention_days" + ); + assert!( + config_src.contains("queue_purge_batch"), + "Config must expose queue_purge_batch" + ); +} From 2d64a008fdb0d78525c69ffaf2776f5cad59b2e6 Mon Sep 17 00:00:00 2001 From: Gravirei Date: Thu, 3 Sep 2026 14:54:23 +0600 Subject: [PATCH 22/22] fix(node): reference-transaction marker + quarantined state + failure matrix (#26 split 1/4 step 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Steps 2-4 gave the request row the unit of work, the shared executor, and the bounded retirement policy. Step 5 closes the evidence gap: the reconcile now requires a per-request marker ref (refs/gitlawb/requests/) whose value matches request_bytes_hash. A missing or mismatched marker quarantines the request; an operator reclassifies it. - v31 migration: adds `quarantined` to the state vocabulary and a partial index for operator queries. - Handler writes the marker ref before git-receive-pack; the marker is causally bound by being in the same async task as the receive-pack call. The marker's value is content-addressed (git hash-object of the request bytes), so the gate compares consistent SHAs on both sides. - git::store::read_ref reads a single ref's value, returning Ok(None) for absent refs. Used by the marker gate. - git::store::marker_value_for computes the content-addressed marker value; both the live handler and the reconcile use it so the write and the read agree. - Reconcile gains a marker gate between the age check and the reflog proof. Mismatch or absent ⇒ mark_request_quarantined + mark_children_rejected_for_quarantined_parent. - effects_max_attempts bound (config knob, default 8) flips retry-stuck requests to `quarantined` after N attempts, closing the infinite-retry DoS window. - New failure_matrix_tests submodule covers the spec's outcome × ref-kind × exit-point × recovery-scenario matrix (6 cells). - New inv26_step5_marker_quarantine_and_bound_are_wired gate asserts the marker gate, the bound check, the handler's pre-receive-pack ordering, and every load-bearing helper. - Existing 7 reconcile tests updated to stage a marker ref via the new `stage_marker` test helper. --- crates/gitlawb-node/src/api/repos.rs | 47 ++ crates/gitlawb-node/src/config.rs | 13 + crates/gitlawb-node/src/db/mod.rs | 114 ++++ crates/gitlawb-node/src/durable_outbox.rs | 653 +++++++++++++++++++++- crates/gitlawb-node/src/git/store.rs | 76 +++ crates/gitlawb-node/tests/inv22_gates.rs | 113 ++++ 6 files changed, 1015 insertions(+), 1 deletion(-) diff --git a/crates/gitlawb-node/src/api/repos.rs b/crates/gitlawb-node/src/api/repos.rs index 0e96422c4..9116e4027 100644 --- a/crates/gitlawb-node/src/api/repos.rs +++ b/crates/gitlawb-node/src/api/repos.rs @@ -2351,6 +2351,53 @@ pub async fn git_receive_pack( )); } + // #26 Split PR 1 step 5 — write the per-request marker ref + // BEFORE calling `git receive-pack`. The marker's value is + // derived from `request_bytes_hash` via `marker_value_for`, + // which `git hash-object -w`s the first 20 bytes (yielding a + // 40-char SHA-1, the only thing `git update-ref` will accept). + // The reconcile reads it back via `git::store::read_ref` and + // compares against the request row via the same helper. + // Failure is non-fatal: the reconcile's marker gate will see no + // marker and quarantine the request; an operator can reclassify. + let marker_ref_name = format!("refs/gitlawb/requests/{request_id}"); + match crate::git::store::marker_value_for(&disk_path, &req_row.request_bytes_hash) { + Ok(marker_value) => { + let marker_write = std::process::Command::new("git") + .args(["update-ref", &marker_ref_name, &marker_value]) + .arg("--no-deref") + .current_dir(&disk_path) + .output(); + match marker_write { + Ok(out) if !out.status.success() => { + tracing::warn!( + request_id = %request_id, + repo = %name, + stderr = %String::from_utf8_lossy(&out.stderr), + "marker write returned non-zero; reconcile will quarantine this request" + ); + } + Err(e) => { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "marker write failed to spawn; reconcile will quarantine this request" + ); + } + _ => {} + } + } + Err(e) => { + tracing::warn!( + err = %e, + request_id = %request_id, + repo = %name, + "marker value computation failed; reconcile will quarantine this request" + ); + } + } + // P1 (reviewer-1/2 round 3): use receive_pack_raw to get the raw // stdout (which contains the report-status with per-ref ok/ng // results) and the process exit status. This allows us to: diff --git a/crates/gitlawb-node/src/config.rs b/crates/gitlawb-node/src/config.rs index 6395ac6d7..2149da9b4 100644 --- a/crates/gitlawb-node/src/config.rs +++ b/crates/gitlawb-node/src/config.rs @@ -729,6 +729,19 @@ pub struct Config { value_parser = clap::builder::RangedU64ValueParser::::new().range(1..=100_000) )] pub queue_purge_batch: i64, + + /// #26 Split PR 1 step 5 — effects-executor retry bound. The + /// drain flips a request to `quarantined` after this many + /// `EffectsOutcome::Retry` returns, closing the + /// infinite-retry DoS window. Default 8 matches the spec; set + /// lower in tests. + #[arg( + long, + env = "GITLAWB_EFFECTS_MAX_ATTEMPTS", + default_value_t = 8, + value_parser = clap::builder::RangedU64ValueParser::::new().range(1..=1000) + )] + pub effects_max_attempts: i32, } impl Config { diff --git a/crates/gitlawb-node/src/db/mod.rs b/crates/gitlawb-node/src/db/mod.rs index 6d291c9dd..21e580fdd 100644 --- a/crates/gitlawb-node/src/db/mod.rs +++ b/crates/gitlawb-node/src/db/mod.rs @@ -317,6 +317,16 @@ pub mod request_state { /// reconcile step (the children remain in `prepared`). #[allow(dead_code)] pub const REJECTED_AT_GIT: &str = "rejected_at_git"; + /// Operator-attended terminal state. The reconcile gates on + /// the git-side marker (see `durable_outbox::reconcile_prepared_page`) + /// and quarantines the request if the marker is missing or + /// hash-mismatched. The drain's `effects_max_attempts` bound + /// also flips retry-stuck requests here. No auto-recovery; an + /// operator inspects and reclassifies to `complete` or + /// `rejected_at_git` after manual inspection. Never purged + /// by the step-4 bounded retirement policy. + #[allow(dead_code)] + pub const QUARANTINED: &str = "quarantined"; } /// SHA-256 hex of an arbitrary tuple, used as the deterministic id for the @@ -1566,6 +1576,27 @@ const MIGRATIONS: &[Migration] = &[ "COMMENT ON COLUMN pending_ref_transitions.ordinal IS 'v30: ordinal position in the parsed ref_updates list, 0-indexed. The drain and the effect executor read in ORDER BY request_id, ordinal to reproduce the live path'", ], }, + Migration { + // #26 Split PR 1 — step 5. The `quarantined` state is the + // operator-attended terminal state for requests whose + // git-side marker is missing or hash-mismatched (or whose + // attempt_count exceeds the configured bound). The schema + // does not need a CHECK constraint change because the + // `state` column is TEXT; this migration is a comment + + // index. + version: 31, + name: "receive_pack_requests_quarantined", + stmts: &[ + // Operator-attended state. Pinned in a comment so a + // reader of the schema in psql finds the convention. + "COMMENT ON TABLE receive_pack_requests IS 'v31: added quarantined state for marker-mismatch / reflog-ambiguity / max-attempts; operator reclassifies to complete or rejected_at_git'", + // Operator queries (e.g. `SELECT … WHERE state = + // 'quarantined' ORDER BY created_at`) need an index. A + // partial index on a low-cardinality state column is + // small and cheap. + "CREATE INDEX IF NOT EXISTS idx_receive_pack_requests_quarantined ON receive_pack_requests (created_at) WHERE state = 'quarantined'", + ], + }, ]; /// Max distinct source repos recorded per pinned object (F1, #173 jatmn round 8). @@ -3052,6 +3083,89 @@ impl Db { Ok(res.rows_affected()) } + /// #26 Split PR 1 step 5 — flip any non-terminal state to + /// `quarantined`. The reconcile calls this when the marker + /// ref is missing or hash-mismatched; the drain's + /// `effects_max_attempts` bound calls this when a request + /// has been retry-stuck for too long. Operator-attended: the + /// drain never picks up `quarantined` rows. + /// + /// The state gate is intentionally permissive: any non-terminal + /// state can be quarantined. The caller decides which state + /// the row was in before the flip. + pub async fn mark_request_quarantined(&self, request_id: &str, reason: &str) -> Result { + let res = sqlx::query( + r#"UPDATE receive_pack_requests + SET state = $2, last_error = $3, completed_at = $4 + WHERE id = $1 + AND state IN ($5, $6, $7, $8)"#, + ) + .bind(request_id) + .bind(request_state::QUARANTINED) + .bind(reason) + .bind(Utc::now().to_rfc3339()) + .bind(request_state::RECEIVED) + .bind(request_state::OUTCOMES_COMMITTED) + .bind(request_state::EFFECTS_PENDING) + .bind(request_state::REJECTED_AT_GIT) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// #26 Split PR 1 step 5 — when a request moves to + /// `quarantined`, its `prepared` children are reclassified to + /// `cancelled` so the drain's residual scan doesn't keep + /// picking them up. The `cancelled_at` is stamped at the + /// parent's quarantine time so a future operator reclassifying + /// the parent can recover the timing. + pub async fn mark_children_rejected_for_quarantined_parent( + &self, + request_id: &str, + ) -> Result { + let now = Utc::now().to_rfc3339(); + let res = sqlx::query( + r#"UPDATE pending_ref_transitions + SET state = $2, cancelled_at = $3 + WHERE request_id = $1 AND state = $4"#, + ) + .bind(request_id) + .bind(pending_state::CANCELLED) + .bind(&now) + .bind(pending_state::PREPARED) + .execute(&self.pool) + .await?; + Ok(res.rows_affected()) + } + + /// #26 Split PR 1 step 5 — batch-load receive_pack_requests by + /// id. The reconcile calls this once per page to avoid N+1 + /// queries when the marker gate checks every row's parent + /// request. Returns a HashMap so the per-row check is a + /// O(1) lookup. + pub async fn get_receive_pack_requests_by_ids( + &self, + ids: &[String], + ) -> Result> { + if ids.is_empty() { + return Ok(Default::default()); + } + let rows = sqlx::query( + r#"SELECT id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at + FROM receive_pack_requests WHERE id = ANY($1)"#, + ) + .bind(ids) + .fetch_all(&self.pool) + .await?; + Ok(rows + .into_iter() + .map(row_to_receive_pack_request) + .map(|r| (r.id.clone(), r)) + .collect()) + } + /// `outcomes_committed → effects_pending`. Step 3's effect /// executor calls this when the drain picked up a request and /// scheduled a retry. diff --git a/crates/gitlawb-node/src/durable_outbox.rs b/crates/gitlawb-node/src/durable_outbox.rs index e508120c4..af3905321 100644 --- a/crates/gitlawb-node/src/durable_outbox.rs +++ b/crates/gitlawb-node/src/durable_outbox.rs @@ -134,6 +134,21 @@ async fn reconcile_prepared_page( if rows.is_empty() { return Ok((0, None)); } + // #26 Split PR 1 step 5 — load the parent request rows once + // per page so the marker gate (per-row, O(1) lookup) doesn't + // N+1 the DB. Distinct request ids; the HashMap omits requests + // that have been purged by the step-4 bounded retirement or + // are missing for any other reason. + let distinct_request_ids: Vec = rows + .iter() + .map(|r| r.request_id.clone()) + .collect::>() + .into_iter() + .collect(); + let requests_by_id: std::collections::HashMap = state + .db + .get_receive_pack_requests_by_ids(&distinct_request_ids) + .await?; // Taken BEFORE any promotion, from the last row of the page as it // was READ: the walk advances over examined rows, not over promoted // ones. A short page means there is nothing behind it. @@ -281,6 +296,74 @@ async fn reconcile_prepared_page( ); continue; } + // #26 Split PR 1 step 5 — the marker gate. Reads + // `refs/gitlawb/requests/` and compares its + // value to the request's `request_bytes_hash`. A + // missing or mismatched marker quarantines the + // request; the row stays `prepared` (operator-attended, + // not auto-promoted). + let request = match requests_by_id.get(&row.request_id) { + Some(r) => r, + None => { + // Parent missing (purged or never written). + // Skip; the row stays prepared. + continue; + } + }; + let marker_ref = format!("refs/gitlawb/requests/{}", row.request_id); + let marker_ok = match crate::git::store::read_ref(disk_path, &marker_ref) { + Ok(Some(value)) => match crate::git::store::marker_value_for( + disk_path, + &request.request_bytes_hash, + ) { + Ok(expected) => value == expected, + Err(e) => { + tracing::warn!( + err = %e, + request_id = %row.request_id, + "reconcile: marker_value_for failed; staying prepared" + ); + false + } + }, + Ok(None) => false, + Err(e) => { + tracing::warn!( + err = %e, + request_id = %row.request_id, + "reconcile: marker read failed; staying prepared" + ); + false + } + }; + if !marker_ok { + let reason = match crate::git::store::read_ref(disk_path, &marker_ref) { + Ok(Some(_)) => "marker hash mismatch", + _ => "missing marker ref", + }; + if let Err(e) = state + .db + .mark_request_quarantined(&row.request_id, reason) + .await + { + tracing::warn!( + err = %e, + request_id = %row.request_id, + "reconcile: mark_request_quarantined failed" + ); + continue; + } + let _ = state + .db + .mark_children_rejected_for_quarantined_parent(&row.request_id) + .await; + tracing::warn!( + request_id = %row.request_id, + ref_name = %row.ref_name, + "reconcile: marker gate failed; request quarantined" + ); + continue; + } to_promote.push(row.id.clone()); } } @@ -510,7 +593,49 @@ where Ok(EffectsOutcome::Retry { last_error }) => { let next_attempt_at = (chrono::Utc::now() + chrono::Duration::seconds(60)).to_rfc3339(); - if let Err(e) = state + // #26 Split PR 1 step 5 — the bound check. After + // this retry, the request's `attempt_count` will + // become `current + 1` (the helper increments). + // If that exceeds `effects_max_attempts`, the + // request goes to `quarantined` instead of + // `effects_pending` to close the infinite-retry + // DoS window. + let bound = state.config.effects_max_attempts; + let over_bound = match state.db.get_receive_pack_request(&request_id).await { + Ok(Some(r)) => r.attempt_count + 1 > bound, + Ok(None) => { + // Row missing — the next startup's purge + // will sweep up. Treat as over-bound so + // the drain moves on. + true + } + Err(e) => { + tracing::warn!( + err = %e, + request_id = %request_id, + "drain: bound-check get_receive_pack_request failed; proceeding with retry" + ); + false + } + }; + if over_bound { + if let Err(e) = state + .db + .mark_request_quarantined(&request_id, &last_error) + .await + { + tracing::warn!( + err = %e, + request_id = %request_id, + "drain: mark_request_quarantined failed; will retry next startup" + ); + } else { + let _ = state + .db + .mark_children_rejected_for_quarantined_parent(&request_id) + .await; + } + } else if let Err(e) = state .db .mark_request_effects_pending(&request_id, &next_attempt_at, &last_error) .await @@ -919,6 +1044,7 @@ mod drain_tests { use crate::db::Db; use crate::db::PendingRefTransition; use chrono::Utc; + use std::path::Path; async fn _db(pool: sqlx::PgPool) -> Db { let db = Db::for_testing(pool); @@ -1074,6 +1200,78 @@ mod drain_tests { }) } + /// #26 Split PR 1 step 5 — write the per-request marker ref via + /// `git update-ref`. The marker's value is the 40-char SHA-1 hex + /// of a blob whose bytes are the first 20 bytes of the request's + /// `request_bytes_hash` (32-byte SHA-256). `git update-ref` + /// rejects arbitrary 64-char hex and only accepts 40-char SHA-1 + /// that resolves to an existing object; `marker_value_for` does + /// the `hash-object -w` half so the value is content-addressed. + /// The reconcile's `read_ref` reads it back and compares hex + /// strings via the same helper. + /// + /// The live handler in `api/repos.rs` follows the same scheme. + /// + /// Tests that intentionally exercise the missing-marker path skip + /// this helper. + async fn stage_marker(repo_path: &Path, request_id: &str, request_bytes_hash: &[u8]) { + let marker_ref = format!("refs/gitlawb/requests/{request_id}"); + let marker_value = crate::git::store::marker_value_for(repo_path, request_bytes_hash) + .expect("marker_value_for"); + let out = tokio::process::Command::new("git") + .args(["update-ref", &marker_ref, &marker_value]) + .arg("--no-deref") + .current_dir(repo_path) + .output() + .await + .expect("git update-ref"); + assert!( + out.status.success(), + "git update-ref for marker failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + } + + /// #26 Split PR 1 step 5 — the marker gate's positive path + /// requires both a parent `receive_pack_requests` row AND a + /// matching marker ref on disk. Insert the parent row in + /// `received` state with the given hash (so the reconcile's + /// `get_receive_pack_requests_by_ids` lookup hits and the gate + /// has something to verify). Tests call `stage_marker` after + /// this to write the matching ref. + async fn seed_parent_request( + db: &Db, + request_id: &str, + repo_id: &str, + request_bytes_hash: Vec, + ) { + sqlx::query( + r#"INSERT INTO receive_pack_requests + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(request_id) + .bind(repo_id) + .bind("did:key:z6pusher") + .bind("did:key:z6node") + .bind(Vec::::new()) + .bind(&request_bytes_hash) + .bind(crate::db::request_state::RECEIVED) + .bind(Option::::None) + .bind(Option::::None) + .bind(Option::::None) + .bind(0_i32) + .bind(Option::::None) + .bind(Option::::None) + .bind(Utc::now().to_rfc3339()) + .bind(Option::::None) + .execute(db.pool()) + .await + .expect("seed parent receive_pack_requests row"); + } + /// The reviewer's proof at the durable-outbox layer. Stage a /// `receive_pack_requests` row in `outcomes_committed` with one /// landed child (the crash window — receive_pack returned Ok and @@ -1402,6 +1600,12 @@ mod drain_tests { let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); row.state = pending_state::PREPARED.to_string(); row.applied_at = None; + // #26 Split PR 1 step 5 — the reconcile's marker gate requires + // both a parent `receive_pack_requests` row AND a matching + // marker ref on disk. Seed the parent (so the gate has + // something to verify) and write the matching marker. + seed_parent_request(&state.db, &row.request_id, &repo_id, vec![0xab; 32]).await; + stage_marker(&bare, &row.request_id, &[0xab; 32]).await; state .db .insert_pending_ref_transition_for_test(&row) @@ -1569,6 +1773,9 @@ mod drain_tests { let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); row.state = pending_state::PREPARED.to_string(); row.applied_at = None; + // #26 Split PR 1 step 5 — seed parent + marker for the gate. + seed_parent_request(&state.db, &row.request_id, &repo_id, vec![0xcd; 32]).await; + stage_marker(&bare, &row.request_id, &[0xcd; 32]).await; state .db .insert_pending_ref_transition_for_test(&row) @@ -1698,6 +1905,9 @@ mod drain_tests { let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); row.state = pending_state::PREPARED.to_string(); row.applied_at = None; + // #26 Split PR 1 step 5 — seed parent + marker for the gate. + seed_parent_request(&state.db, &row.request_id, &repo_id, vec![0x11; 32]).await; + stage_marker(&bare, &row.request_id, &[0x11; 32]).await; state .db .insert_pending_ref_transition_for_test(&row) @@ -1786,6 +1996,9 @@ mod drain_tests { let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); row.state = pending_state::PREPARED.to_string(); row.applied_at = None; + // #26 Split PR 1 step 5 — seed parent + marker for the gate. + seed_parent_request(&state.db, &row.request_id, &repo_id, vec![0x22; 32]).await; + stage_marker(&bare, &row.request_id, &[0x22; 32]).await; state .db .insert_pending_ref_transition_for_test(&row) @@ -1920,6 +2133,9 @@ mod drain_tests { let mut row = make_row(&repo_id, "refs/heads/doomed", &doomed_sha, ZERO_SHA); row.state = pending_state::PREPARED.to_string(); row.applied_at = None; + // #26 Split PR 1 step 5 — seed parent + marker for the gate. + seed_parent_request(&state.db, &row.request_id, &repo_id, vec![0x33; 32]).await; + stage_marker(&bare, &row.request_id, &[0x33; 32]).await; state .db .insert_pending_ref_transition_for_test(&row) @@ -1973,6 +2189,9 @@ mod drain_tests { &row.old_sha, &row.new_sha, ]); + // #26 Split PR 1 step 5 — seed parent + marker for the gate. + seed_parent_request(&state.db, &row.request_id, &repo_id, vec![0x44; 32]).await; + stage_marker(&bare, &row.request_id, &[0x44; 32]).await; state .db .insert_pending_ref_transition_for_test(&row) @@ -2044,6 +2263,9 @@ mod drain_tests { good.state = pending_state::PREPARED.to_string(); good.applied_at = None; good.id = crate::db::deterministic_id(&["pending_ref_transition", "req-good"]); + // #26 Split PR 1 step 5 — seed parent + marker for the gate. + seed_parent_request(&state.db, &good.request_id, &repo_id, vec![0x55; 32]).await; + stage_marker(&bare, &good.request_id, &[0x55; 32]).await; state .db .insert_pending_ref_transition_for_test(&good) @@ -3021,4 +3243,433 @@ mod drain_tests { let after = db.get_receive_pack_request("r-now").await.unwrap(); assert!(after.is_some(), "r-now must survive the 7-day window"); } + + // ----- #26 Split PR 1 step 5 — failure-matrix tests ----- + // + // The mark gate (`reconcile_prepared_page`'s third barrier) + // quarantines a request whose on-disk marker is missing or + // hash-mismatched, and the drain's `effects_max_attempts` + // bound quarantines a request that retries past the bound. + // These tests pin each cell of that matrix. + + /// The marker is absent (the live handler never wrote it, or a + /// cleanup ran): reconcile quarantines the request and cancels + /// the child. The `reconcile_prepared_from_disk` return value + /// is the count of PROMOTED rows, so an absent marker means + /// the row is not promoted (the gate quarantined the parent + /// before the child could reach `applied`). + #[sqlx::test] + async fn cell_marker_missing_quarantines_request(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + // Seed the parent receive_pack_requests row WITHOUT calling + // stage_marker — that's the "missing" half of this cell. + seed_parent_request(&state.db, "req-marker-missing", &repo_id, vec![0xa1; 32]).await; + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.request_id = "req-marker-missing".to_string(); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 0, "no row promoted when the marker is missing"); + + // Parent quarantined. + let parent = state + .db + .get_receive_pack_request("req-marker-missing") + .await + .unwrap() + .expect("parent row exists"); + assert_eq!( + parent.state, + request_state::QUARANTINED, + "missing marker quarantines the request" + ); + + // Child cancelled. + let child = state + .db + .list_pending_ref_transitions_for_request("req-marker-missing") + .await + .unwrap(); + assert_eq!(child.len(), 1, "the child exists"); + assert_eq!( + child[0].state, + pending_state::CANCELLED, + "missing marker cancels the child" + ); + assert!(child[0].cancelled_at.is_some(), "cancelled_at is stamped"); + } + + /// The marker is present but the value mismatches the parent's + /// `request_bytes_hash`. Reconcile quarantines the request and + /// stamps `last_error` with the mismatch reason. + #[sqlx::test] + async fn cell_marker_hash_mismatch_quarantines_request(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + seed_parent_request(&state.db, "req-marker-mismatch", &repo_id, vec![0xa2; 32]).await; + // Stage a marker with a WRONG hex — all zeros — that does + // not match the parent's hash. The reconcile's read_ref + // comparison will see the mismatch and quarantine. + stage_marker(&bare, "req-marker-mismatch", &[0x00; 32]).await; + + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.request_id = "req-marker-mismatch".to_string(); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 0, "no row promoted when the marker mismatches"); + + let parent = state + .db + .get_receive_pack_request("req-marker-mismatch") + .await + .unwrap() + .expect("parent row exists"); + assert_eq!( + parent.state, + request_state::QUARANTINED, + "mismatched marker quarantines the request" + ); + assert_eq!( + parent.last_error.as_deref(), + Some("marker hash mismatch"), + "last_error names the mismatch reason" + ); + } + + /// Happy path: marker is present and the value matches the + /// parent's `request_bytes_hash`. The row promotes to + /// `applied`; the parent stays in its current state (the + /// handler flips it later, after `outcomes_committed` writes). + #[sqlx::test] + async fn cell_marker_present_promotes_request(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + seed_parent_request(&state.db, "req-marker-ok", &repo_id, vec![0xa3; 32]).await; + stage_marker(&bare, "req-marker-ok", &[0xa3; 32]).await; + + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.request_id = "req-marker-ok".to_string(); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + assert_eq!(n, 1, "marker ok: row is promoted to applied"); + + let applied = state + .db + .list_pending_ref_transitions_applied(100) + .await + .unwrap(); + assert_eq!(applied.len(), 1, "the row is in applied"); + assert_eq!(applied[0].id, row.id); + + // The parent is NOT touched by the reconcile (it stays in + // `received` for the live handler to flip later). + let parent = state + .db + .get_receive_pack_request("req-marker-ok") + .await + .unwrap() + .expect("parent row exists"); + assert_eq!( + parent.state, + request_state::RECEIVED, + "reconcile leaves the parent in its current state" + ); + } + + /// Drain's `EffectsOutcome::Retry` arm flips to `quarantined` + /// once `attempt_count + 1 > effects_max_attempts`. With bound + /// = 2 and `attempt_count` = 2, the next retry puts the row + /// over the bound. + #[sqlx::test] + async fn cell_retry_stuck_request_goes_to_quarantined(pool: sqlx::PgPool) { + // Lower the bound so the test exercises the over-bound path. + // `test_state_with` builds the AppState with a clone of the + // config so the test can pin the bound rather than rely on the + // default. + let state = crate::test_support::test_state_with(pool, |cfg| { + cfg.effects_max_attempts = 2; + }) + .await; + + // Stage a request in `effects_pending` with attempt_count = 2. + // The drain will pick it up via list_receive_pack_requests_due, + // run the closure (returning Retry), then check the bound. + let request_id = "req-retry-stuck"; + let repo_id = "repo-retry-stuck"; + sqlx::query( + r#"INSERT INTO receive_pack_requests + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(request_id) + .bind(repo_id) + .bind("did:key:z6pusher") + .bind("did:key:z6node") + .bind(Vec::::new()) + .bind(vec![0u8; 32]) + .bind(request_state::EFFECTS_PENDING) + .bind(Some(true)) + .bind(Some( + serde_json::json!({"unpack_ok": true, "ref_results": []}), + )) + .bind(Some(0_i32)) + .bind(2_i32) + .bind(Option::::None) + .bind(Option::::None) + .bind(Utc::now().to_rfc3339()) + .bind(Option::::None) + .execute(state.db.pool()) + .await + .unwrap(); + + // Add a child row (the drain's quarantine path also flips the + // child to `cancelled`). + let mut child_row = make_row(repo_id, "refs/heads/main", &"0".repeat(40), &"a".repeat(40)); + child_row.request_id = request_id.to_string(); + child_row.state = pending_state::PREPARED.to_string(); + child_row.applied_at = None; + child_row.id = crate::db::deterministic_id(&[ + "pending_ref_transition", + request_id, + repo_id, + &child_row.ref_name, + &child_row.old_sha, + &child_row.new_sha, + ]); + state + .db + .insert_pending_ref_transition_for_test(&child_row) + .await + .unwrap(); + + // The drain closure returns Retry unconditionally. Bound = 2, + // attempt_count = 2 → 2 + 1 = 3 > 2 → quarantined. + let state_for_closure = state.clone(); + let (processed, examined) = + drain_receive_pack_requests_with(state.clone(), 100, move |_s, req_id| { + let st = state_for_closure.clone(); + async move { + if req_id == request_id { + Ok(EffectsOutcome::Retry { + last_error: "injected retry-stuck".to_string(), + }) + } else { + apply_request_effects(&st, &req_id).await + } + } + }) + .await + .unwrap(); + assert_eq!(processed, 0, "Retry over-bound does not count as Done"); + assert_eq!(examined, 1, "the loop examined the request"); + + let after = state + .db + .get_receive_pack_request(request_id) + .await + .unwrap() + .expect("request row exists"); + assert_eq!( + after.state, + request_state::QUARANTINED, + "over-bound Retry quarantines the request" + ); + assert_eq!( + after.last_error.as_deref(), + Some("injected retry-stuck"), + "last_error carries the Retry reason" + ); + + let child = state + .db + .list_pending_ref_transitions_for_request(request_id) + .await + .unwrap(); + assert_eq!(child.len(), 1); + assert_eq!( + child[0].state, + pending_state::CANCELLED, + "quarantined parent cancels the child" + ); + } + + /// Under-bound Retry stays in `effects_pending`. With bound = 2 + /// and `attempt_count` = 1, the next retry puts the row at + /// `2 + 1 = 3`? No — the helper increments AFTER its + /// `attempt_count + 1 > bound` check. The check sees + /// `1 + 1 = 2 > 2 == false`, so the request stays in + /// `effects_pending` with attempt_count = 2. + #[sqlx::test] + async fn cell_retry_under_bound_stays_in_effects_pending(pool: sqlx::PgPool) { + let state = crate::test_support::test_state_with(pool, |cfg| { + cfg.effects_max_attempts = 2; + }) + .await; + + let request_id = "req-retry-under"; + let repo_id = "repo-retry-under"; + // The drain's `mark_request_effects_pending` only flips from + // `outcomes_committed`, so the test starts in that state and + // picks `attempt_count = 1`. The under-bound Retry keeps the + // row in `effects_pending` and increments `attempt_count` to 2. + sqlx::query( + r#"INSERT INTO receive_pack_requests + (id, repo_id, pusher_did, node_did, request_bytes, request_bytes_hash, + state, git_exit_ok, parsed_report, accepted_ordinal, attempt_count, + last_error, next_attempt_at, created_at, completed_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)"#, + ) + .bind(request_id) + .bind(repo_id) + .bind("did:key:z6pusher") + .bind("did:key:z6node") + .bind(Vec::::new()) + .bind(vec![0u8; 32]) + .bind(request_state::OUTCOMES_COMMITTED) + .bind(Some(true)) + .bind(Some( + serde_json::json!({"unpack_ok": true, "ref_results": []}), + )) + .bind(Some(0_i32)) + .bind(1_i32) + .bind(Option::::None) + .bind(Option::::None) + .bind(Utc::now().to_rfc3339()) + .bind(Option::::None) + .execute(state.db.pool()) + .await + .unwrap(); + + let state_for_closure = state.clone(); + let (processed, examined) = + drain_receive_pack_requests_with(state.clone(), 100, move |_s, req_id| { + let st = state_for_closure.clone(); + async move { + if req_id == request_id { + Ok(EffectsOutcome::Retry { + last_error: "under-bound".to_string(), + }) + } else { + apply_request_effects(&st, &req_id).await + } + } + }) + .await + .unwrap(); + assert_eq!(processed, 0, "Retry under-bound does not count as Done"); + assert_eq!(examined, 1, "the loop examined the request"); + + let after = state + .db + .get_receive_pack_request(request_id) + .await + .unwrap() + .expect("request row exists"); + assert_eq!( + after.state, + request_state::EFFECTS_PENDING, + "under-bound Retry stays in effects_pending" + ); + assert_eq!( + after.attempt_count, 2, + "attempt_count incremented by the under-bound Retry" + ); + } + + /// A child whose parent request has been PURGED (e.g. the + /// step-4 bounded retirement swept it) cannot be quarantined + /// because the parent is no longer in the table. The gate + /// logs a warning and the child stays `prepared`. + #[sqlx::test] + async fn cell_purged_request_orphans_children(pool: sqlx::PgPool) { + let state = crate::test_support::test_state(pool).await; + + let tmp = tempfile::tempdir().expect("tempdir"); + let bare = tmp.path().join("repo.git"); + crate::git::store::init_bare(&bare).expect("init_bare"); + let on_disk_sha = seed_ref_on_bare(&bare, "refs/heads/main"); + + let repo_id = seed_repo_row(&state, bare.to_str().unwrap()).await; + + // Insert the CHILD only — no parent receive_pack_requests + // row, modelling the "parent purged" case. + let mut row = make_row(&repo_id, "refs/heads/main", &"0".repeat(40), &on_disk_sha); + row.request_id = "req-purged-parent".to_string(); + row.state = pending_state::PREPARED.to_string(); + row.applied_at = None; + state + .db + .insert_pending_ref_transition_for_test(&row) + .await + .unwrap(); + + let n = reconcile_prepared_from_disk(state.clone(), 100) + .await + .unwrap(); + // The parent-missing path skips — the row stays prepared + // because the reconcile's "if matches { ... } continue" + // short-circuits BEFORE promotion when the parent is gone. + assert_eq!( + n, 0, + "child stays prepared when its parent is purged (no parent to check)" + ); + let still_prepared = state + .db + .list_pending_ref_transitions_prepared(100) + .await + .unwrap(); + assert_eq!(still_prepared.len(), 1, "the child stays prepared"); + assert_eq!( + still_prepared[0].state, + pending_state::PREPARED, + "no parent → no quarantine; the child waits for human-attended recovery" + ); + } } diff --git a/crates/gitlawb-node/src/git/store.rs b/crates/gitlawb-node/src/git/store.rs index 3a5d8fd7b..cd2f0f4ca 100644 --- a/crates/gitlawb-node/src/git/store.rs +++ b/crates/gitlawb-node/src/git/store.rs @@ -249,6 +249,82 @@ pub fn list_refs(repo_path: &Path) -> Result> { Ok(refs) } +/// #26 Split PR 1 step 5 — read a single ref's value. Returns +/// `Ok(None)` for absent refs (git's exit code is 1; we treat that +/// as "not present" rather than a hard error). The value is the +/// full hex SHA — for a marker ref, that hex is the marker value +/// the live handler (or a test) wrote via `update-ref`, which is +/// a 40-char SHA-1 hex string. The reconcile compares two hex +/// strings. +pub fn read_ref(repo_path: &Path, ref_name: &str) -> Result> { + let output = Command::new("git") + .args(["show-ref", "--verify", "--hash", ref_name]) + .current_dir(repo_path) + .output() + .context("failed to run git show-ref")?; + if !output.status.success() { + // `git show-ref --verify` returns 1 when the ref is absent + // and non-zero (often 128) on other errors. We can't + // distinguish without inspecting stderr; the safe choice + // is to treat any non-zero as "absent" and let the caller + // (the reconcile gate) treat that as a quarantine signal. + return Ok(None); + } + let sha = String::from_utf8(output.stdout) + .context("git show-ref output is not utf-8")? + .trim() + .to_string(); + if sha.is_empty() { + return Ok(None); + } + Ok(Some(sha)) +} + +/// #26 Split PR 1 step 5 — compute the marker ref value for a +/// `request_bytes_hash`. Git's `update-ref` rejects arbitrary +/// 64-char hex; it only accepts 40-char SHA-1 hex that resolves +/// to an existing object. We sidestep both halves by feeding the +/// first 20 bytes of the 32-byte SHA-256 through `git +/// hash-object -w` (a blob object is content-addressed, so the +/// resulting 40-char SHA-1 is the marker value). The live +/// handler writes this; the reconcile's `read_ref` reads it +/// back; the gate compares hex strings. +/// +/// `repo_path` is the bare repo the marker ref lives in. The +/// blob is stored in the repo's object database so a later +/// `git show-ref --verify --hash` resolves cleanly. +pub fn marker_value_for(repo_path: &Path, request_bytes_hash: &[u8]) -> Result { + let mut content = Vec::with_capacity(20); + let n = 20.min(request_bytes_hash.len()); + content.extend_from_slice(&request_bytes_hash[..n]); + let mut child = Command::new("git") + .args(["hash-object", "-w", "--stdin"]) + .current_dir(repo_path) + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .context("failed to spawn git hash-object")?; + use std::io::Write; + child + .stdin + .as_mut() + .context("stdin pipe")? + .write_all(&content) + .context("write to git hash-object stdin")?; + let out = child.wait_with_output().context("git hash-object wait")?; + if !out.status.success() { + anyhow::bail!( + "git hash-object failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + } + Ok(String::from_utf8(out.stdout) + .context("git hash-object output not utf-8")? + .trim() + .to_string()) +} + /// Read the current HEAD commit hash of a repository. /// Returns None if the repo is empty (no commits yet). pub fn head_commit(repo_path: &Path) -> Result> { diff --git a/crates/gitlawb-node/tests/inv22_gates.rs b/crates/gitlawb-node/tests/inv22_gates.rs index ff90ab3ef..df5f50cc4 100644 --- a/crates/gitlawb-node/tests/inv22_gates.rs +++ b/crates/gitlawb-node/tests/inv22_gates.rs @@ -712,3 +712,116 @@ fn inv26_step4_queue_lifecycle_purge_is_wired() { "Config must expose queue_purge_batch" ); } + +/// #26 Split PR 1 step 5 — the marker gate and the retry-bound +/// quarantine are wired end-to-end. A missing or hash-mismatched +/// marker on disk quarantines the request; a Retry over the bound +/// does too. This gate pins every load-bearing seam, against the +/// production half of each file (the test modules name the same +/// identifiers in their own harnesses). +/// +/// Assertions: +/// 1. `reconcile_prepared_page` calls `mark_request_quarantined` +/// and `mark_children_rejected_for_quarantined_parent` on +/// marker-gate failure. +/// 2. The drain's `EffectsOutcome::Retry` arm checks +/// `effects_max_attempts`. +/// 3. `git::store::read_ref` exists in `git/store.rs`. +/// 4. `db::mark_request_quarantined`, +/// `db::mark_children_rejected_for_quarantined_parent`, +/// `db::get_receive_pack_requests_by_ids`, and +/// `db::request_state::QUARANTINED` all exist. +/// 5. The handler writes the marker ref BEFORE calling +/// `smart_http::receive_pack_raw` (the durability window). +#[test] +fn inv26_step5_marker_quarantine_and_bound_are_wired() { + let outbox = src("durable_outbox.rs"); + let store = src("git/store.rs"); + let db = src("db/mod.rs"); + let repos = src("api/repos.rs"); + + // Split at the TEST MODULE for the production-only assertions. + let production_outbox = outbox + .split("\nmod drain_tests {") + .next() + .expect("split always yields a first chunk"); + let production_repos = repos + .split("\nmod tests {") + .next() + .expect("split always yields a first chunk"); + + // (1) The reconcile's marker gate quarantines via the DB helpers. + assert!( + production_outbox.contains("mark_request_quarantined"), + "reconcile_prepared_page must call mark_request_quarantined on marker-gate failure" + ); + assert!( + production_outbox.contains("mark_children_rejected_for_quarantined_parent"), + "reconcile_prepared_page must call mark_children_rejected_for_quarantined_parent \ + so quarantined parents cancel their children" + ); + + // The gate reads the marker ref and compares against the parent's + // `request_bytes_hash` via `git::store::read_ref` / + // `git::store::marker_value_for`. Reverting either reintroduces + // the DoS window the marker gate exists to close. + assert!( + production_outbox.contains("git::store::read_ref"), + "reconcile_prepared_page must read the marker ref via git::store::read_ref" + ); + assert!( + production_outbox.contains("marker_value_for"), + "reconcile_prepared_page must compute the expected marker value via \ + git::store::marker_value_for" + ); + + // (2) The drain's `EffectsOutcome::Retry` arm checks the bound. + assert!( + production_outbox.contains("effects_max_attempts"), + "drain_receive_pack_requests_with must consult effects_max_attempts on Retry" + ); + + // (3) `git::store::read_ref` is the read seam the gate depends on. + assert!( + store.contains("pub fn read_ref("), + "git::store::read_ref must exist; the marker gate reads through it" + ); + assert!( + store.contains("pub fn marker_value_for("), + "git::store::marker_value_for must exist; the marker gate computes the expected \ + value with it (and the live handler writes the value via the same helper)" + ); + + // (4) DB-side seams the gate depends on. + assert!( + db.contains("pub async fn mark_request_quarantined"), + "Db::mark_request_quarantined must exist" + ); + assert!( + db.contains("pub async fn mark_children_rejected_for_quarantined_parent"), + "Db::mark_children_rejected_for_quarantined_parent must exist" + ); + assert!( + db.contains("pub async fn get_receive_pack_requests_by_ids"), + "Db::get_receive_pack_requests_by_ids must exist (avoids N+1 in the marker gate)" + ); + assert!( + db.contains("pub const QUARANTINED: &str = \"quarantined\""), + "request_state::QUARANTINED must be defined" + ); + + // (5) The handler writes the marker ref BEFORE the durability + // boundary (smart_http::receive_pack_raw). Severing the + // ordering re-opens the marker-gate DoS window for live pushes. + let marker_write = production_repos + .find("git::store::marker_value_for") + .expect("U5 gate stale: the live handler no longer computes the marker value"); + let receive_raw = production_repos + .find("smart_http::receive_pack_raw(") + .expect("U5 gate stale: git_receive_pack no longer calls smart_http::receive_pack_raw"); + assert!( + marker_write < receive_raw, + "U5 gate bypassed: the marker ref must be written BEFORE receive_pack_raw so the \ + reconcile's gate has evidence of the live push" + ); +}