diff --git a/ROADMAP.md b/ROADMAP.md index c2aa6ed5..ab737f3a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1036,12 +1036,24 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in `--allow-private-branches` (the daemon needs non-interactive git auth); (9) a download budget: every fetch into the cache runs under a watchdog that kills git over its budget (disk bytes per fetch; not RAM inside `index-pack`, not the - total cache). (4) validated against a private GitHub repository on 2026-11-21 (custom + total cache); (11) fewer round trips per push: `Bucket::put_new_batch` and + `exists_batch` (one `git push` per chunk of at most 64 MiB of values, one + create-only lease per ref, never `+` or `--atomic`; one `ls-remote` per chunk + of exact refs) carry the pack and idx and the pre-commit presence check, the + view after a won manifest CAS is built from the manifest just written instead + of read back, and `has_object` trusts a 2 s memo of an empty remote instead of + probing once per object. Measured with a logging `SC_GIT` shim against a local + bare repository, one small commit through `sc push`: a first push 19 network + git calls → 10, a second push 11 → 8 (the open-time manifest read included; + the three probes of an empty remote, at open, in `list_refs` and in + `update_ref`, remain because only `has_object` may use the memo). Not in scope: + `ref_guard::check_ref_move` still downloads the old tip's pack to compare kinds + on every push after the first (bandwidth, not round trips; the idx carries no + kind). (4) validated against a private GitHub repository on 2026-11-21 (custom refs, lease CAS with one winner of two racing pushers, filtered fetch, compaction 18 → 5 refs); still unexercised there: per-push and per-blob limits, rulesets on custom refs, and CAS across GitHub replicas beyond one race. Still - deferred: (11) fewer round trips per push (a first small push to GitHub took - 56 s, one HTTPS round trip per git call); (10) a group-leading supervisor for the serve-mode watchdog: run git + deferred: (10) a group-leading supervisor for the serve-mode watchdog: run git under a std-only `sh` leader (`sh -c 'git "$@"; echo $? > ; read _' sh `, stdin piped from sc) so the process group id stays pinned after git exits, and the watchdog can group-kill a helper that outlives git (the @@ -1057,12 +1069,16 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in does not support them and sc-forge is SHA-1, so the value is low and the cost moderate (the cache's object format would have to be detected before the first fetch); (S2, from review) the ssh user name that `remote list` prints is not a - credential, and `http(s)` userinfo is refused at validation. Known issue, - recorded and not fixed: a push to `refs/sc/K` fails with "dst refspec matches + credential, and `http(s)` userinfo is refused at validation. Declined: a push to `refs/sc/K` fails with "dst refspec matches more than one" when a decoy branch such as `refs/heads/refs/sc/K` exists on the - host (found while building `serve --store`). Whoever can push to the - repository can plant such a decoy, so it is a writer-only denial of service on - that key; reads compare exact full ref names and are not affected. + host (found while building `serve --store`; a create goes to the decoy and is + lease-rejected). Git has no option to turn off the push destination's DWIM + lookup, so a fix would mean avoiding refspecs for these keys altogether. Writers + who can push branches can also push `refs/sc/*` on hosts where custom refs are + not separately protected (verified for sc-forge, unverified for GitHub + rulesets), and a writer can already delete or overwrite any bucket ref, so the + decoy adds no capability; reads compare exact full ref names (`exact_only`) and + are not affected. - **Serve-side persistent pack cache (P36c follow-on).** A bucket-backed serve instance re-downloads packs per connection; a content-addressed on-disk cache in the serve home would make warm instances cheap without diff --git a/crates/objio/src/git.rs b/crates/objio/src/git.rs index 6954cd23..dfe2050b 100644 --- a/crates/objio/src/git.rs +++ b/crates/objio/src/git.rs @@ -56,6 +56,10 @@ const FETCH_BATCH: usize = 256; /// 32 KiB, so the budget stays well under it). const DELETE_BATCH: usize = 128; const DELETE_ARG_BUDGET: usize = 24 << 10; +/// Most value bytes one batched create push carries (an item larger than this +/// goes in a push of its own). The item and command-line bounds of the leased +/// delete apply too, so a batch never outgrows what a host accepts in one push. +const PUSH_VALUE_BUDGET: u64 = 64 << 20; /// How long a fetch may neither grow `objects/pack` nor exit before the /// watchdog kills it (a served store depends on its git child, so a silent /// host must not park a session forever). @@ -563,6 +567,27 @@ fn classify_refusal(ours: &str, lease: &str, remote: Option<&str>) -> Settled { } } +/// How many leading items of `rest` go in one batched git call: the first +/// always does, and more join while the item count stays within +/// [`DELETE_BATCH`], their summed command-line cost within [`DELETE_ARG_BUDGET`] +/// and their summed value bytes within [`PUSH_VALUE_BUDGET`]. `cost` gives an +/// item's `(command-line bytes, value bytes)`. +fn batch_len(rest: &[T], cost: impl Fn(&T) -> (usize, u64)) -> usize { + let (mut n, mut args, mut value) = (0, 0usize, 0u64); + while let Some(item) = rest.get(n) { + let (a, v) = cost(item); + if n > 0 + && (n >= DELETE_BATCH || args + a > DELETE_ARG_BUDGET || value + v > PUSH_VALUE_BUDGET) + { + break; + } + args += a; + value += v; + n += 1; + } + n +} + /// Result of [`GitBucket::put`]. enum Written { Landed(String), @@ -697,6 +722,14 @@ impl GitBucket { self.tune.family_budget = Some(bytes); } + /// Test hook for other crates: [`GitBucket::open`] running `program` instead + /// of `git` (a shim that logs the calls it passes through), without touching + /// the process environment. Not for production use. + #[doc(hidden)] + pub fn open_with_program(url: &str, program: &str) -> Result { + Self::open_with(url, program.to_string(), CHUNK) + } + fn open_with(url: &str, git: String, chunk: usize) -> Result { Self::open_full(url, git, chunk, None, false) } @@ -1097,18 +1130,14 @@ impl GitBucket { /// `ls-remote` lines `(oid, refname)` for `pattern`. Patterns tail-match, so /// callers must compare full refnames themselves. fn ls_remote(&self, pattern: &str) -> Result> { - let out = self.run( - &[ - "ls-remote", - "--refs", - "--end-of-options", - &self.url, - pattern, - ], - None, - &[], - LS_REMOTE_CAP, - )?; + self.ls_remote_all(&[pattern]) + } + + /// [`Self::ls_remote`] for several patterns in one call. + fn ls_remote_all(&self, patterns: &[&str]) -> Result> { + let mut a = vec!["ls-remote", "--refs", "--end-of-options", self.url.as_str()]; + a.extend(patterns); + let out = self.run(&a, None, &[], LS_REMOTE_CAP)?; if !out.success { return Err(self.fail("ls-remote", &out)); } @@ -1581,16 +1610,90 @@ impl GitBucket { Settled::Landed => {} } } - self.lock().insert( - key.to_string(), - Known { - oid: oid.clone(), - create_only: create_only_key(key), - at: Instant::now(), - }, - ); + self.lock().insert(key.to_string(), Self::known(key, &oid)); Ok(Written::Landed(oid)) } + + /// The cached state of a key this instance just wrote at `oid`. + fn known(key: &str, oid: &str) -> Known { + Known { + oid: oid.to_string(), + create_only: create_only_key(key), + at: Instant::now(), + } + } + + /// The remote oid of each exactly named `refs/sc/` among `keys`, from + /// one `ls-remote` per command-line-bounded chunk. Patterns tail-match, so + /// each line is compared by its full refname. + fn remote_refs(&self, keys: &[&str]) -> Result> { + let mut found = HashMap::new(); + let mut rest = keys; + while !rest.is_empty() { + let (chunk, tail) = rest.split_at(batch_len(rest, |k| (NS.len() + k.len() + 1, 0))); + rest = tail; + let patterns: Vec = chunk.iter().map(|k| format!("{NS}{k}")).collect(); + let pats: Vec<&str> = patterns.iter().map(String::as_str).collect(); + for (oid, name) in self.ls_remote_all(&pats)? { + if patterns.contains(&name) { + found.insert(name, oid); + } + } + } + Ok(found) + } + + /// One create-only push of `chunk` (`(key, commit)` pairs): a + /// `--force-with-lease=refs/sc/K:` and an `oid:refs/sc/K` refspec per key, + /// never `+`, never `--atomic` (the keys are independent). Which keys landed + /// is read from git's per-ref porcelain lines whatever its exit status: a + /// push where the host refused some refs exits non-zero yet applied the rest. + /// Returns the landed flags and, when any key did not land, the refusal text. + fn push_new_chunk(&self, chunk: &[(&str, &str)]) -> Result<(Vec, Option)> { + let leases: Vec = chunk + .iter() + .map(|(k, _)| format!("--force-with-lease={NS}{k}:")) + .collect(); + let specs: Vec = chunk + .iter() + .map(|(k, oid)| format!("{oid}:{NS}{k}")) + .collect(); + let mut a = vec!["push", "--porcelain", "--no-verify"]; + a.extend(leases.iter().map(String::as_str)); + a.extend(["--end-of-options", &self.url]); + a.extend(specs.iter().map(String::as_str)); + let out = self.run(&a, None, &[], SMALL_CAP)?; + let mut accepted = std::collections::HashSet::new(); + for line in String::from_utf8_lossy(&out.stdout).lines() { + let mut f = line.split('\t'); + let flag = f.next().unwrap_or(""); + let to = f.next().and_then(|r| r.rsplit_once(':')).map(|(_, t)| t); + if let (Some(to), " " | "+" | "*") = (to, flag) { + accepted.insert(to.to_string()); + } + } + let landed: Vec = chunk + .iter() + .map(|(k, _)| accepted.contains(&format!("{NS}{k}"))) + .collect(); + let refusal = (!landed.iter().all(|l| *l)).then(|| { + // Every rejected ref: a retry can see one key's "stale info" before + // the host's own refusal of another. + let rejected: Vec = out + .stdout + .split(|b| *b == b'\n') + .filter(|l| l.first() == Some(&b'!')) + .map(|l| self.scrub(&String::from_utf8_lossy(l))) + .collect(); + format!( + "git push failed (exit {}): {} {}", + out.code.map_or("signal".to_string(), |c| c.to_string()), + out.stderr, + rejected.join("; ") + ) + }); + Ok((landed, refusal)) + } } impl Drop for GitBucket { @@ -1842,6 +1945,106 @@ impl Bucket for GitBucket { Ok(self.remote_ref(key)?.is_some()) } + /// One `ls-remote` per command-line-bounded chunk, every `refs/sc/` a + /// pattern, compared by exact refname. Never answered from the local state: + /// a key another writer deleted must read as absent. + fn exists_batch(&self, keys: &[&str]) -> Result> { + for k in keys { + check_key(k)?; + } + let found = self.remote_refs(keys)?; + Ok(keys + .iter() + .map(|k| found.contains_key(&format!("{NS}{k}"))) + .collect()) + } + + /// Builds every value commit locally, then pushes in chunks (see + /// [`batch_len`]): one `git push` per chunk instead of one per key. A key the + /// push did not report as landed drops its cached state, and all such keys are + /// classified together by one `ls-remote` of their exact refs with + /// [`classify_refusal`], as [`GitBucket::put`] does for one key: our commit + /// there is a lost ack (landed), another value is a lost race (`false`), + /// nothing is a refusal by the host (an error). If the re-read itself fails + /// the push refusal is returned, never `false`. An error may leave any subset + /// of the keys created: orphans a compaction sweeps. + fn put_new_batch(&self, items: &[(&str, &[u8])]) -> Result> { + if items.is_empty() { + return Ok(Vec::new()); + } + for (k, _) in items { + check_key(k)?; + } + crate::check_batch(items.iter().map(|(k, _)| *k))?; + let oids = items + .iter() + .map(|(k, v)| self.make_commit(k, v, "new")) + .collect::>>()?; + let mut landed = vec![false; items.len()]; + let mut refusal: Option = None; + let mut start = 0; + while start < items.len() { + let n = batch_len(&items[start..], |(k, v)| { + (2 * (NS.len() + k.len()) + 64, v.len() as u64) + }); + let chunk: Vec<(&str, &str)> = (start..start + n) + .map(|i| (items[i].0, oids[i].as_str())) + .collect(); + match self.push_new_chunk(&chunk) { + Ok((flags, msg)) => { + landed[start..start + n].copy_from_slice(&flags); + if refusal.is_none() { + refusal = msg.map(Error::Backend); + } + } + Err(e) => { + refusal.get_or_insert(e); + break; + } + } + start += n; + } + { + let mut st = self.lock(); + for (i, (k, _)) in items.iter().enumerate() { + if landed[i] { + st.insert(k.to_string(), Self::known(k, &oids[i])); + } else { + st.remove(*k); + } + } + } + let unlanded: Vec = (0..items.len()).filter(|i| !landed[*i]).collect(); + if unlanded.is_empty() { + return Ok(landed); + } + let refusal = + refusal.unwrap_or_else(|| Error::Backend("git push did not report the refs".into())); + let keys: Vec<&str> = unlanded.iter().map(|i| items[*i].0).collect(); + // Cannot re-read (network): report the push failure itself. + let Ok(remote) = self.remote_refs(&keys) else { + return Err(refusal); + }; + let mut genuine = false; + for i in unlanded { + let key = items[i].0; + let cur = remote.get(&format!("{NS}{key}")).map(String::as_str); + match classify_refusal(&oids[i], "", cur) { + Settled::Landed => { + landed[i] = true; + self.lock() + .insert(key.to_string(), Self::known(key, &oids[i])); + } + Settled::Lost => {} + Settled::Genuine => genuine = true, + } + } + if genuine { + return Err(refusal); + } + Ok(landed) + } + fn delete(&self, key: &str) -> Result<()> { check_key(key)?; let mut st = self.lock(); @@ -3062,6 +3265,298 @@ mod tests { assert_eq!(leases, N, "one lease per ref"); } + /// [`Bucket::put_new_batch`] over string values. + fn batch(b: &GitBucket, items: &[(&str, &str)]) -> Result> { + let v: Vec<(&str, &[u8])> = items.iter().map(|(k, v)| (*k, v.as_bytes())).collect(); + b.put_new_batch(&v) + } + + fn value_of(b: &GitBucket, key: &str) -> String { + let Fetched::New { bytes, .. } = b.get(key, None).unwrap() else { + panic!("{key} should exist") + }; + String::from_utf8(bytes).unwrap() + } + + /// Shell that sets `s` to the git subcommand (skipping `-c ` pairs). + #[cfg(unix)] + const SUBCOMMAND: &str = "s=; k=0; for x in \"$@\"; do if [ $k = 1 ]; then k=0; continue; fi; if [ \"$x\" = -c ]; then k=1; continue; fi; s=$x; break; done"; + + /// A shim that logs the full command line of every push and the subcommand + /// of every other network call. + #[cfg(unix)] + fn call_log_shim(t: &Tmp, log: &Path) -> String { + shim( + t, + "calls.sh", + &format!( + "{SUBCOMMAND}; case \"$s\" in push) echo \"push $*\" >> {l};; fetch) echo \"$s\" >> {l};; ls-remote) echo \"ls-remote $*\" >> {l};; esac", + l = log.display() + ), + ) + } + + #[cfg(unix)] + fn logged(log: &Path, prefix: &str) -> Vec { + std::fs::read_to_string(log) + .unwrap_or_default() + .lines() + .filter(|l| l.starts_with(prefix)) + .map(String::from) + .collect() + } + + #[test] + fn put_new_batch_reports_each_key_and_leaves_an_existing_one_alone() { + need_git!(); + let t = Tmp::new("batch-mixed"); + let (_, url) = hub(&t); + assert!(open(&url).put_new("log/2", b"theirs").unwrap()); + // this instance has never seen log/2 + let b = open(&url); + let out = batch( + &b, + &[("log/1", "one"), ("log/2", "mine"), ("log/3", "three")], + ) + .unwrap(); + assert_eq!(out, [true, false, true]); + let r = open(&url); + assert_eq!(value_of(&r, "log/1"), "one"); + assert_eq!(value_of(&r, "log/2"), "theirs"); + assert_eq!(value_of(&r, "log/3"), "three"); + // the lost key was not cached as ours, the landed ones are + assert_eq!(value_of(&b, "log/2"), "theirs"); + assert_eq!(value_of(&b, "log/1"), "one"); + } + + #[cfg(unix)] + #[test] + fn put_new_batch_reports_a_ref_the_host_refuses_and_a_retry_finishes() { + use std::os::unix::fs::PermissionsExt; + need_git!(); + let t = Tmp::new("batch-hook"); + let (hubdir, url) = hub(&t); + let log = t.0.join("calls.log"); + let b = open_git(&url, &call_log_shim(&t, &log)); + // An `update` hook judges each ref on its own: it refuses log/2 only. + let hook = hubdir.join("hooks/update"); + std::fs::write( + &hook, + "#!/bin/sh\ncase \"$1\" in refs/sc/log/2) echo 'keep log two' >&2; exit 1;; esac\nexit 0\n", + ) + .unwrap(); + std::fs::set_permissions(&hook, std::fs::Permissions::from_mode(0o755)).unwrap(); + let items = [("log/1", "one"), ("log/2", "two"), ("log/3", "three")]; + let e = batch(&b, &items).unwrap_err().to_string(); + assert!(e.contains("keep log two"), "{e}"); + // the push exits non-zero but its per-ref lines say log/1 and log/3 + // landed, so only log/2 is re-read + let reads = logged(&log, "ls-remote"); + assert_eq!(reads.len(), 1, "{reads:?}"); + assert!(reads[0].contains("refs/sc/log/2"), "{reads:?}"); + assert!( + !reads[0].contains("refs/sc/log/1") && !reads[0].contains("refs/sc/log/3"), + "{reads:?}" + ); + // not mistaken for a lost race, and the other keys landed + assert_eq!( + names(&open(&url).list_tags("").unwrap()), + ["log/1", "log/3"] + ); + // once the host allows it, the same call finishes the job: the keys that + // landed report false, the refused one true + std::fs::remove_file(&hook).unwrap(); + assert_eq!(batch(&b, &items).unwrap(), [false, true, false]); + let r = open(&url); + assert_eq!(value_of(&r, "log/2"), "two"); + assert_eq!( + names(&r.list_tags("").unwrap()), + ["log/1", "log/2", "log/3"] + ); + } + + #[cfg(unix)] + #[test] + fn put_new_batch_never_reports_false_when_it_cannot_tell() { + need_git!(); + // push and the re-read both fail: an error carrying the push refusal + let t = Tmp::new("batch-blind"); + let (_, url) = hub(&t); + let sh = shim( + &t, + "blind.sh", + &format!("{SUBCOMMAND}; case \"$s\" in push|ls-remote) echo boom >&2; exit 1;; esac"), + ); + let b = open_git(&url, &sh); + let e = batch(&b, &[("log/1", "one")]).unwrap_err().to_string(); + assert!(e.contains("git push failed") && e.contains("boom"), "{e}"); + // push fails, the re-read works and finds nothing: the host refused + let sh = shim( + &t, + "refuse.sh", + &format!("{SUBCOMMAND}; case \"$s\" in push) echo boom >&2; exit 1;; esac"), + ); + let b = open_git(&url, &sh); + let e = batch(&b, &[("log/1", "one")]).unwrap_err().to_string(); + assert!(e.contains("boom"), "{e}"); + assert!(!open(&url).exists("log/1").unwrap()); + } + + #[cfg(unix)] + #[test] + fn a_lost_batch_acknowledgement_is_success_when_the_remote_holds_our_commits() { + need_git!(); + let t = Tmp::new("batch-ack"); + let (_, url) = hub(&t); + // the push lands but its answer (exit status and per-ref lines) is lost + let sh = shim_script( + &t, + "ack.sh", + &format!("{SUBCOMMAND}\ncase \"$s\" in push) git \"$@\" >/dev/null 2>&1; exit 1;; esac\nexec git \"$@\"\n"), + ); + let b = open_git(&url, &sh); + assert_eq!( + batch(&b, &[("log/1", "one"), ("log/2", "two")]).unwrap(), + [true, true] + ); + assert_eq!(value_of(&open(&url), "log/2"), "two"); + } + + #[cfg(unix)] + #[test] + fn put_new_batch_is_one_push_per_chunk_with_a_lease_per_ref_and_never_plus_or_atomic() { + need_git!(); + let t = Tmp::new("batch-chunks"); + let (_, url) = hub(&t); + let log = t.0.join("calls.log"); + let b = open_git(&url, &call_log_shim(&t, &log)); + // a few keys: one push, and no re-read when everything landed + let few: Vec = (0..5).map(|i| format!("log/{i:04}")).collect(); + let items: Vec<(&str, &str)> = few.iter().map(|k| (k.as_str(), "v")).collect(); + assert_eq!(batch(&b, &items).unwrap(), [true; 5]); + assert_eq!(logged(&log, "push").len(), 1); + assert!(logged(&log, "ls-remote").is_empty() && logged(&log, "fetch").is_empty()); + // what landed is cached as ours: a create-only key is read with no call + assert!(matches!( + b.get("log/0003", None).unwrap(), + Fetched::New { .. } + )); + assert!(logged(&log, "fetch").is_empty() && logged(&log, "ls-remote").is_empty()); + // more than DELETE_BATCH keys: split, each push within the line budget + let n = DELETE_BATCH + 2; + let many: Vec = (0..n).map(|i| format!("packs/h{i:04}.idx")).collect(); + let items: Vec<(&str, &str)> = many.iter().map(|k| (k.as_str(), "v")).collect(); + assert_eq!(batch(&b, &items).unwrap(), vec![true; n]); + let pushes = logged(&log, "push"); + assert_eq!(pushes.len(), 3, "1 + ceil({n} / {DELETE_BATCH})"); + let mut leases = 0; + for p in &pushes[1..] { + assert!(p.len() < 32 * 1024, "command line of {} bytes", p.len()); + assert!(!p.contains("--atomic"), "{p}"); + for tok in p.split(' ') { + assert!(!tok.starts_with('+'), "plus refspec: {tok}"); + if let Some(l) = tok.strip_prefix("--force-with-lease=refs/sc/packs/") { + assert!(l.ends_with(".idx:"), "create-only lease expected: {tok}"); + leases += 1; + } + } + } + assert_eq!(leases, n, "one lease per ref"); + assert_eq!(b.list("packs/").unwrap().len(), n); + } + + #[test] + fn batch_chunks_respect_the_item_line_and_value_bounds() { + let mib = 1u64 << 20; + // value bytes: 40 + 40 would pass 64, so they split; an item over the + // budget goes alone + let sizes = [40 * mib, 40 * mib, 100 * mib, 1]; + let mut rest: &[u64] = &sizes; + let mut chunks = Vec::new(); + while !rest.is_empty() { + let n = batch_len(rest, |v| (100, *v)); + chunks.push(n); + rest = &rest[n..]; + } + assert_eq!(chunks, [1, 1, 1, 1]); + assert_eq!(batch_len(&[mib; 64], |v| (100, *v)), 64); + assert_eq!( + batch_len(&[mib; 65], |v| (100, *v)), + 64, + "64 MiB fits, 65 do not" + ); + // item count + assert_eq!(batch_len(&[0u64; 500], |v| (1, *v)), DELETE_BATCH); + // command-line bytes: the first item always fits, then the budget bounds + assert_eq!(batch_len(&[0u64; 10], |_| (DELETE_ARG_BUDGET, 0)), 1); + assert_eq!(batch_len(&[0u64; 10], |_| (DELETE_ARG_BUDGET / 4, 0)), 4); + } + + #[cfg(unix)] + #[test] + fn batch_calls_validate_before_any_git_call_and_empty_is_free() { + need_git!(); + let t = Tmp::new("batch-validate"); + let (_, url) = hub(&t); + let log = t.0.join("calls.log"); + let b = open_git(&url, &call_log_shim(&t, &log)); + assert!(b.put_new_batch(&[]).unwrap().is_empty()); + assert!(b.exists_batch(&[]).unwrap().is_empty()); + assert!(batch(&b, &[("log/1", "x"), ("bad key", "y")]).is_err()); + assert!(batch(&b, &[("log/1", "x"), ("log/2/../3", "y")]).is_err()); + assert!(batch(&b, &[("log/1", "x"), ("log/1", "y")]).is_err()); + // acceptable to the shared key check, not to a refspec + assert!(batch(&b, &[("log/1", "x"), ("log/a..b", "y")]).is_err()); + assert!(batch(&b, &[("log/1", "x"), ("log/a:b", "y")]).is_err()); + assert!(b.exists_batch(&["log/1", "a b"]).is_err()); + assert!(b.exists_batch(&["log/1", "log/*"]).is_err()); + assert!( + std::fs::read_to_string(&log).unwrap_or_default().is_empty(), + "no network call for an empty or invalid batch" + ); + assert!(!open(&url).exists("log/1").unwrap()); + } + + #[cfg(unix)] + #[test] + fn exists_batch_compares_exact_refnames_and_never_trusts_the_local_state() { + need_git!(); + let t = Tmp::new("exists-batch"); + let (hubdir, url) = hub(&t); + let log = t.0.join("calls.log"); + let b = open_git(&url, &call_log_shim(&t, &log)); + assert!(batch(&b, &[("log/1", "one"), ("packs/a.pack", "p")]) + .unwrap() + .iter() + .all(|x| *x)); + let oid = git(&hubdir, &["rev-parse", "refs/sc/log/1"], ""); + // a branch whose tail looks like a bucket key is not that key + git( + &hubdir, + &["update-ref", "refs/heads/x/refs/sc/log/9", &oid], + "", + ); + let before = logged(&log, "ls-remote").len(); + assert_eq!( + b.exists_batch(&["log/9", "log/1", "log/2", "packs/a.pack"]) + .unwrap(), + [false, true, false, true] + ); + assert_eq!(logged(&log, "ls-remote").len(), before + 1, "one call"); + // a key another writer deleted reads as absent although we wrote it + git(&hubdir, &["update-ref", "-d", "refs/sc/log/1"], ""); + assert_eq!(b.exists_batch(&["log/1"]).unwrap(), [false]); + // a long list is split by command-line bytes, one ls-remote per chunk + let keys: Vec = (0..300) + .map(|i| format!("packs/{}{i:04}.pack", "h".repeat(50))) + .collect(); + let refs: Vec<&str> = keys.iter().map(String::as_str).collect(); + let before = logged(&log, "ls-remote").len(); + let got = b.exists_batch(&refs).unwrap(); + assert!(got.iter().all(|x| !x)); + assert_eq!(logged(&log, "ls-remote").len() - before, 3); + } + /// Shell snippet for a shim that logs the git subcommand of every call. #[cfg(unix)] fn count_snippet(log: &Path) -> String { @@ -3844,7 +4339,15 @@ mod tests { // The holder going away (a killed server) is what frees the cache. drop(held); - assert_eq!(GitBucket::reap_stale(&parent), 1); + // A child another test thread is forking can hold a copy of the lock's + // descriptor until its exec closes it, so the lock frees a moment later. + let deadline = Instant::now() + Duration::from_secs(2); + let mut reaped = GitBucket::reap_stale(&parent); + while reaped == 0 && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(20)); + reaped = GitBucket::reap_stale(&parent); + } + assert_eq!(reaped, 1); assert!(!held_dir.exists()); // So does dropping a bucket: it removes its own directory. let cache = a.cache.clone(); diff --git a/crates/objio/src/lib.rs b/crates/objio/src/lib.rs index 51dd78e2..f4f8b23f 100644 --- a/crates/objio/src/lib.rs +++ b/crates/objio/src/lib.rs @@ -74,6 +74,24 @@ pub trait Bucket: Send { fn list_meta(&self, prefix: &str) -> Result>; /// Whether `key` exists (a HEAD: needs read rights only, no listing). fn exists(&self, key: &str) -> Result; + /// [`Bucket::put_new`] for several keys at once: one outcome per item, in + /// input order (`true` = created, `false` = the key already existed and was + /// left untouched). Empty input is `Ok(vec![])`; a bad or duplicate key is + /// refused before any write. An error may leave any subset of the keys created + /// (orphans a compaction sweeps); a retry of the same items is safe because + /// every key is create-only. A backend that can batch the round trips + /// overrides this; the default is one `put_new` per item and stops at the + /// first error. + fn put_new_batch(&self, items: &[(&str, &[u8])]) -> Result> { + check_batch(items.iter().map(|(k, _)| *k))?; + items.iter().map(|(k, v)| self.put_new(k, v)).collect() + } + /// [`Bucket::exists`] for several keys at once, one answer per key in + /// input order. Empty input is `Ok(vec![])`. The default is one `exists` per + /// key. + fn exists_batch(&self, keys: &[&str]) -> Result> { + keys.iter().map(|k| self.exists(k)).collect() + } /// Remove `key`. Idempotent: an absent key is success. fn delete(&self, key: &str) -> Result<()>; /// Keys starting with the raw string `prefix`, each with the tag @@ -116,6 +134,19 @@ pub(crate) fn validate_key(key: &str) -> Result<()> { Ok(()) } +/// Refuse a batch that holds a bad key or names a key twice, before any +/// backend writes anything. +pub(crate) fn check_batch<'a>(keys: impl Iterator) -> Result<()> { + let mut seen = std::collections::HashSet::new(); + for k in keys { + validate_key(k)?; + if !seen.insert(k) { + return Err(Error::Backend(format!("duplicate key {k:?} in a batch"))); + } + } + Ok(()) +} + /// Conformance checks every `Bucket` implementation must pass. /// Panics on violation (test helper). pub fn contract_suite(b: &dyn Bucket) { @@ -190,6 +221,38 @@ pub fn contract_suite(b: &dyn Bucket) { assert_eq!(b.list_meta("packs/").unwrap().len(), 2); // a deleted key can be created again assert!(b.put_new("packs/ab12.pack", b"again").unwrap()); + // batches: empty is a no-op, outcomes come back in input order, an existing + // key is reported false and keeps its value + assert!(b.put_new_batch(&[]).unwrap().is_empty()); + assert!(b.exists_batch(&[]).unwrap().is_empty()); + assert_eq!( + b.put_new_batch(&[ + ("batch/a", b"1".as_slice()), + ("log/1", b"clobber".as_slice()), + ("batch/b", b"2".as_slice()), + ]) + .unwrap(), + vec![true, false, true] + ); + let Fetched::New { bytes, .. } = b.get("log/1", None).unwrap() else { + panic!() + }; + assert_eq!(bytes, b"entry-one"); + assert_eq!( + b.exists_batch(&["batch/b", "batch/none", "batch/a", "log/1"]) + .unwrap(), + vec![true, false, true, true] + ); + // a repeated key is refused before anything is written + assert!(b + .put_new_batch(&[("batch/c", b"1".as_slice()), ("batch/c", b"2".as_slice())]) + .is_err()); + assert!(!b.exists("batch/c").unwrap()); + // a bad key is refused before anything is written + assert!(b + .put_new_batch(&[("batch/d", b"1".as_slice()), ("../x", b"2".as_slice())]) + .is_err()); + assert!(!b.exists("batch/d").unwrap()); } /// Conformance checks for a backend that overrides [`Bucket::delete_leased`] diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs index b1f23ace..dcf3a329 100644 --- a/crates/repo/src/bucket_transport.rs +++ b/crates/repo/src/bucket_transport.rs @@ -13,9 +13,12 @@ use scl_objio::{Bucket, Fetched}; use std::cell::{Cell, RefCell}; use std::collections::BTreeMap; use std::io::Write; +use std::time::{Duration, Instant}; mod compact; #[cfg(test)] +mod git_push_tests; +#[cfg(test)] mod git_remote_tests; #[cfg(test)] mod git_serve_tests; @@ -26,6 +29,11 @@ pub use compact::{CompactOptions, CompactReport}; /// moved, so a handful is plenty: one per compaction that lands mid-read. const MAX_RESTARTS: u32 = 8; +/// How long `has_object` trusts a manifest probe that found the remote empty +/// before it probes again (one per pushed object otherwise). Only the per-object +/// probe uses it; every other read re-reads the manifest. +const ABSENT_MEMO: Duration = Duration::from_secs(2); + /// Reconstructed state of the WAL at one manifest tag: the manifest itself, /// the flattened branch tips after replaying the parent chain oldest-first, /// and a lookup from object id to the pack that holds it. @@ -97,6 +105,10 @@ pub struct BucketTransport { /// manifest and tell a compaction that moved on (restart) from a corrupt /// bucket (fail closed). saw_absent: Cell, + /// When the last manifest read found no manifest at all; cleared by any + /// other read result and by a successful manifest CAS. Consulted only by + /// `has_object`, which answers "absent" from it for [`ABSENT_MEMO`]. + absent_at: Cell>, /// Tag of the manifest the in-progress `refresh_once` is building from, so /// a failed attempt can say which manifest it failed on. attempt_tag: RefCell>, @@ -390,6 +402,7 @@ impl BucketTransport { pending_objects: RefCell::new(PendingObjects::default()), leaf_cache: RefCell::new(BTreeMap::new()), saw_absent: Cell::new(false), + absent_at: Cell::new(None), attempt_tag: RefCell::new(None), git_carrier, refuse_sealed: false, @@ -458,114 +471,122 @@ impl BucketTransport { fn refresh_once(&self) -> Result<()> { let cached_tag = self.view.borrow().as_ref().map(|v| v.tag.clone()); match self.bucket.get("manifest", cached_tag.as_deref())? { - Fetched::Unchanged => Ok(()), + Fetched::Unchanged => { + self.absent_at.set(None); + Ok(()) + } Fetched::Absent => { *self.view.borrow_mut() = None; + self.absent_at.set(Some(Instant::now())); Ok(()) } Fetched::New { bytes, tag } => { - *self.attempt_tag.borrow_mut() = Some(tag.clone()); - let bytes = capped("manifest", bytes)?; - let manifest = Manifest::decode(&bytes)?; - let stop = manifest.checkpoint_seq; - // Seed from the checkpoint when the manifest names one. The - // checkpoint is untrusted input like everything else here. - let (mut refs, mut packs): (BTreeMap, Vec) = if stop != 0 - { - let ck_bytes = match self.bucket.get(&checkpoint_key(stop), None)? { - Fetched::New { bytes, .. } => capped("checkpoint", bytes)?, - _ => { - return Err(self.gone(format!( - "checkpoint {stop} referenced by manifest but absent" - ))) - } - }; - let ck = Checkpoint::decode(&ck_bytes)?; - if ck.seq != stop { - return Err(Error::Wal(format!( - "checkpoint at {stop} claims seq {}", - ck.seq - ))); - } - let mut refs = BTreeMap::new(); - for (branch, id) in &ck.refs { - crate::refs::validate_branch_name(branch)?; - refs.insert(branch.clone(), *id); - } - (refs, ck.packs) - } else { - (BTreeMap::new(), Vec::new()) - }; - let mut entries = Vec::new(); - let mut seq = manifest.head_seq; - while seq != stop { - if seq < stop { - return Err(Error::Wal(format!( - "log chain bypasses checkpoint {stop} (reached {seq})" - ))); - } - let Fetched::New { bytes, .. } = self.bucket.get(&log_key(seq), None)? else { - return Err( - self.gone(format!("log entry {seq} referenced by chain but absent")) - ); - }; - let e = LogEntry::decode(&capped("log entry", bytes)?)?; - if e.seq != seq { - return Err(Error::Wal(format!( - "log entry at {} claims seq {}", - seq, e.seq - ))); - } - if e.parent_seq >= e.seq { - return Err(Error::Wal(format!( - "log entry {} has non-decreasing parent {}", - e.seq, e.parent_seq - ))); - } - seq = e.parent_seq; - entries.push(e); - } - entries.reverse(); // oldest first - for e in &entries { - for u in &e.updates { - crate::refs::validate_branch_name(&u.branch)?; - refs.insert(u.branch.clone(), u.new); - } - for hash in &e.packs { - packs.push(hash.clone()); - } - } - // Index build: over the FULL cumulative pack list (checkpoint - // packs + tail packs), fetching each idx exactly as today. - let mut index = BTreeMap::new(); - let mut pack_sizes = BTreeMap::new(); - for hash in &packs { - let Fetched::New { bytes, .. } = self.bucket.get(&idx_key(hash), None)? else { - return Err(self.gone(format!("pack {hash} on chain but idx absent"))); - }; - let bytes = capped("pack idx", bytes)?; - let entries = parse_index(&bytes)?; - pack_sizes.insert(hash.clone(), declared_pack_len(&entries)); - for IndexEntry { id, offset, length } in entries { - index.insert(id, (hash.clone(), offset, length)); - } - } - // Packs a compaction retired will never be asked about again. - { - let live: std::collections::HashSet<&String> = packs.iter().collect(); - self.leaf_cache.borrow_mut().retain(|h, _| live.contains(h)); + self.absent_at.set(None); + self.build_view(bytes, tag) + } + } + } + + /// Install the view for manifest `bytes` read at `tag` (or just written + /// there): walk the log chain and idx files it names and replace `view`. + fn build_view(&self, bytes: Vec, tag: String) -> Result<()> { + *self.attempt_tag.borrow_mut() = Some(tag.clone()); + let bytes = capped("manifest", bytes)?; + let manifest = Manifest::decode(&bytes)?; + let stop = manifest.checkpoint_seq; + // Seed from the checkpoint when the manifest names one. The + // checkpoint is untrusted input like everything else here. + let (mut refs, mut packs): (BTreeMap, Vec) = if stop != 0 { + let ck_bytes = match self.bucket.get(&checkpoint_key(stop), None)? { + Fetched::New { bytes, .. } => capped("checkpoint", bytes)?, + _ => { + return Err(self.gone(format!( + "checkpoint {stop} referenced by manifest but absent" + ))) } - *self.view.borrow_mut() = Some(WalView { - tag, - manifest, - refs, - index, - packs, - pack_sizes, - }); - Ok(()) + }; + let ck = Checkpoint::decode(&ck_bytes)?; + if ck.seq != stop { + return Err(Error::Wal(format!( + "checkpoint at {stop} claims seq {}", + ck.seq + ))); + } + let mut refs = BTreeMap::new(); + for (branch, id) in &ck.refs { + crate::refs::validate_branch_name(branch)?; + refs.insert(branch.clone(), *id); } + (refs, ck.packs) + } else { + (BTreeMap::new(), Vec::new()) + }; + let mut entries = Vec::new(); + let mut seq = manifest.head_seq; + while seq != stop { + if seq < stop { + return Err(Error::Wal(format!( + "log chain bypasses checkpoint {stop} (reached {seq})" + ))); + } + let Fetched::New { bytes, .. } = self.bucket.get(&log_key(seq), None)? else { + return Err(self.gone(format!("log entry {seq} referenced by chain but absent"))); + }; + let e = LogEntry::decode(&capped("log entry", bytes)?)?; + if e.seq != seq { + return Err(Error::Wal(format!( + "log entry at {} claims seq {}", + seq, e.seq + ))); + } + if e.parent_seq >= e.seq { + return Err(Error::Wal(format!( + "log entry {} has non-decreasing parent {}", + e.seq, e.parent_seq + ))); + } + seq = e.parent_seq; + entries.push(e); + } + entries.reverse(); // oldest first + for e in &entries { + for u in &e.updates { + crate::refs::validate_branch_name(&u.branch)?; + refs.insert(u.branch.clone(), u.new); + } + for hash in &e.packs { + packs.push(hash.clone()); + } + } + // Index build: over the FULL cumulative pack list (checkpoint + // packs + tail packs), fetching each idx exactly as today. + let mut index = BTreeMap::new(); + let mut pack_sizes = BTreeMap::new(); + for hash in &packs { + let Fetched::New { bytes, .. } = self.bucket.get(&idx_key(hash), None)? else { + return Err(self.gone(format!("pack {hash} on chain but idx absent"))); + }; + let bytes = capped("pack idx", bytes)?; + let entries = parse_index(&bytes)?; + pack_sizes.insert(hash.clone(), declared_pack_len(&entries)); + for IndexEntry { id, offset, length } in entries { + index.insert(id, (hash.clone(), offset, length)); + } + } + // Packs a compaction retired will never be asked about again. + { + let live: std::collections::HashSet<&String> = packs.iter().collect(); + self.leaf_cache.borrow_mut().retain(|h, _| live.contains(h)); } + *self.view.borrow_mut() = Some(WalView { + tag, + manifest, + refs, + index, + packs, + pack_sizes, + }); + Ok(()) } /// Canonical `encode()` bytes of one object, fetched via its pack @@ -693,8 +714,11 @@ impl BucketTransport { fn upload_pack(&self, objects: &[(ObjectId, Vec)]) -> Result<(String, Vec)> { let (pack, idx) = scl_core::pack::build_pack(objects)?; let hash = hex::encode(blake3::hash(&pack).as_bytes()); - self.bucket.put_new(&pack_key(&hash), &pack)?; - self.bucket.put_new(&idx_key(&hash), &idx)?; + // One batch (one push on a Git carrier). `false` = already present. + self.bucket.put_new_batch(&[ + (pack_key(&hash).as_str(), pack.as_slice()), + (idx_key(&hash).as_str(), idx.as_slice()), + ])?; Ok((hash, objects.iter().map(|(id, _)| *id).collect())) } @@ -726,17 +750,22 @@ impl BucketTransport { /// (a compaction swept it as an unreferenced orphan after an unusually long /// gap, ADR-0057). Committing a log entry that names a deleted pack would /// corrupt the remote; failing lets the caller retry the push, which - /// re-uploads. Two HEADs per pending pack (read rights only: pushers need - /// no list or delete rights). + /// re-uploads. One batched existence check of every pack and idx (read rights + /// only: pushers need no list or delete rights). /// Called right before the manifest CAS. fn ensure_packs_present(&self, hashes: &[String]) -> Result<()> { - for hash in hashes { - if !(self.bucket.exists(&pack_key(hash))? && self.bucket.exists(&idx_key(hash))?) { - return Err(Error::Remote(format!( - "pack {hash} was removed from the bucket before this push committed \ - (bucket compaction swept it); retry the push" - ))); - } + let keys: Vec = hashes + .iter() + .flat_map(|h| [pack_key(h), idx_key(h)]) + .collect(); + let refs: Vec<&str> = keys.iter().map(String::as_str).collect(); + let present = self.bucket.exists_batch(&refs)?; + if let Some(i) = present.iter().position(|p| !p) { + return Err(Error::Remote(format!( + "pack {} was removed from the bucket before this push committed \ + (bucket compaction swept it); retry the push", + hashes[i / 2] + ))); } Ok(()) } @@ -918,7 +947,15 @@ impl Transport for BucketTransport { } fn has_object(&self, id: &ObjectId) -> Result { - self.refresh()?; + // A push asks once per object; an empty remote needs no re-probe for each. + let recently_absent = self.view.borrow().is_none() + && self + .absent_at + .get() + .is_some_and(|t| t.elapsed() < ABSENT_MEMO); + if !recently_absent { + self.refresh()?; + } Ok(self .view .borrow() @@ -1164,15 +1201,19 @@ impl Transport for BucketTransport { // Last look before the commit point, to keep the window in which a // concurrent sweep could still remove a pack as small as possible. self.ensure_packs_present(&pending)?; - if self - .bucket - .put_if_tag("manifest", &manifest.encode(), prev_tag.as_deref())? - .is_some() + if let Some(new_tag) = + self.bucket + .put_if_tag("manifest", &manifest.encode(), prev_tag.as_deref())? { - // Committed. The log entry we just claimed is now on-chain; - // pull the fresh view (cheap: one conditional GET, since our - // own write just changed the tag). - self.refresh()?; + // Committed. The manifest we wrote is the current one at the tag + // the CAS returned, so build the view from it instead of reading + // it back; if that fails, fall back to a plain re-read. + self.absent_at.set(None); + // An empty tag (an S3-compatible host that sent no ETag) is no + // tag to build a view on. + if new_tag.is_empty() || self.build_view(manifest.encode(), new_tag).is_err() { + self.refresh()?; + } let _ = self.maybe_fold_checkpoint(); // best-effort by design (see its doc) return Ok(()); } diff --git a/crates/repo/src/bucket_transport/git_push_tests.rs b/crates/repo/src/bucket_transport/git_push_tests.rs new file mode 100644 index 00000000..a776a424 --- /dev/null +++ b/crates/repo/src/bucket_transport/git_push_tests.rs @@ -0,0 +1,541 @@ +//! What a push to a bucket costs in round trips (ADR-0060 addendum): the pack +//! and its idx go up in one batch, the pre-commit presence check is one call, +//! the view after a won manifest CAS is built from the manifest just written, +//! and an empty remote is probed once, not once per object. Most tests count +//! calls on a recording wrapper over a `DirBucket`; the last one counts the real +//! `git` calls of a push to a `sc+git::` carrier through a logging shim. + +use super::*; +use crate::repo::Repo; +use scl_objio::{DirBucket, ObjectMeta}; +use std::io::Cursor; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; + +/// Every bucket call, in order, as `name` or `name:detail`. +#[derive(Clone, Default)] +struct Calls(Arc>>); + +impl Calls { + fn note(&self, s: impl Into) { + self.0.lock().unwrap().push(s.into()); + } + fn count(&self, prefix: &str) -> usize { + self.0 + .lock() + .unwrap() + .iter() + .filter(|c| c.starts_with(prefix)) + .count() + } +} + +/// Forwards every call to `inner` and records it. The batch methods are +/// forwarded as batches (not split into singles), so what is recorded is what +/// the transport asked for. +struct Counting { + inner: Box, + calls: Calls, + /// Fail the next read of a log entry once (armed by a won manifest CAS). + fail_log_read_after_cas: Arc, + fail_log_read: Arc, +} + +impl Counting { + fn over(root: &std::path::Path) -> (Counting, Calls) { + let calls = Calls::default(); + let c = Counting { + inner: Box::new(DirBucket::open(root).unwrap()), + calls: calls.clone(), + fail_log_read_after_cas: Arc::new(AtomicBool::new(false)), + fail_log_read: Arc::new(AtomicBool::new(false)), + }; + (c, calls) + } +} + +impl Bucket for Counting { + fn get(&self, key: &str, tag: Option<&str>) -> scl_objio::Result { + self.calls.note(format!("get:{key}")); + if key.starts_with("log/") && self.fail_log_read.swap(false, Ordering::SeqCst) { + return Err(scl_objio::Error::Backend("injected read failure".into())); + } + self.inner.get(key, tag) + } + fn put_new(&self, key: &str, bytes: &[u8]) -> scl_objio::Result { + self.calls.note(format!("put_new:{key}")); + self.inner.put_new(key, bytes) + } + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + expected: Option<&str>, + ) -> scl_objio::Result> { + self.calls.note(format!("put_if_tag:{key}")); + let r = self.inner.put_if_tag(key, bytes, expected)?; + if r.is_some() && self.fail_log_read_after_cas.load(Ordering::SeqCst) { + self.fail_log_read.store(true, Ordering::SeqCst); + } + Ok(r) + } + fn list(&self, prefix: &str) -> scl_objio::Result> { + self.inner.list(prefix) + } + fn list_meta(&self, prefix: &str) -> scl_objio::Result> { + self.inner.list_meta(prefix) + } + fn exists(&self, key: &str) -> scl_objio::Result { + self.calls.note(format!("exists:{key}")); + self.inner.exists(key) + } + fn delete(&self, key: &str) -> scl_objio::Result<()> { + self.inner.delete(key) + } + fn put_new_batch(&self, items: &[(&str, &[u8])]) -> scl_objio::Result> { + self.calls.note(format!("put_new_batch:{}", items.len())); + self.inner.put_new_batch(items) + } + fn exists_batch(&self, keys: &[&str]) -> scl_objio::Result> { + self.calls.note(format!("exists_batch:{}", keys.len())); + self.inner.exists_batch(keys) + } +} + +type Objects = Vec<(ObjectId, Vec)>; + +/// `n` commits on one file, each with a different content: per commit its tip +/// and every object reachable from it. +fn history(tag: &str, n: usize) -> Vec<(ObjectId, Objects)> { + let root = std::env::temp_dir().join(format!("scl-bt-push-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(&root).unwrap(); + let repo = Repo::init(&root).unwrap(); + let mut out = Vec::new(); + for i in 0..n { + std::fs::write(root.join("f.txt"), format!("content {tag} {i}")).unwrap(); + let tip = repo.commit("t", &format!("c{i}")).unwrap(); + let store_arc = repo.vfs().store(); + let mut store = store_arc.lock().unwrap(); + let objects = crate::reachable::reachable_objects(&mut *store, &[tip]) + .unwrap() + .into_iter() + .map(|id| (id, store.get(&id).unwrap().encode())) + .collect(); + out.push((tip, objects)); + } + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + out +} + +/// Exactly what `Repo::push` does against a transport: read the refs, probe +/// each object, send the missing ones as one pack, move the ref. +fn push(t: &BucketTransport, tip: ObjectId, objects: &Objects) -> Result<()> { + let expected = t + .list_refs()? + .into_iter() + .find(|(b, _)| b == "main") + .map(|(_, id)| id); + let mut missing = Vec::new(); + for (id, bytes) in objects { + if !t.has_object(id)? { + missing.push((*id, bytes.clone())); + } + } + if !missing.is_empty() { + let (pack, _) = scl_core::pack::build_pack(&missing)?; + t.put_pack(&mut Cursor::new(pack))?; + } + t.update_ref("main", &tip, expected.as_ref()) +} + +fn bucket_root(tag: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!("scl-bt-pushbk-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + d +} + +fn cleanup(root: &std::path::Path) { + std::fs::remove_dir_all(root).unwrap(); + assert!(!root.exists()); +} + +#[test] +fn a_first_push_costs_one_batch_per_step_and_probes_the_empty_remote_per_reader_not_per_object() { + let root = bucket_root("first"); + let (c, calls) = Counting::over(&root); + let t = BucketTransport::from_bucket(Box::new(c)).unwrap(); + let h = history("first", 1); + let (tip, objects) = &h[0]; + assert!(objects.len() >= 3, "a commit, a tree and a blob at least"); + push(&t, *tip, objects).unwrap(); + // pack + idx in one batch; the pre-commit check is one call over both keys + assert_eq!(calls.count("put_new_batch"), 1); + assert_eq!(calls.count("put_new_batch:2"), 1); + assert_eq!(calls.count("put_new:pack"), 0); + assert_eq!(calls.count("exists_batch"), 1); + assert_eq!(calls.count("exists_batch:2"), 1); + assert_eq!(calls.count("exists:"), 0); + // the manifest is read at open, by list_refs and by update_ref; the + // per-object probes after list_refs and the read after the won CAS add none + assert_eq!(calls.count("get:manifest"), 3); + // and the push is on the remote + let r = BucketTransport::from_bucket(Box::new(DirBucket::open(&root).unwrap())).unwrap(); + assert_eq!(r.list_refs().unwrap(), vec![("main".to_string(), *tip)]); + drop((t, r)); + cleanup(&root); +} + +#[test] +fn a_second_push_on_the_same_transport_succeeds_on_the_view_the_first_installed() { + let root = bucket_root("second"); + let (c, calls) = Counting::over(&root); + let t = BucketTransport::from_bucket(Box::new(c)).unwrap(); + let h = history("second", 2); + let (tip1, objects1) = &h[0]; + let (tip2, objects2) = &h[1]; + push(&t, *tip1, objects1).unwrap(); + let after_first = calls.count("get:manifest"); + // the view is the one the commit produced: the tag is the bucket's + let tag = |t: &BucketTransport| t.view.borrow().as_ref().map(|v| v.tag.clone()); + let current = match DirBucket::open(&root) + .unwrap() + .get("manifest", None) + .unwrap() + { + Fetched::New { tag, .. } => tag, + _ => panic!("manifest should exist"), + }; + assert_eq!(tag(&t), Some(current)); + // a ref move with `Some(old)` straight after, with no new read in between + let missing: Objects = objects2 + .iter() + .filter(|(id, _)| !objects1.iter().any(|(o, _)| o == id)) + .cloned() + .collect(); + t.put_pack(&mut Cursor::new( + scl_core::pack::build_pack(&missing).unwrap().0, + )) + .unwrap(); + t.update_ref("main", tip2, Some(tip1)).unwrap(); + assert_eq!( + calls.count("get:manifest"), + after_first + 1, + "one read per update_ref" + ); + assert_eq!(t.list_refs().unwrap(), vec![("main".to_string(), *tip2)]); + drop(t); + cleanup(&root); +} + +#[test] +fn if_the_view_cannot_be_built_from_the_manifest_just_written_it_is_read_back() { + let root = bucket_root("fallback"); + let (c, calls) = Counting::over(&root); + c.fail_log_read_after_cas.store(true, Ordering::SeqCst); + let t = BucketTransport::from_bucket(Box::new(c)).unwrap(); + let h = history("fallback", 1); + let (tip, objects) = &h[0]; + push(&t, *tip, objects).unwrap(); + // the commit landed, the first build failed, the re-read built the view + assert_eq!(calls.count("get:manifest"), 4); + assert_eq!(t.view.borrow().as_ref().map(|v| v.refs.len()), Some(1)); + assert_eq!(t.list_refs().unwrap(), vec![("main".to_string(), *tip)]); + drop(t); + cleanup(&root); +} + +#[test] +fn a_pack_that_is_already_in_the_bucket_is_still_a_successful_upload() { + let root = bucket_root("present"); + let h = history("present", 1); + let (tip, objects) = &h[0]; + // an earlier attempt uploaded the pack and idx but never moved the ref + let a = BucketTransport::from_bucket(Box::new(DirBucket::open(&root).unwrap())).unwrap(); + a.put_pack(&mut Cursor::new( + scl_core::pack::build_pack(objects).unwrap().0, + )) + .unwrap(); + let (c, calls) = Counting::over(&root); + let t = BucketTransport::from_bucket(Box::new(c)).unwrap(); + // the objects are not on the chain yet, so the push sends them again + push(&t, *tip, objects).unwrap(); + assert_eq!(calls.count("put_new_batch:2"), 1); + assert_eq!(t.list_refs().unwrap(), vec![("main".to_string(), *tip)]); + drop((a, t)); + cleanup(&root); +} + +#[test] +fn the_pre_commit_check_covers_every_pending_pack_in_one_call_and_names_the_missing_one() { + let root = bucket_root("ensure"); + let (c, calls) = Counting::over(&root); + let t = BucketTransport::from_bucket(Box::new(c)).unwrap(); + let h = history("ensure", 2); + let (tip2, objects2) = &h[1]; + let (_, objects1) = &h[0]; + t.put_pack(&mut Cursor::new( + scl_core::pack::build_pack(objects1).unwrap().0, + )) + .unwrap(); + let missing: Objects = objects2 + .iter() + .filter(|(id, _)| !objects1.iter().any(|(o, _)| o == id)) + .cloned() + .collect(); + t.put_pack(&mut Cursor::new( + scl_core::pack::build_pack(&missing).unwrap().0, + )) + .unwrap(); + let pending = t.pending_packs.borrow().clone(); + assert_eq!(pending.len(), 2); + // a compaction sweeps the second pack's body before the commit + DirBucket::open(&root) + .unwrap() + .delete(&pack_key(&pending[1])) + .unwrap(); + let e = t.update_ref("main", tip2, None).unwrap_err().to_string(); + assert!( + e.contains(&pending[1]) && e.contains("retry the push"), + "{e}" + ); + assert!(!e.contains(&pending[0]), "named the wrong pack: {e}"); + assert_eq!( + calls.count("exists_batch:4"), + 1, + "both packs, both keys, one call" + ); + assert_eq!(calls.count("exists_batch"), 1); + drop(t); + cleanup(&root); +} + +/// Another writer publishes the first manifest while this transport still +/// remembers the remote as empty. +fn empty_then_published( + tag: &str, +) -> ( + std::path::PathBuf, + BucketTransport, + Calls, + ObjectId, + ObjectId, +) { + let root = bucket_root(tag); + let (c, calls) = Counting::over(&root); + let t1 = BucketTransport::from_bucket(Box::new(c)).unwrap(); + let h = history(tag, 1); + let t2 = BucketTransport::from_bucket(Box::new(DirBucket::open(&root).unwrap())).unwrap(); + push(&t2, h[0].0, &h[0].1).unwrap(); + (root, t1, calls, h[0].0, h[0].1[0].0) +} + +#[test] +fn only_the_per_object_probe_trusts_a_recent_empty_manifest() { + // has_object answers from the memo and asks the bucket nothing + let root = bucket_root("memo"); + // Built first, so the probes below run well inside the memo window. + let h = history("memo", 1); + let (c, calls) = Counting::over(&root); + let t = BucketTransport::from_bucket(Box::new(c)).unwrap(); + let before = calls.count("get:manifest"); + assert_eq!(before, 1, "opened on an empty remote"); + for (id, _) in &h[0].1 { + assert!(!t.has_object(id).unwrap()); + } + assert_eq!(calls.count("get:manifest"), before); + // an older memo is probed again + t.absent_at + .set(Some(Instant::now().checked_sub(ABSENT_MEMO * 2).unwrap())); + assert!(!t.has_object(&h[0].1[0].0).unwrap()); + assert_eq!(calls.count("get:manifest"), before + 1); + drop(t); + cleanup(&root); + + // list_refs, head_branch and update_ref never use it + let (root, t1, calls, tip, _) = empty_then_published("memo-list"); + let before = calls.count("get:manifest"); + assert_eq!(t1.list_refs().unwrap(), vec![("main".to_string(), tip)]); + assert_eq!(calls.count("get:manifest"), before + 1); + drop(t1); + cleanup(&root); + + let (root, t1, _, _, _) = empty_then_published("memo-head"); + assert_eq!(t1.head_branch().unwrap(), "main"); + drop(t1); + cleanup(&root); + + // update_ref sees the published manifest at once: with a stale "empty" it + // would try to create the branch instead of refusing the move + let (root, t1, _, tip, _) = empty_then_published("memo-update"); + let h = history("memo-update-other", 1); + t1.put_pack(&mut Cursor::new( + scl_core::pack::build_pack(&h[0].1).unwrap().0, + )) + .unwrap(); + let e = t1.update_ref("main", &h[0].0, None).unwrap_err(); + assert!(matches!(e, Error::NonFastForward), "{e}"); + assert_eq!(t1.list_refs().unwrap(), vec![("main".to_string(), tip)]); + drop(t1); + cleanup(&root); +} + +#[test] +fn the_empty_memo_is_cleared_by_any_other_read_and_by_a_won_cas() { + let (root, t1, _, _, id) = empty_then_published("memo-clear"); + assert!(t1.absent_at.get().is_some(), "set by the read at open"); + // Restart the window: building and publishing the history took time. + t1.absent_at.set(Some(Instant::now())); + // inside the window the probe is answered from the memo + assert!( + !t1.has_object(&id).unwrap(), + "stale by design, for one window" + ); + // any other read result clears it, and then the probe asks again + t1.list_refs().unwrap(); + assert!(t1.absent_at.get().is_none()); + assert!(t1.has_object(&id).unwrap()); + drop(t1); + cleanup(&root); + + // a won CAS clears it too + let root = bucket_root("memo-cas"); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&root).unwrap())).unwrap(); + assert!(t.absent_at.get().is_some()); + let h = history("memo-cas", 1); + push(&t, h[0].0, &h[0].1).unwrap(); + assert!(t.absent_at.get().is_none()); + drop(t); + cleanup(&root); +} + +#[cfg(unix)] +mod carrier { + use super::*; + use scl_objio::GitBucket; + use std::os::unix::fs::PermissionsExt; + use std::path::PathBuf; + use std::process::Command; + + fn have_git() -> bool { + Command::new("git") + .arg("--version") + .output() + .is_ok_and(|o| o.status.success()) + } + + /// A bare hub plus a `git` shim that logs the subcommand of every call. + struct Carrier { + dir: PathBuf, + url: String, + shim: String, + log: PathBuf, + } + + impl Carrier { + fn new(tag: &str) -> Carrier { + let dir = bucket_root(&format!("carrier-{tag}")); + std::fs::create_dir_all(&dir).unwrap(); + let bare = dir.join("hub.git"); + let ok = Command::new("git") + .args(["init", "--bare", "-q", bare.to_str().unwrap()]) + .env("GIT_CONFIG_GLOBAL", "/dev/null") + .env("GIT_CONFIG_NOSYSTEM", "1") + .status() + .unwrap(); + assert!(ok.success()); + let log = dir.join("calls.log"); + let shim = dir.join("git-shim.sh"); + std::fs::write( + &shim, + format!( + "#!/bin/sh\ns=; k=0; for x in \"$@\"; do if [ $k = 1 ]; then k=0; continue; fi; if [ \"$x\" = -c ]; then k=1; continue; fi; s=$x; break; done\necho \"$s\" >> {}\nexec git \"$@\"\n", + log.display() + ), + ) + .unwrap(); + std::fs::set_permissions(&shim, std::fs::Permissions::from_mode(0o755)).unwrap(); + Carrier { + url: format!("file://{}", bare.display()), + shim: shim.to_str().unwrap().to_string(), + dir, + log, + } + } + + /// Network `git` calls made so far (push, fetch, ls-remote). + fn network_calls(&self) -> Vec { + std::fs::read_to_string(&self.log) + .unwrap_or_default() + .lines() + .filter(|l| matches!(*l, "push" | "fetch" | "ls-remote")) + .map(String::from) + .collect() + } + + /// What `sc push` does: open the carrier (one manifest read), then push. + fn push(&self, tip: ObjectId, objects: &Objects) -> Vec { + let before = self.network_calls().len(); + let b = GitBucket::open_with_program(&self.url, &self.shim).unwrap(); + let t = BucketTransport::build(Box::new(b), true).unwrap(); + push(&t, tip, objects).unwrap(); + drop(t); + self.network_calls().split_off(before) + } + } + + impl Drop for Carrier { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.dir); + } + } + + #[test] + fn a_push_to_a_git_carrier_costs_a_bounded_number_of_network_calls() { + if !have_git() { + eprintln!("git not available; skipping"); + return; + } + let c = Carrier::new("counts"); + let h = history("counts", 2); + let first = c.push(h[0].0, &h[0].1); + let second = c.push(h[1].0, &h[1].1); + // Each probe of the still-empty remote is a failed fetch plus the + // ls-remote that tells "absent" from "failed": the read at open, + // list_refs and update_ref (the per-object probes are free). Then the + // pack and idx in one push, the log entry, one existence check of pack + // and idx, the manifest. Before this change the same push cost 19. + assert_eq!( + first, + [ + "fetch", + "ls-remote", + "fetch", + "ls-remote", + "push", + "fetch", + "ls-remote", + "push", + "ls-remote", + "push" + ] + ); + // The manifest, the log family and the idx family read cold, the pack + // and idx in one push, the old pack body the ref rule reads, the log + // entry, one existence check, the manifest. Before this change: 11. + assert_eq!( + second, + [ + "fetch", + "fetch", + "fetch", + "push", + "fetch", + "push", + "ls-remote", + "push" + ] + ); + } +} diff --git a/docs/adr/0046-wal-bucket-remotes.md b/docs/adr/0046-wal-bucket-remotes.md index 07daaef0..0de52538 100644 --- a/docs/adr/0046-wal-bucket-remotes.md +++ b/docs/adr/0046-wal-bucket-remotes.md @@ -238,6 +238,11 @@ ADR-0057. Effects on this ADR's contracts: - **A pending pack is verified present right before the manifest CAS**, so a push whose pack was swept fails with a retry message rather than committing an entry that names a deleted pack. +- **`has_object` may answer "absent" from a 2 s memo of an empty remote** (a + manifest read that found none), instead of probing once per pushed object; + every other read still re-reads the manifest. `Bucket` also gained + `put_new_batch` and `exists_batch` for the upload and the check above (ADR-0060 + addendum). - `objio`'s `Bucket` gained `list_meta` and `delete`. The "nothing already in the bucket is ever overwritten" property holds; compaction only adds new objects and removes unreferenced ones. diff --git a/docs/adr/0060-git-carried-bucket-remotes.md b/docs/adr/0060-git-carried-bucket-remotes.md index 34f92e0c..ecfd346d 100644 --- a/docs/adr/0060-git-carried-bucket-remotes.md +++ b/docs/adr/0060-git-carried-bucket-remotes.md @@ -118,6 +118,27 @@ under the temp directory (mode 0700), removed on `Drop`. by one. All fetches are `--depth=1`, so a hostile commit's parents are never downloaded. A counting `SC_GIT` shim test bounds the git network calls of a cold clone and an incremental push independent of the log length. +- **Batched writes and presence checks.** `Bucket::put_new_batch` pushes the + value commits of several create-only keys in one `git push --porcelain` per + chunk (at most 64 MiB of values, an item over that alone, and the leased + delete's item and command-line bounds), with one `--force-with-lease=refs/sc/K:` + per ref; never `+`, never `--atomic`, since the keys are independent (a pack + without its idx is an orphan compaction sweeps, as after two separate pushes). + Which refs landed is read from the per-ref porcelain lines whatever the exit + status; every other ref is dropped from the cache and classified together by + one `ls-remote` of its exact refs, with the same rules as a single `put` + (our commit there: landed; another value: lost; nothing: the host refused). + If that re-read fails the push refusal is returned, never `false`. + `exists_batch` is one `ls-remote` per command-line-bounded chunk with an exact + refname compare, and is never answered from the local cache. `BucketTransport` + uploads a pack and its idx with one `put_new_batch`, checks every pending pack + and idx with one `exists_batch` before the manifest CAS, and after a won CAS + builds its view from the manifest it just wrote instead of reading it back. + Counted with a logging `SC_GIT` shim through `sc push` of one small commit: a + first push 19 network git calls → 10 (three probes of the empty remote at open, + in `list_refs` and in `update_ref`, three pushes, one existence check), a second + push 11 → 8. The old tip's pack is still downloaded by the ref rule on every + push after the first (bandwidth, not round trips). - A mutable key (the manifest) is read with one call, `git fetch --depth=1 +refs/sc/K:refs/sc/K`, instead of `ls-remote` followed by `fetch`. The local ref after the fetch is the tag; if it equals the caller's cached tag the read is @@ -439,8 +460,12 @@ every transport path. home whose owner lock is free at startup (unix). A normal exit and an error returned to `main` remove it. - Scaling: `ls-remote` and push download the whole ref advertisement (the pattern given to `ls-remote` filters on the client; a packet trace showed the server sends every ref; the manifest read no longer does, it is one fetch that a v2 host answers with that ref alone), and a cold read fetches every `log/` and `checkpoints/` entry behind the last checkpoint. Refs grow by at least three per push (a log entry, a pack and its idx); `sc remote compact` (above) is what bounds that, folding the small packs and deleting what nothing names, so a long-lived store should be compacted on a schedule. Between runs the advertisement still grows with the pushes since the last one, until a ref-filtering host exists. - Each cold read or push costs git round trips. The freshness window and family - batching bound them, but a Git host is slower than S3. If push over `sc+git::` - stays slow on large histories, the next step is a batched `has_objects`. + batching bound them, but a Git host is slower than S3. The per-object probe is + free on an empty remote (the transport remembers it for 2 s). On a non-empty + `sc+git::` remote the freshness window answers the per-object probes, and git + reads the manifest again only once the cached tag is older than 2 s; S3 and + `sc+wal://` have no such window and read the manifest on every probe. If push stays slow on large histories, the next step is a batched + `has_objects`. - sc-forge repositories keep a deleted or replaced value fetchable by id until the gc prune stage runs. - The "not independently audited, do not commit production secrets" warning is