From 5a69996464f5024594abd729072404e415373874 Mon Sep 17 00:00:00 2001 From: Toni Bergholm Date: Sat, 21 Nov 2026 18:06:24 +0200 Subject: [PATCH] docs: record the sc+git:: validation against a real GitHub repository --- .claude/skills/sc-cli-reference/SKILL.md | 4 ++-- CLAUDE.md | 4 ++-- README.md | 3 ++- ROADMAP.md | 11 +++++++---- docs/THREAT-MODEL.md | 4 +++- docs/adr/0057-bucket-compaction.md | 2 +- docs/adr/0060-git-carried-bucket-remotes.md | 16 ++++++++++++---- 7 files changed, 29 insertions(+), 15 deletions(-) diff --git a/.claude/skills/sc-cli-reference/SKILL.md b/.claude/skills/sc-cli-reference/SKILL.md index dbef5a7b..30787ae5 100644 --- a/.claude/skills/sc-cli-reference/SKILL.md +++ b/.claude/skills/sc-cli-reference/SKILL.md @@ -433,8 +433,8 @@ cargo run --bin sc -- remote add sc+git:: [--allow-private-branc # with push to the repo holds bucket put + delete; # deleting a ref is not erasure. `remote compact` # and `serve --store` work (below); - # clone --filter refused as for any bucket. Untested - # against real GitHub + # clone --filter refused as for any bucket. Works on + # GitHub (slow: one HTTPS round trip per git call) cargo run --bin sc -- remote compact [--dry-run] [--prune-expire 24h] [--allow-short-grace] # bucket remotes only (#216, ADR-0057): merge small # packs behind one manifest CAS, then delete log diff --git a/CLAUDE.md b/CLAUDE.md index 8da15218..71aca96e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -164,7 +164,7 @@ the code, those win. | P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill `); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) | | P37d | Agent sessions on a partial clone: `sc ws fork`/`run`/`harvest`/`abandon` and `sc work` work; a workspace or agent sees the host's sparse view only (`sc work` no longer gets a complete tree there), the view closure is filled (lazy, online) or refused with the gap error before anything is written, a workspace commit grafts the out-of-view subtrees from the base without reading them, and harvest lands through the ADR-0049 merge (any conflict falls back to a `work-` branch as on a full clone) | [0050](docs/adr/0050-sessions-on-partial-clones.md) | | P38 | Gitea-compatible forge over bare Git and smart HTTP: recorded API subset, scoped auth, protected merges, maintenance and container operations; 24 recorded scenarios and six first-consumer acceptance scenarios pass. Hosted repos have none of the `.sc/` features; SSH and Actions remain outside v1. | [0058](docs/adr/0058-sc-forge-gitea-compatible-git-forge.md) | -| P39 | `sc+git::` remotes: the bucket WAL stored as one parentless commit per key at `refs/sc/` in any Git repository (system `git` via `objio::GitBucket`, lease CAS, no `+` refspec), so private branches, protected paths and secrets travel through a Git host (local bare repo, sc-forge; GitHub untested); private-branch push needs per-remote `--allow-private-branches`; `remote compact` works on it (first-sighting age record, leased deletes; ADR-0057 addendum); `sc serve --store sc+git::` serves it and refuses private-branch objects unless the operator passes `--allow-private-branches`, and every fetch runs under a download watchdog; sc-forge accepts `refs/sc/` | [0060](docs/adr/0060-git-carried-bucket-remotes.md) | +| P39 | `sc+git::` remotes: the bucket WAL stored as one parentless commit per key at `refs/sc/` in any Git repository (system `git` via `objio::GitBucket`, lease CAS, no `+` refspec), so private branches, protected paths and secrets travel through a Git host (local bare repo, sc-forge, and a private GitHub repository; GitHub round trips make it slow: ~1 min for a first small push); private-branch push needs per-remote `--allow-private-branches`; `remote compact` works on it (first-sighting age record, leased deletes; ADR-0057 addendum); `sc serve --store sc+git::` serves it and refuses private-branch objects unless the operator passes `--allow-private-branches`, and every fetch runs under a download watchdog; sc-forge accepts `refs/sc/` | [0060](docs/adr/0060-git-carried-bucket-remotes.md) | ## Standing boundaries & gotchas @@ -247,7 +247,7 @@ transport-adjacent. The rest, imperatively: repository. A killed process can leave `sc-git-bucket--` in the temp dir (`sc serve` keeps it under the serve home's `.sc/tmp/`, locks `/.owner.lock` for the bucket's life, and reaps at startup the caches - whose lock is free, unix). Not tested against real GitHub. + whose lock is free, unix). Validated on a private GitHub repository (2026-11-21, ADR-0060): custom refs, lease CAS, filtered fetch and compaction behave as on sc-forge; GitHub size limits and rulesets were not exercised. - **`sc serve --store sc+git::` (ADR-0060) refuses private-branch objects unless the operator passes `--allow-private-branches`** (refused on any other store). The gate is `BucketTransport.refuse_sealed`, set by `open_for_serve`: diff --git a/README.md b/README.md index 8f5a6ae7..6e06c4bf 100644 --- a/README.md +++ b/README.md @@ -146,7 +146,8 @@ everyone who can read the repository). `sc remote compact` works on it (run it from one host; the first run only records when it first saw each object). `sc serve --store sc+git::` serves it and refuses private branches unless the operator passes `--allow-private-branches`. Proven by -`demo/run_git_carrier_demo.sh`; not tested against GitHub itself. +`demo/run_git_carrier_demo.sh`, and checked by hand against a private GitHub +repository (slow there: a first small push took about a minute). ## How Phase 1 works diff --git a/ROADMAP.md b/ROADMAP.md index 6221deea..c2aa6ed5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -700,7 +700,7 @@ across every phase. no `+` refspec, rejections classified by re-reading the ref, 2 s freshness window, create-only caching and family-batched fetches). Private branches, protected paths and secrets travel through a Git host (a local bare repo and - sc-forge verified; real GitHub untested); a plain `git clone` never fetches + sc-forge and a private GitHub repository verified); a plain `git clone` never fetches `refs/sc/*`, and the same repository can also mirror public branches through a `--git` remote. A private-branch push needs the per-remote opt-in `--allow-private-branches`; `sc remote compact` works on `sc+git::` (a @@ -1036,9 +1036,12 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in `--allow-private-branches` (the daemon needs non-interactive git auth); (9) a download budget: every fetch into the cache runs under a watchdog that kills git over its budget (disk bytes per fetch; not RAM inside `index-pack`, not the - total cache). Still deferred: (4) validation against real GitHub (per-ref and - per-push limits, rulesets, hosts that refuse unknown ref namespaces), untested - so far; (10) a group-leading supervisor for the serve-mode watchdog: run git + total cache). (4) validated against a private GitHub repository on 2026-11-21 (custom + refs, lease CAS with one winner of two racing pushers, filtered fetch, + compaction 18 → 5 refs); still unexercised there: per-push and per-blob limits, + rulesets on custom refs, and CAS across GitHub replicas beyond one race. Still + deferred: (11) fewer round trips per push (a first small push to GitHub took + 56 s, one HTTPS round trip per git call); (10) a group-leading supervisor for the serve-mode watchdog: run git under a std-only `sh` leader (`sh -c 'git "$@"; echo $? > ; read _' sh `, stdin piped from sc) so the process group id stays pinned after git exits, and the watchdog can group-kill a helper that outlives git (the diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index 161d3213..be10b6c2 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -856,7 +856,9 @@ The bucket model above applies unchanged, with the Git repository as the bucket session's cache on the same home is never removed and pid reuse, pid-1 containers and several pid namespaces on one host do not matter; it is an advisory lock on one host, unreliable over NFS and similar, as for `DirBucket`. -- Not tested against real GitHub. The "not independently audited, do not commit production +- Validated against a private GitHub repository (ADR-0060): custom refs, lease CAS, + the filtered fetch and compaction behave as on sc-forge; GitHub's size limits, rulesets + on custom refs and CAS across its replicas were not exercised. The "not independently audited, do not commit production secrets" warning at the top of this document applies unchanged. ## By-id object reads (`GetObjects`, `sc cat`) — ADR-0047 diff --git a/docs/adr/0057-bucket-compaction.md b/docs/adr/0057-bucket-compaction.md index 3ee8944f..8245ff15 100644 --- a/docs/adr/0057-bucket-compaction.md +++ b/docs/adr/0057-bucket-compaction.md @@ -390,7 +390,7 @@ rights away from pushers; branch protection does not cover `refs/sc/*`. Deleting a ref is not erasure: the host may keep the objects fetchable by id, and forks and mirrors keep their own copies; sealed content stays ciphertext, so rotation (`sc rewrap --rotate`) is the cutover. Tested over a bare repository -and sc-forge; not tested against GitHub. +and sc-forge, and on a private GitHub repository (two runs, 18 → 5 refs; ADR-0060). As built: `compact/sightings.rs` (the record), `compact.rs` (`list_for_sweep`, `delete_keys`, `manifest_unmoved`), `WalView.pack_sizes`, `Bucket::list_tags` diff --git a/docs/adr/0060-git-carried-bucket-remotes.md b/docs/adr/0060-git-carried-bucket-remotes.md index 11d8855a..34f92e0c 100644 --- a/docs/adr/0060-git-carried-bucket-remotes.md +++ b/docs/adr/0060-git-carried-bucket-remotes.md @@ -370,7 +370,7 @@ sighting is immediately old). Consequences to know: - A push that stalls longer than the grace window between uploading a pack and committing it can lose the pack, as on any backend; byte-identical reuse of an aged orphan pack at the moment its delete lands remains a narrowed race. -- Deleting a ref is not erasure (see Consequences). Untested on GitHub. +- Deleting a ref is not erasure (see Consequences). ### Deferred in v1 @@ -405,9 +405,17 @@ every transport path. push, fetch and clone private branches, protected paths and secrets through a Git repository. A keyless clone sees what a keyless clone of an `sc+wal://` remote sees. Proven by `crates/cli/tests/git_bucket.rs`, the forge end-to-end - tests, and `demo/run_git_carrier_demo.sh`. Not tested against real GitHub - (outside this work's grant); expect per-ref and per-push limits and any host - that refuses unknown ref namespaces to be the failure modes. + tests, and `demo/run_git_carrier_demo.sh`. Validated against a private GitHub repository + on 2026-11-21 (git 2.54, HTTPS through the gh credential helper, synthetic data only, + every ref deleted afterwards): GitHub accepted `refs/sc/*` creates, lease + CAS updates and deletes; of two racing pushers exactly one won and the other got + non-fast-forward; a v2 fetch of `refs/sc/manifest` was answered with that ref + only; keyless and keyed clones behaved as on sc-forge; two compaction runs took + the store from 18 refs to 5 and a clone afterwards read the private branch. + Each git call is its own HTTPS round trip (about 1-2 s), so a first push of a + tiny repository took 56 s and a clone 13 s. Not exercised on GitHub: its + per-push (2 GiB) and per-blob (100 MB; values are chunked at 32 MiB) limits, + rulesets on custom refs, and CAS atomicity across its replicas beyond one race. - **The writer set is wider than S3.** Everyone with push access to the Git repository (collaborators, deploy keys, CI tokens) holds bucket put and delete, and there is no object versioning to recover. They can rewind