Repository navigation
feat: serve sc+git:: stores, read the manifest in one filtered fetch, and cap fetch downloads - #371
Merged
Merged
Conversation
…ry fetch The manifest read was `ls-remote` followed by `fetch`. It is now one `git fetch --depth=1 +refs/sc/K:refs/sc/K` under protocol v2, so the host filters its ref advertisement to that ref (verified with GIT_TRACE_PACKET against a local bare repository and against sc-forge over smart HTTP). A failed fetch falls back to an exact `ls-remote` to tell an absent ref from an error. Git resolves a refspec source that matches nothing exactly by its abbreviation rules, so a branch or tag spelled `refs/sc/K` after its own prefix could stand in for an absent key; the fetch carries five negative refspecs that keep the exact-name semantics of the old read. `git fetch` has no input-size limit, so every fetch now runs under a watchdog: `fetch.unpackLimit=1` puts the download into one `tmp_pack_*` file, the growth of `objects/pack` is polled every 50 ms, and git is killed over the budget (the key's value cap plus 0.1% and 1 MiB, 2 GiB for a family or batch) or after 120 s without progress. A fetch that ends between two polls is caught by a check after it exits. A breach removes the files the fetch added, every local ref and the lock files a killed git cannot release, and the error names the cap. The bound is disk per fetch, not RAM inside index-pack, not the total cache. Pipe readers are detached threads, so a helper that outlives a killed git cannot block the call. Serve mode (unix) runs git in its own process group and kills the group before reaping the leader; the CLI keeps Ctrl-C and ssh prompts by killing git alone. GitBucket::open_in puts the cache under a given directory and enables serve mode. GitBucket::reap_stale removes caches of dead owners only, so several `sc serve --stdio` processes can share one home. Tests: one filtered fetch per manifest read with the packet trace, decoys at all five abbreviations, oversized planted value in both modes and both check paths with a working instance afterwards, a runaway fetch killed with its descendants and locks, the no-progress deadline, a helper holding the pipes, the reaper, and a forge end-to-end check of the filtered advertisement.
…ch gate `sc serve --store sc+git::<url>` now works over --stdio and --http. The daemon needs git to authenticate without prompting. The private-branch opt-in moves to the operator: `--allow-private-branches` (refused for any other store). Without it the server refuses, as InvalidArgument with no wire change, every object of a sealed-branch kind (Sealed, Manifest): - PutPack stops at the first such object before upload_pack, in any record order, so a mixed pack adds no refs/sc/*. - PutObject refuses before staging, so a later UpdateRef cannot flush it. - UpdateRef refuses a manifest tip even when its objects are already on the chain, so an existing private branch cannot be advanced or re-pointed. The flag lives on BucketTransport (set by open_for_serve) and is plumbed through WirePolicy and ServeLimits; false means gated. The serve cache lives under the serve home's .sc/tmp/ and `sc serve` reaps dead processes' caches at startup. Tests drive the hostile orders against a real bare repository and inspect it afterwards (no refs, no objects, no sealed object in any stored pack), over pipes and through a real HTTP listener with real clients, and through the binary. Disabling each of the three checks in turn fails a test.
… usable A depth-1 fetch records a value commit that has a parent as a shallow boundary, and a discarded download deletes that object. The new test checks that the same instance still reads a good key and refuses the hostile one again, in serve and CLI mode and on both check paths. The timing upper bounds of the watchdog tests move to 90 s (they only guard against a hang of 600 s) and wait_gone waits up to 20 s, so a loaded machine does not fail them.
…hdog Review fixes for serve --store sc+git::. Reaper liveness is now an advisory file lock, not a pid. Each bucket opened by open_in holds File::lock on <cache>/.owner.lock for its life; reap_stale removes a sc-git-bucket-* directory only when it can take that lock (never a symlink, a directory without a lock file, or another name). That drops the dependency on the `kill` binary and the own-pid exemption, and survives pid reuse, pid-1 containers and several pid namespaces on one host. The sweep holds the lock while it removes the directory, and an opener whose lock file vanished starts over. One host only; unreliable over NFS, as for DirBucket. Watchdog: - If the `kill` program for a process group cannot be run, serve mode falls back to killing git alone and warns once on stderr. - After a clean exit with a helper holding the pipes, both pipe receives share one deadline (not twice the idle time) and serve mode stops the group before returning the error. - transfer.bundleURI=false is pinned beside fetch.uriProtocols=. - The family budget error says a legitimate store over 2 GiB needs compaction. The serve gate's refusal now says "No private-branch object was uploaded": public objects of the same push may already be on the host as an orphan pack. Docs: ADR-0060, THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md and the skill state the lock semantics, the `kill` dependency, that only serve mode on unix is disk-bounded against a hostile HTTP host, that refs written by a family fetch are not measured, the 2 GiB family cliff, the unwatched ls-remote and push (ServerAliveInterval over ssh), that a fetch group outlives a killed sc serve, that an ssh user who runs sc serve is an operator, the per-connection cold cache cost times --max-connections, and the 2 ms to 50 ms poll back-off. forge e2e: the new test no longer splits raw_update from its doc comment and fetches with all five negative refspecs. Tests (each fails without its fix): reap_stale_goes_by_the_owner_lock_not_by_pid, a_sweep_never_removes_a_live_bucket_while_others_open (a smoke test of the create/sweep race), the_pipe_wait_after_exit_has_one_deadline, a_helper_holding_the_pipes_after_exit_is_an_error (group variant), a_missing_kill_program_falls_back_to_killing_git_and_warns_once, the_forced_config_pins_the_download_redirect_switches, a_family_fetch_over_budget_names_the_cap_and_compaction, the serve-gate wording in git_serve_tests, and serve_start_reaps_caches_nobody_holds.
…or every breach
Sol review round 1 on serve --store sc+git::.
S1: git runs with the cache directory as its working directory and GIT_DIR
set to the same path, so a relative serve home (`.`) was resolved from inside
the cache and every Git-backed session failed ("not a git repository:
./.sc/tmp/sc-git-bucket-..."). open_full now makes the cache parent absolute
(std::path::absolute) before any git launch, for the serve home and for the
system temp directory alike.
S2: the final budget check discarded the download without handling a
surviving helper, and the pipe-timeout path stopped the group without
discarding. Every breach (budget during the run, budget at the final check,
no progress, a pipe held open after exit) now takes terminate_and_discard:
stop git (the group in serve mode), reap it, keep watching until the pipes
have closed and objects/pack has stopped changing or a 5 s settle deadline
passes, then discard what the fetch added.
Tests, each failing without its fix (checked by mutation):
- a_relative_cache_parent_works (objio) and
serve_store_sc_git_works_with_a_relative_serve_home (cli, serve home `.`
with the server's cwd set to it, push and clone through it).
- a_clean_exit_over_budget_still_stops_a_helper_and_discards (final check).
- a_helper_holding_the_pipes_after_exit_is_an_error and
a_silent_fetch_hits_the_no_progress_deadline now also assert that the
added files are gone; a_runaway_fetch_is_killed_with_its_descendants
covers the in-run budget path. Serve mode asserts no surviving helper.
ADR-0060 and THREAT-MODEL describe the single breach path and the absolute
cache path.
… with evidence Sol review round 2 on serve --store sc+git::. S2: the final size check ran before the pipe wait, so a helper that outlived git could cross the budget during that wait and close both pipes before the deadline, and the watchdog returned success without discarding the oversized download. After git exits it now polls objects/pack for the one deadline both pipes share, and measures once more after both pipes have closed, before it returns success. A breach there takes terminate_and_discard like every other. S3: once the leader is reaped, an empty process group's id can be reused by an unrelated group of the same user, and stop() signalled -<pgid> anyway. The decision is now a small pure function, may_signal_group: free while the leader is unreaped (its pid pins the group id), and after the reap only while a pipe reader is still waiting (an open pipe has a live process holding its write end). With both pipes closed nothing is signalled; the watchdog only settles and discards. A try_wait error counts as an unknown reap state: no group signal. Tests, each failing without its fix (checked by mutation): - a_helper_that_crosses_the_budget_after_git_exits_is_caught (serve and CLI mode): the helper writes past the budget after git exited and closes its pipes. - the_group_is_signalled_only_when_its_id_is_pinned_or_a_member_is_proven (the decision's truth table). - a_reaped_leader_with_both_pipes_closed_gets_no_group_signal, with a recording kill program: no signal is sent, and the download is still discarded. - a_clean_exit_over_budget_still_stops_a_helper_and_discards now also asserts the one group signal in serve mode when a helper holds the pipes. ADR-0060 and THREAT-MODEL describe the post-exit measuring and the signalling rule.
Sol review round 3 on serve --store sc+git::. An open pipe proves a live group member only at the instant it is looked at, and the external `kill` runs later, so once the leader is reaped any group signal can hit a reused group id of an unrelated process of the same user. Holding the pid without reaping needs waitid(WNOWAIT), which is unsafe and not allowed here. may_signal_group now takes only the leader's state and returns true only while the leader is unreaped (its pid pins the group id); after the reap the group is never signalled, whatever the pipes say. A breach found after the reap still settles for a bounded time and discards what the fetch added. The cost is stated in the function's doc comment, ADR-0060, THREAT-MODEL, CLAUDE.md and ARCHITECTURE.md: a helper that outlives git's exit (a smart-HTTP v0/v1 helper chain, an ssh ControlPersist master, a hostile process) is not killed in any mode and may keep writing after the discard until it exits. Tests, each failing without the fix (checked by mutation): - the_group_is_signalled_only_while_the_leader_is_unreaped (truth table). - a_reaped_leader_gets_no_group_signal_even_with_a_pipe_open, with a recording kill program, for a helper that holds the pipes and for none; the download is discarded in both. - a_clean_exit_over_budget_discards_but_leaves_the_helper_alone: no group signal after a clean exit, the helper survives, the download is discarded. - a_helper_holding_the_pipes_after_exit_is_an_error: the helper now outlives the call in serve mode too.
…h compaction Merges PR 2's docs and demo with PR 1's compaction on sc+git:: (#370). Both now hold: `sc remote compact` works on a carrier, `sc serve --store sc+git::` works behind a server-side private-branch gate, the manifest read is one filtered fetch, and every fetch runs under a download watchdog. ADR-0060 gains "Serving a Git store", the fetch-based read with its abbreviation guard, and the watchdog with its exact limits: the group is signalled only while git is unreaped, so a helper that outlives git's exit is not killed and may keep writing after the discard. The planned fix, a group-leading `sh` supervisor, is noted there and as ROADMAP follow-on 10. THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md, README and the sc-cli-reference skill carry the same facts. ROADMAP's `sc+git::` follow-ons mark (2) and (9) built and the manifest-read part of (7) done, next to PR 1's list. The carrier demo keeps PR 1's compaction step (4) and adds the serve step (7), the traced filtered-read step (8) and the zero-residue step (9), on the bare repository and sc-forge legs.
Sol review round 4 on serve --store sc+git::. A carrier whose log/, checkpoints/ or packs/*.idx family passed the 2 GiB family download budget could never be read, including by `sc remote compact`, whose cold refresh reads the same families, so the store could not compact itself out of that state. When a whole-family fetch ends by passing its budget (the watchdog already killed and discarded it), get_family now falls back to the watched single-key path for the requested key. Each exact-key fetch has its own value cap (256 MiB, 2 GiB for a .pack). The instance remembers that the family is too big, prints one note on stderr, and reads that family's other keys by single fetches from then on, with no further family attempts; each family is judged on its own. A family fetch that fails for any other reason is an error as before, and a hostile value in an oversized family still fails its own cap. The budget breach is told from other failures by a flag the watchdog sets (Watch.over), not by message text. The family-budget error text no longer says it needs compaction: an oversized family is no longer a dead end. GitBucket::cap_family_fetches_at is a hidden test hook so other crates can make a family "too big" without gigabytes. Tests, each failing without the fix (checked by mutation): - an_oversized_family_degrades_to_one_fetch_per_key: a counting shim shows one family attempt per family, later keys of it go straight to single fetches, and an absent key of an oversized family is absent. - a_hostile_value_in_an_oversized_family_still_fails_its_own_cap. - a_carrier_whose_families_exceed_the_family_budget_still_compacts (repo, over a real GitBucket with the family budget capped at one byte): both runs complete and the second deletes what aged. ADR-0060, THREAT-MODEL and CLAUDE.md say the family budget is not a cliff.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
tonibergholm
approved these changes
Nov 21, 2026
tonibergholm
left a comment
Member
There was a problem hiding this comment.
Approved after agent reviews (Mythos, Astra, code-reviewer, 5 sol-loop rounds) and green CI on 61013a7.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three follow-ons to
sc+git::remotes (#369, #370).sc serve --storerefusedsc+git::stores, every manifest read cost an unfilteredls-remoteplus a fetch, and a hostile host could make a fetch fill the temp disk before any size cap ran.sc serve --store sc+git::<url>now works, so ansc+https://server can front a Git repository. Clients of that server don't run the client-side private-branch check, so the server enforces it. Withoutsc serve ... --allow-private-branches,BucketTransportrefuses private-branch objects (theSealedandManifestkinds) at three points:PutObject, before staging;PutPack, before anything is uploaded;UpdateRef, for a ref pointed at a manifest already in the store.Protected paths and secrets are unaffected. Tests cover mixed packs, loose sealed objects, a public push in the same session, and advancing an existing private branch, over both stdio and HTTP. Each asserts the Git host received no sealed object. Serve-mode caches live under the serve home's
.sc/tmp/and are reaped at startup only when their owner lock is free, so a live session's cache is never removed. The serve home may be relative.One filtered fetch per manifest read. A manifest read is now a single
git fetch --depth=1instead ofls-remoteplusfetch. Protocol v2 sendsref-prefix refs/sc/manifest, and the host replies with that ref only. Negative refspecs stop an abbreviated decoy such asrefs/heads/refs/sc/manifestfrom standing in for an absent manifest.protocol.version=2,fetch.uriProtocols=andtransfer.bundleURI=falseare pinned.Fetch watchdog. Fetches keep everything they download in one pack (
fetch.unpackLimit=1). The watchdog polls the growth ofobjects/packand kills the fetch when it passes its budget: the key's value cap plus margin for one key, or 2 GiB for a whole key family. A family over 2 GiB is not a dead end: that read falls back to one fetch per key, each under its own cap, so a large store can still be read and compacted. It also kills a fetch after 120 s without progress. It measures again after git exits and after its pipes close. Every breach goes through one terminate-and-discard path that removes what the fetch added. In serve mode git runs in its own process group, which is killed only while the leader is unreaped, so a signal can never reach a reused group id.Known limits, documented in ADR-0060 and THREAT-MODEL:
shgroup leader.ls-remoteand push are not watched.index-packand the total cache across connections are not bounded.Proof:
demo/run_git_carrier_demo.shnow covers four things in both legs, a bare repo and a real sc-forge over smart HTTP:sc serve --store sc+git::step, where a private push is refused without the flag and works with it;refs/sc/*going 7 → 12 → 5;Watchdog tests cover a runaway fetch killed with its descendants, the no-progress deadline, a helper that keeps the pipes after exit, a breach after exit, the no-signal-after-reap rule and an oversized value discarded with the cache still usable. Each rule was also broken on purpose, and its test failed. fmt, clippy
-D warningsandcargo test --workspacepass: 63 suites, 1938 passed.The plan was reviewed by Fable, Mythos (security), Astra and code-reviewer. The diff was reviewed by Mythos and code-reviewer and with sol-loop.
Reviewed with sol-loop: 5 rounds with gpt-6.2-sol on high. Sol verdict: approved
Created with Claude Opus 5.5 in Claude Code (T3 Code), with Claude Sonnet 5.5 subagents writing most of the implementation.