Skip to content

feat: serve sc+git:: stores, read the manifest in one filtered fetch, and cap fetch downloads - #371

Merged
tonibergholm-codento merged 9 commits into
mainfrom
feat/sc-git-serve-store
Nov 21, 2026
Merged

tonibergholm-codento merged 9 commits into
mainfrom
feat/sc-git-serve-store

Conversation

@tonibergholm-codento

Copy link
Copy Markdown
Collaborator

Three follow-ons to sc+git:: remotes (#369, #370). sc serve --store refused sc+git:: stores, every manifest read cost an unfiltered ls-remote plus a fetch, and a hostile host could make a fetch fill the temp disk before any size cap ran.

sc serve --store sc+git::<url> now works, so an sc+https:// server can front a Git repository. Clients of that server don't run the client-side private-branch check, so the server enforces it. Without sc serve ... --allow-private-branches, BucketTransport refuses private-branch objects (the Sealed and Manifest kinds) at three points:

  • in PutObject, before staging;
  • in PutPack, before anything is uploaded;
  • in UpdateRef, for a ref pointed at a manifest already in the store.

Protected paths and secrets are unaffected. Tests cover mixed packs, loose sealed objects, a public push in the same session, and advancing an existing private branch, over both stdio and HTTP. Each asserts the Git host received no sealed object. Serve-mode caches live under the serve home's .sc/tmp/ and are reaped at startup only when their owner lock is free, so a live session's cache is never removed. The serve home may be relative.

One filtered fetch per manifest read. A manifest read is now a single git fetch --depth=1 instead of ls-remote plus fetch. Protocol v2 sends ref-prefix refs/sc/manifest, and the host replies with that ref only. Negative refspecs stop an abbreviated decoy such as refs/heads/refs/sc/manifest from standing in for an absent manifest. protocol.version=2, fetch.uriProtocols= and transfer.bundleURI=false are pinned.

Fetch watchdog. Fetches keep everything they download in one pack (fetch.unpackLimit=1). The watchdog polls the growth of objects/pack and kills the fetch when it passes its budget: the key's value cap plus margin for one key, or 2 GiB for a whole key family. A family over 2 GiB is not a dead end: that read falls back to one fetch per key, each under its own cap, so a large store can still be read and compacted. It also kills a fetch after 120 s without progress. It measures again after git exits and after its pipes close. Every breach goes through one terminate-and-discard path that removes what the fetch added. In serve mode git runs in its own process group, which is killed only while the leader is unreaped, so a signal can never reach a reused group id.

Known limits, documented in ADR-0060 and THREAT-MODEL:

  • A helper that outlives git's exit is not killed. Examples are a smart-HTTP v0/v1 helper chain or an ssh ControlPersist master. ROADMAP follow-on (10) is the planned fix: a std-only sh group leader.
  • ls-remote and push are not watched.
  • RAM inside index-pack and the total cache across connections are not bounded.

Proof: demo/run_git_carrier_demo.sh now covers four things in both legs, a bare repo and a real sc-forge over smart HTTP:

  • an sc serve --store sc+git:: step, where a private push is refused without the flag and works with it;
  • a packet trace of the filtered manifest read, where the host listed 1 of 13 refs;
  • feat: compact sc+git:: stores so their refs stop growing #370's compaction, with refs/sc/* going 7 → 12 → 5;
  • zero residue.

Watchdog tests cover a runaway fetch killed with its descendants, the no-progress deadline, a helper that keeps the pipes after exit, a breach after exit, the no-signal-after-reap rule and an oversized value discarded with the cache still usable. Each rule was also broken on purpose, and its test failed. fmt, clippy -D warnings and cargo test --workspace pass: 63 suites, 1938 passed.

The plan was reviewed by Fable, Mythos (security), Astra and code-reviewer. The diff was reviewed by Mythos and code-reviewer and with sol-loop.

Reviewed with sol-loop: 5 rounds with gpt-6.2-sol on high. Sol verdict: approved

Created with Claude Opus 5.5 in Claude Code (T3 Code), with Claude Sonnet 5.5 subagents writing most of the implementation.

…ry fetch

The manifest read was `ls-remote` followed by `fetch`. It is now one
`git fetch --depth=1 +refs/sc/K:refs/sc/K` under protocol v2, so the host
filters its ref advertisement to that ref (verified with GIT_TRACE_PACKET
against a local bare repository and against sc-forge over smart HTTP). A
failed fetch falls back to an exact `ls-remote` to tell an absent ref from an
error. Git resolves a refspec source that matches nothing exactly by its
abbreviation rules, so a branch or tag spelled `refs/sc/K` after its own
prefix could stand in for an absent key; the fetch carries five negative
refspecs that keep the exact-name semantics of the old read.

`git fetch` has no input-size limit, so every fetch now runs under a
watchdog: `fetch.unpackLimit=1` puts the download into one `tmp_pack_*` file,
the growth of `objects/pack` is polled every 50 ms, and git is killed over the
budget (the key's value cap plus 0.1% and 1 MiB, 2 GiB for a family or batch)
or after 120 s without progress. A fetch that ends between two polls is caught
by a check after it exits. A breach removes the files the fetch added, every
local ref and the lock files a killed git cannot release, and the error names
the cap. The bound is disk per fetch, not RAM inside index-pack, not the total
cache. Pipe readers are detached threads, so a helper that outlives a killed
git cannot block the call. Serve mode (unix) runs git in its own process
group and kills the group before reaping the leader; the CLI keeps Ctrl-C and
ssh prompts by killing git alone.

GitBucket::open_in puts the cache under a given directory and enables serve
mode. GitBucket::reap_stale removes caches of dead owners only, so several
`sc serve --stdio` processes can share one home.

Tests: one filtered fetch per manifest read with the packet trace, decoys at
all five abbreviations, oversized planted value in both modes and both check
paths with a working instance afterwards, a runaway fetch killed with its
descendants and locks, the no-progress deadline, a helper holding the pipes,
the reaper, and a forge end-to-end check of the filtered advertisement.
…ch gate

`sc serve --store sc+git::<url>` now works over --stdio and --http. The
daemon needs git to authenticate without prompting. The private-branch
opt-in moves to the operator: `--allow-private-branches` (refused for any
other store). Without it the server refuses, as InvalidArgument with no wire
change, every object of a sealed-branch kind (Sealed, Manifest):

- PutPack stops at the first such object before upload_pack, in any record
  order, so a mixed pack adds no refs/sc/*.
- PutObject refuses before staging, so a later UpdateRef cannot flush it.
- UpdateRef refuses a manifest tip even when its objects are already on the
  chain, so an existing private branch cannot be advanced or re-pointed.

The flag lives on BucketTransport (set by open_for_serve) and is plumbed
through WirePolicy and ServeLimits; false means gated. The serve cache lives
under the serve home's .sc/tmp/ and `sc serve` reaps dead processes' caches
at startup.

Tests drive the hostile orders against a real bare repository and inspect it
afterwards (no refs, no objects, no sealed object in any stored pack), over
pipes and through a real HTTP listener with real clients, and through the
binary. Disabling each of the three checks in turn fails a test.
… usable

A depth-1 fetch records a value commit that has a parent as a shallow
boundary, and a discarded download deletes that object. The new test checks
that the same instance still reads a good key and refuses the hostile one
again, in serve and CLI mode and on both check paths. The timing upper bounds
of the watchdog tests move to 90 s (they only guard against a hang of 600 s)
and wait_gone waits up to 20 s, so a loaded machine does not fail them.
…hdog

Review fixes for serve --store sc+git::.

Reaper liveness is now an advisory file lock, not a pid. Each bucket opened
by open_in holds File::lock on <cache>/.owner.lock for its life; reap_stale
removes a sc-git-bucket-* directory only when it can take that lock (never a
symlink, a directory without a lock file, or another name). That drops the
dependency on the `kill` binary and the own-pid exemption, and survives pid
reuse, pid-1 containers and several pid namespaces on one host. The sweep
holds the lock while it removes the directory, and an opener whose lock file
vanished starts over. One host only; unreliable over NFS, as for DirBucket.

Watchdog:
- If the `kill` program for a process group cannot be run, serve mode falls
  back to killing git alone and warns once on stderr.
- After a clean exit with a helper holding the pipes, both pipe receives share
  one deadline (not twice the idle time) and serve mode stops the group before
  returning the error.
- transfer.bundleURI=false is pinned beside fetch.uriProtocols=.
- The family budget error says a legitimate store over 2 GiB needs compaction.

The serve gate's refusal now says "No private-branch object was uploaded":
public objects of the same push may already be on the host as an orphan pack.

Docs: ADR-0060, THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md and the skill state
the lock semantics, the `kill` dependency, that only serve mode on unix is
disk-bounded against a hostile HTTP host, that refs written by a family fetch
are not measured, the 2 GiB family cliff, the unwatched ls-remote and push
(ServerAliveInterval over ssh), that a fetch group outlives a killed sc serve,
that an ssh user who runs sc serve is an operator, the per-connection cold
cache cost times --max-connections, and the 2 ms to 50 ms poll back-off.

forge e2e: the new test no longer splits raw_update from its doc comment and
fetches with all five negative refspecs.

Tests (each fails without its fix): reap_stale_goes_by_the_owner_lock_not_by_pid,
a_sweep_never_removes_a_live_bucket_while_others_open (a smoke test of the
create/sweep race), the_pipe_wait_after_exit_has_one_deadline,
a_helper_holding_the_pipes_after_exit_is_an_error (group variant),
a_missing_kill_program_falls_back_to_killing_git_and_warns_once,
the_forced_config_pins_the_download_redirect_switches,
a_family_fetch_over_budget_names_the_cap_and_compaction, the serve-gate
wording in git_serve_tests, and serve_start_reaps_caches_nobody_holds.
…or every breach

Sol review round 1 on serve --store sc+git::.

S1: git runs with the cache directory as its working directory and GIT_DIR
set to the same path, so a relative serve home (`.`) was resolved from inside
the cache and every Git-backed session failed ("not a git repository:
./.sc/tmp/sc-git-bucket-..."). open_full now makes the cache parent absolute
(std::path::absolute) before any git launch, for the serve home and for the
system temp directory alike.

S2: the final budget check discarded the download without handling a
surviving helper, and the pipe-timeout path stopped the group without
discarding. Every breach (budget during the run, budget at the final check,
no progress, a pipe held open after exit) now takes terminate_and_discard:
stop git (the group in serve mode), reap it, keep watching until the pipes
have closed and objects/pack has stopped changing or a 5 s settle deadline
passes, then discard what the fetch added.

Tests, each failing without its fix (checked by mutation):
- a_relative_cache_parent_works (objio) and
  serve_store_sc_git_works_with_a_relative_serve_home (cli, serve home `.`
  with the server's cwd set to it, push and clone through it).
- a_clean_exit_over_budget_still_stops_a_helper_and_discards (final check).
- a_helper_holding_the_pipes_after_exit_is_an_error and
  a_silent_fetch_hits_the_no_progress_deadline now also assert that the
  added files are gone; a_runaway_fetch_is_killed_with_its_descendants
  covers the in-run budget path. Serve mode asserts no surviving helper.

ADR-0060 and THREAT-MODEL describe the single breach path and the absolute
cache path.
… with evidence

Sol review round 2 on serve --store sc+git::.

S2: the final size check ran before the pipe wait, so a helper that outlived
git could cross the budget during that wait and close both pipes before the
deadline, and the watchdog returned success without discarding the oversized
download. After git exits it now polls objects/pack for the one deadline both
pipes share, and measures once more after both pipes have closed, before it
returns success. A breach there takes terminate_and_discard like every other.

S3: once the leader is reaped, an empty process group's id can be reused by an
unrelated group of the same user, and stop() signalled -<pgid> anyway. The
decision is now a small pure function, may_signal_group: free while the leader
is unreaped (its pid pins the group id), and after the reap only while a pipe
reader is still waiting (an open pipe has a live process holding its write
end). With both pipes closed nothing is signalled; the watchdog only settles
and discards. A try_wait error counts as an unknown reap state: no group signal.

Tests, each failing without its fix (checked by mutation):
- a_helper_that_crosses_the_budget_after_git_exits_is_caught (serve and CLI
  mode): the helper writes past the budget after git exited and closes its pipes.
- the_group_is_signalled_only_when_its_id_is_pinned_or_a_member_is_proven (the
  decision's truth table).
- a_reaped_leader_with_both_pipes_closed_gets_no_group_signal, with a recording
  kill program: no signal is sent, and the download is still discarded.
- a_clean_exit_over_budget_still_stops_a_helper_and_discards now also asserts
  the one group signal in serve mode when a helper holds the pipes.

ADR-0060 and THREAT-MODEL describe the post-exit measuring and the signalling
rule.
Sol review round 3 on serve --store sc+git::.

An open pipe proves a live group member only at the instant it is looked at,
and the external `kill` runs later, so once the leader is reaped any group
signal can hit a reused group id of an unrelated process of the same user.
Holding the pid without reaping needs waitid(WNOWAIT), which is unsafe and not
allowed here. may_signal_group now takes only the leader's state and returns
true only while the leader is unreaped (its pid pins the group id); after the
reap the group is never signalled, whatever the pipes say. A breach found after
the reap still settles for a bounded time and discards what the fetch added.

The cost is stated in the function's doc comment, ADR-0060, THREAT-MODEL,
CLAUDE.md and ARCHITECTURE.md: a helper that outlives git's exit (a smart-HTTP
v0/v1 helper chain, an ssh ControlPersist master, a hostile process) is not
killed in any mode and may keep writing after the discard until it exits.

Tests, each failing without the fix (checked by mutation):
- the_group_is_signalled_only_while_the_leader_is_unreaped (truth table).
- a_reaped_leader_gets_no_group_signal_even_with_a_pipe_open, with a recording
  kill program, for a helper that holds the pipes and for none; the download
  is discarded in both.
- a_clean_exit_over_budget_discards_but_leaves_the_helper_alone: no group
  signal after a clean exit, the helper survives, the download is discarded.
- a_helper_holding_the_pipes_after_exit_is_an_error: the helper now outlives
  the call in serve mode too.
…h compaction

Merges PR 2's docs and demo with PR 1's compaction on sc+git:: (#370). Both
now hold: `sc remote compact` works on a carrier, `sc serve --store sc+git::`
works behind a server-side private-branch gate, the manifest read is one
filtered fetch, and every fetch runs under a download watchdog.

ADR-0060 gains "Serving a Git store", the fetch-based read with its
abbreviation guard, and the watchdog with its exact limits: the group is
signalled only while git is unreaped, so a helper that outlives git's exit is
not killed and may keep writing after the discard. The planned fix, a
group-leading `sh` supervisor, is noted there and as ROADMAP follow-on 10.
THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md, README and the sc-cli-reference skill
carry the same facts. ROADMAP's `sc+git::` follow-ons mark (2) and (9) built and
the manifest-read part of (7) done, next to PR 1's list.

The carrier demo keeps PR 1's compaction step (4) and adds the serve step (7),
the traced filtered-read step (8) and the zero-residue step (9), on the bare
repository and sc-forge legs.
Sol review round 4 on serve --store sc+git::.

A carrier whose log/, checkpoints/ or packs/*.idx family passed the 2 GiB family
download budget could never be read, including by `sc remote compact`, whose
cold refresh reads the same families, so the store could not compact itself out
of that state.

When a whole-family fetch ends by passing its budget (the watchdog already
killed and discarded it), get_family now falls back to the watched single-key
path for the requested key. Each exact-key fetch has its own value cap (256 MiB,
2 GiB for a .pack). The instance remembers that the family is too big, prints
one note on stderr, and reads that family's other keys by single fetches from
then on, with no further family attempts; each family is judged on its own. A
family fetch that fails for any other reason is an error as before, and a
hostile value in an oversized family still fails its own cap. The budget breach
is told from other failures by a flag the watchdog sets (Watch.over), not by
message text.

The family-budget error text no longer says it needs compaction: an oversized
family is no longer a dead end. GitBucket::cap_family_fetches_at is a hidden
test hook so other crates can make a family "too big" without gigabytes.

Tests, each failing without the fix (checked by mutation):
- an_oversized_family_degrades_to_one_fetch_per_key: a counting shim shows one
  family attempt per family, later keys of it go straight to single fetches, and
  an absent key of an oversized family is absent.
- a_hostile_value_in_an_oversized_family_still_fails_its_own_cap.
- a_carrier_whose_families_exceed_the_family_budget_still_compacts (repo, over a
  real GitBucket with the family budget capped at one byte): both runs complete
  and the second deletes what aged.

ADR-0060, THREAT-MODEL and CLAUDE.md say the family budget is not a cliff.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T15:33:46.305790Z 61013a7 PR opened
🔒 Security Review ✅ Completed 2026-11-21T15:40:01.411890Z 61013a7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 21, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 763aad45-e7cc-4bc6-a998-ba2c96860d37
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@tonibergholm tonibergholm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after agent reviews (Mythos, Astra, code-reviewer, 5 sol-loop rounds) and green CI on 61013a7.

@tonibergholm-codento
tonibergholm-codento merged commit 2430b39 into main Nov 21, 2026
11 checks passed
@tonibergholm-codento
tonibergholm-codento deleted the feat/sc-git-serve-store branch November 21, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants