Skip to content

feat: compact sc+git:: stores so their refs stop growing - #370

Merged
tonibergholm-codento merged 7 commits into
mainfrom
feat/sc-git-compaction
Nov 21, 2026
Merged

tonibergholm-codento merged 7 commits into
mainfrom
feat/sc-git-compaction

Conversation

@tonibergholm-codento

Copy link
Copy Markdown
Collaborator

An sc+git:: store (#369) only grew. sc remote compact refused it, so every push added at least three refs/sc/* refs for good. Every ls-remote and push then downloaded a longer ref list, and a cold read fetched every log entry and checkpoint. Compaction was refused because it deletes by age, and a Git host has no trustworthy age: the committer date is set by the writer and predates publication.

This PR lets sc remote compact work on sc+git:: remotes:

  • Its own age record. Compaction keeps an age record at compact/sightings, with one line per key: the first time the compactor saw it, on the compactor's own clock, plus the commit id it saw. A changed commit id counts as a new sighting, so a re-uploaded pack starts its grace period over. The clock is read after the listing returns. The record is parsed strictly, a bad record blocks compaction, and a dry run never writes it. The record only supplies ages: a key the live chain or retained structure names is never deleted, whatever the record says.
  • Leased deletes. Each delete is leased to the commit id the listing showed (--force-with-lease=refs/sc/K:<oid> with :refs/sc/K, never +), so a key re-created after the listing survives. Bucket::delete_leased has no default that drops the lease, and Bucket::list_tags lists keys with their tags without downloading values.
  • Fresh manifest checks. Before every delete call, an uncached manifest read must show the manifest unmoved. This covers the idx deletes and the body deletes separately, and it now runs on every backend.
  • Sizes without downloads. Merge planning takes pack sizes from each live idx instead of downloading every pack.

ADR-0057 has an addendum, ADR-0060 and THREAT-MODEL are updated, and ROADMAP records the remaining follow-ons.

Proof: demo/run_git_carrier_demo.sh now compacts twice in both legs, a bare repo and a real sc-forge over smart HTTP. The refs/sc/* count goes 7 → 12 → 5: the first run only records sightings, the second deletes what has aged. A keyless clone and a keyed clone still read everything afterwards. New tests over a real GitBucket cover:

  • two-run deletion,
  • a re-created key,
  • a key re-created between the listing and the delete,
  • a push landing after the listing, and between the idx and body deletes,
  • two compactors racing on the record,
  • a failure at every delete call,
  • a refusing host,
  • malformed, future and backdated records,
  • the every-member-aged rule,
  • saturating byte counts.

Each new rule was also broken on purpose, and its test failed. fmt, clippy -D warnings and cargo test --workspace pass.

The plan was reviewed by Fable, Mythos (security), Astra and code-reviewer. The diff was reviewed by Mythos and code-reviewer and with sol-loop.

Still open, as follow-ons:

  • Clocks. Run compaction from one host, because ages compare the clocks of whichever hosts ran it.
  • Push access. Every pusher holds delete rights on a Git host.
  • Erasure. Deleting a ref does not erase its data.
  • GitHub. Untested.

Reviewed with sol-loop: 3 rounds with gpt-6.2-sol on high. Sol verdict: approved

Created with Claude Opus 5.5 in Claude Code (T3 Code), with Claude Sonnet 5.5 subagents writing most of the implementation.

Bucket gains list_tags (keys with the tag get would report, one listing,
no downloads) and delete_leased (delete each (key, tag) only if the key
still holds that tag). The defaults are an error and an unconditional
delete; GitBucket overrides both.

GitBucket::delete_leased pushes ':refs/sc/K' with one
--force-with-lease=refs/sc/K:<oid> per key (never '+', never --atomic) in
chunks of at most 128 refs and 24 KiB of arguments. A key re-created since
the caller's listing holds another oid and survives; after a refused push
every ref of the chunk is re-read, and a key still at its lease is an error,
so a hook refusal is never mistaken for a lost race.

Contract suite covers the common part; the git tests cover a moved ref, a
re-created ref, an update hook refusing one ref, bad keys and leases, and
the command line (one lease per ref, no '+', under 32 KiB).
A Git host reports no trustworthy object age (committer time is
writer-controlled), so v1 refused 'sc remote compact' there. Compaction now
keeps its own clock.

Age source: refs/sc/compact/sightings holds '<first_seen_unix_secs> <oid>
<key>' per packs/ and checkpoints/ key, generation-aware by oid, strictly
decoded (a bad record, or a time over five minutes ahead of the compactor,
blocks the run before anything is written). It supplies ages only; the
candidates still come from the listing minus the live chain and retained
structure. It is written with a CAS before any delete decision uses it; a
dry run never writes it.

Sizes: the view keeps each live idx's declared pack length, so the merge plan
never calls list_meta (which downloads every value on a carrier); a pack far
over what its idx declares is left unmerged.

Deletes: leased to the oid the one listing showed (Bucket::delete_leased),
idx before bodies. Before every destructive batch, on every backend, the
manifest is read with no cache and its tag compared (a carrier answers a
conditional read from its 2 s window with no network call).

The v1 refusals (repo compact(), refuse_git_compact, the CLI check) are
removed. Tests run over a real GitBucket on file:// hubs with an injected
clock and cover two runs, backdated, malformed, future-dated and manifest-
naming records, dry run, a re-created key, a push landing before the
manifest check, a key re-created before its leased delete, two compactors
racing on the record, crashes at each delete call, a refusing host, and
every operation boundary over an in-memory double with Git's tag semantics.
The CLI scenario compacts twice and then clones.
ADR-0057 gets an addendum with the first-sighting record, idx sizing, leased
deletes and the uncached manifest check; ADR-0060 drops "compaction refused",
corrects the freshness-window sentence (a stale view is not harmless for a
delete), and records that every pusher holds delete and that deleting a ref is
not erasure. THREAT-MODEL, ROADMAP (follow-ons 1, 5a, 6, 7 addressed as far as
they go; 3, 5b, 8 and S2 declined with reasons), CLAUDE.md, ARCHITECTURE.md,
README and the sc-cli-reference skill follow.

demo/run_git_carrier_demo.sh gains a compaction step in both legs (bare repo and
sc-forge): two runs a grace window apart, printing the refs/sc/* count before
and after, ahead of the keyless and keyed clones.
The crash test now uploads a pack that only the crashing run sees, so the run
has something new to record, and asserts the record holds it at that run's time
when the crash comes at each delete call. Also: the sweep comments and the
exhaustion note say that log and checkpoint deletes wait for the uncached
manifest check too (a busy remote can defer cleanup to a rerun), ADR-0057 and
CLAUDE.md say the same, and two doc sentences are tightened.
Each behavior fix has a test that fails without it (checked by reverting it).

- The manifest check is pinned: the late-push test now runs over a bucket that
  answers a conditional manifest read Unchanged from its freshness window, so a
  check that reuses the cached tag deletes the pack and fails the test.
- A hash is deleted only when every present object of it has aged: a test ages a
  lone .pack, uploads a fresh .idx, and expects both to survive.
- The sweep clock is read per attempt, so a key first listed after a replan is
  not stamped with the time the sweep began.
- A retained log entry or checkpoint whose tag differs from its listed tag
  keeps every candidate pack (a carrier serves create-only keys from its cache).
- Byte sums saturate (a hostile idx can declare u64::MAX, which panicked a debug
  build), including the merge planner's group sum.
- The note for exhausted attempts names the age record when that is what lost;
  the carrier byte-count note appears whenever a carrier run deletes anything.

Docs and comments: report counts are attempts, from_bucket takes the unleased
path (Git carriers must use open), the stale list_meta comment, the late-push
test comment, THREAT-MODEL (an understating idx is downloaded whole), ADR-0057,
and the decoy-branch push failure recorded as a known issue in ROADMAP.
…re the body deletes

Each fix has a test that fails without it (checked by reverting it).

- The sweep clock on a carrier is read after the tagged listing returns, and
  that one instant is both the first-seen stamp and the age cutoff. Before, it
  was read at the start of the attempt: a process that stalled before listing
  stamped a pack uploaded during the stall with an older time, and the next run
  could delete it while its push was still staging.
- A batch of pack deletes is two calls (idx files, then bodies). The uncached
  manifest check now runs before each: a writer can reuse a body that is still
  there (put_new leaves its oid unchanged), create the idx again and commit
  between the calls, and the body's lease still matched. A stop in that window
  keeps the bodies and says so. Byte counts are added per call so a stop reports
  only what was attempted.
- The default Bucket::delete_leased is an error instead of an unconditional
  delete, so a lease is never silently dropped. Only the Git carrier overrides
  it; compaction deletes from the other backends with plain delete (tested). The
  lease contract checks move to leased_delete_suite, run on GitBucket only.
A lone aged .idx (body already gone) made the report count the declared
length of a body that no longer exists.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3f9045d4-2dba-46cc-ae3b-b70c1fffd405
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T14:39:42.977024Z 2159061 PR opened
🔒 Security Review ✅ Completed 2026-11-21T14:44:25.019878Z 2159061 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@tonibergholm tonibergholm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after agent reviews (Mythos, Astra, code-reviewer, sol-loop) and green CI on 2159061.

@tonibergholm-codento
tonibergholm-codento merged commit 7ed3504 into main Nov 21, 2026
11 checks passed
@tonibergholm-codento
tonibergholm-codento deleted the feat/sc-git-compaction branch November 21, 2026 14:52
tonibergholm-codento added a commit that referenced this pull request Nov 21, 2026
… and cap fetch downloads (#371)

* feat(objio): read a mutable key with one filtered fetch and watch every fetch

The manifest read was `ls-remote` followed by `fetch`. It is now one
`git fetch --depth=1 +refs/sc/K:refs/sc/K` under protocol v2, so the host
filters its ref advertisement to that ref (verified with GIT_TRACE_PACKET
against a local bare repository and against sc-forge over smart HTTP). A
failed fetch falls back to an exact `ls-remote` to tell an absent ref from an
error. Git resolves a refspec source that matches nothing exactly by its
abbreviation rules, so a branch or tag spelled `refs/sc/K` after its own
prefix could stand in for an absent key; the fetch carries five negative
refspecs that keep the exact-name semantics of the old read.

`git fetch` has no input-size limit, so every fetch now runs under a
watchdog: `fetch.unpackLimit=1` puts the download into one `tmp_pack_*` file,
the growth of `objects/pack` is polled every 50 ms, and git is killed over the
budget (the key's value cap plus 0.1% and 1 MiB, 2 GiB for a family or batch)
or after 120 s without progress. A fetch that ends between two polls is caught
by a check after it exits. A breach removes the files the fetch added, every
local ref and the lock files a killed git cannot release, and the error names
the cap. The bound is disk per fetch, not RAM inside index-pack, not the total
cache. Pipe readers are detached threads, so a helper that outlives a killed
git cannot block the call. Serve mode (unix) runs git in its own process
group and kills the group before reaping the leader; the CLI keeps Ctrl-C and
ssh prompts by killing git alone.

GitBucket::open_in puts the cache under a given directory and enables serve
mode. GitBucket::reap_stale removes caches of dead owners only, so several
`sc serve --stdio` processes can share one home.

Tests: one filtered fetch per manifest read with the packet trace, decoys at
all five abbreviations, oversized planted value in both modes and both check
paths with a working instance afterwards, a runaway fetch killed with its
descendants and locks, the no-progress deadline, a helper holding the pipes,
the reaper, and a forge end-to-end check of the filtered advertisement.

* feat(serve): serve --store sc+git:: behind a server-side private-branch gate

`sc serve --store sc+git::<url>` now works over --stdio and --http. The
daemon needs git to authenticate without prompting. The private-branch
opt-in moves to the operator: `--allow-private-branches` (refused for any
other store). Without it the server refuses, as InvalidArgument with no wire
change, every object of a sealed-branch kind (Sealed, Manifest):

- PutPack stops at the first such object before upload_pack, in any record
  order, so a mixed pack adds no refs/sc/*.
- PutObject refuses before staging, so a later UpdateRef cannot flush it.
- UpdateRef refuses a manifest tip even when its objects are already on the
  chain, so an existing private branch cannot be advanced or re-pointed.

The flag lives on BucketTransport (set by open_for_serve) and is plumbed
through WirePolicy and ServeLimits; false means gated. The serve cache lives
under the serve home's .sc/tmp/ and `sc serve` reaps dead processes' caches
at startup.

Tests drive the hostile orders against a real bare repository and inspect it
afterwards (no refs, no objects, no sealed object in any stored pack), over
pipes and through a real HTTP listener with real clients, and through the
binary. Disabling each of the three checks in turn fails a test.

* test(objio): a discarded hostile value with a parent leaves the cache usable

A depth-1 fetch records a value commit that has a parent as a shallow
boundary, and a discarded download deletes that object. The new test checks
that the same instance still reads a good key and refuses the hostile one
again, in serve and CLI mode and on both check paths. The timing upper bounds
of the watchdog tests move to 90 s (they only guard against a hang of 600 s)
and wait_gone waits up to 20 s, so a loaded machine does not fail them.

* fix(objio): reap serve caches by file lock, and harden the fetch watchdog

Review fixes for serve --store sc+git::.

Reaper liveness is now an advisory file lock, not a pid. Each bucket opened
by open_in holds File::lock on <cache>/.owner.lock for its life; reap_stale
removes a sc-git-bucket-* directory only when it can take that lock (never a
symlink, a directory without a lock file, or another name). That drops the
dependency on the `kill` binary and the own-pid exemption, and survives pid
reuse, pid-1 containers and several pid namespaces on one host. The sweep
holds the lock while it removes the directory, and an opener whose lock file
vanished starts over. One host only; unreliable over NFS, as for DirBucket.

Watchdog:
- If the `kill` program for a process group cannot be run, serve mode falls
  back to killing git alone and warns once on stderr.
- After a clean exit with a helper holding the pipes, both pipe receives share
  one deadline (not twice the idle time) and serve mode stops the group before
  returning the error.
- transfer.bundleURI=false is pinned beside fetch.uriProtocols=.
- The family budget error says a legitimate store over 2 GiB needs compaction.

The serve gate's refusal now says "No private-branch object was uploaded":
public objects of the same push may already be on the host as an orphan pack.

Docs: ADR-0060, THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md and the skill state
the lock semantics, the `kill` dependency, that only serve mode on unix is
disk-bounded against a hostile HTTP host, that refs written by a family fetch
are not measured, the 2 GiB family cliff, the unwatched ls-remote and push
(ServerAliveInterval over ssh), that a fetch group outlives a killed sc serve,
that an ssh user who runs sc serve is an operator, the per-connection cold
cache cost times --max-connections, and the 2 ms to 50 ms poll back-off.

forge e2e: the new test no longer splits raw_update from its doc comment and
fetches with all five negative refspecs.

Tests (each fails without its fix): reap_stale_goes_by_the_owner_lock_not_by_pid,
a_sweep_never_removes_a_live_bucket_while_others_open (a smoke test of the
create/sweep race), the_pipe_wait_after_exit_has_one_deadline,
a_helper_holding_the_pipes_after_exit_is_an_error (group variant),
a_missing_kill_program_falls_back_to_killing_git_and_warns_once,
the_forced_config_pins_the_download_redirect_switches,
a_family_fetch_over_budget_names_the_cap_and_compaction, the serve-gate
wording in git_serve_tests, and serve_start_reaps_caches_nobody_holds.

* fix(objio): absolute cache path, and one terminate-and-discard path for every breach

Sol review round 1 on serve --store sc+git::.

S1: git runs with the cache directory as its working directory and GIT_DIR
set to the same path, so a relative serve home (`.`) was resolved from inside
the cache and every Git-backed session failed ("not a git repository:
./.sc/tmp/sc-git-bucket-..."). open_full now makes the cache parent absolute
(std::path::absolute) before any git launch, for the serve home and for the
system temp directory alike.

S2: the final budget check discarded the download without handling a
surviving helper, and the pipe-timeout path stopped the group without
discarding. Every breach (budget during the run, budget at the final check,
no progress, a pipe held open after exit) now takes terminate_and_discard:
stop git (the group in serve mode), reap it, keep watching until the pipes
have closed and objects/pack has stopped changing or a 5 s settle deadline
passes, then discard what the fetch added.

Tests, each failing without its fix (checked by mutation):
- a_relative_cache_parent_works (objio) and
  serve_store_sc_git_works_with_a_relative_serve_home (cli, serve home `.`
  with the server's cwd set to it, push and clone through it).
- a_clean_exit_over_budget_still_stops_a_helper_and_discards (final check).
- a_helper_holding_the_pipes_after_exit_is_an_error and
  a_silent_fetch_hits_the_no_progress_deadline now also assert that the
  added files are gone; a_runaway_fetch_is_killed_with_its_descendants
  covers the in-run budget path. Serve mode asserts no surviving helper.

ADR-0060 and THREAT-MODEL describe the single breach path and the absolute
cache path.

* fix(objio): keep measuring after git exits, and signal the group only with evidence

Sol review round 2 on serve --store sc+git::.

S2: the final size check ran before the pipe wait, so a helper that outlived
git could cross the budget during that wait and close both pipes before the
deadline, and the watchdog returned success without discarding the oversized
download. After git exits it now polls objects/pack for the one deadline both
pipes share, and measures once more after both pipes have closed, before it
returns success. A breach there takes terminate_and_discard like every other.

S3: once the leader is reaped, an empty process group's id can be reused by an
unrelated group of the same user, and stop() signalled -<pgid> anyway. The
decision is now a small pure function, may_signal_group: free while the leader
is unreaped (its pid pins the group id), and after the reap only while a pipe
reader is still waiting (an open pipe has a live process holding its write
end). With both pipes closed nothing is signalled; the watchdog only settles
and discards. A try_wait error counts as an unknown reap state: no group signal.

Tests, each failing without its fix (checked by mutation):
- a_helper_that_crosses_the_budget_after_git_exits_is_caught (serve and CLI
  mode): the helper writes past the budget after git exited and closes its pipes.
- the_group_is_signalled_only_when_its_id_is_pinned_or_a_member_is_proven (the
  decision's truth table).
- a_reaped_leader_with_both_pipes_closed_gets_no_group_signal, with a recording
  kill program: no signal is sent, and the download is still discarded.
- a_clean_exit_over_budget_still_stops_a_helper_and_discards now also asserts
  the one group signal in serve mode when a helper holds the pipes.

ADR-0060 and THREAT-MODEL describe the post-exit measuring and the signalling
rule.

* fix(objio): never signal a git process group after its leader is reaped

Sol review round 3 on serve --store sc+git::.

An open pipe proves a live group member only at the instant it is looked at,
and the external `kill` runs later, so once the leader is reaped any group
signal can hit a reused group id of an unrelated process of the same user.
Holding the pid without reaping needs waitid(WNOWAIT), which is unsafe and not
allowed here. may_signal_group now takes only the leader's state and returns
true only while the leader is unreaped (its pid pins the group id); after the
reap the group is never signalled, whatever the pipes say. A breach found after
the reap still settles for a bounded time and discards what the fetch added.

The cost is stated in the function's doc comment, ADR-0060, THREAT-MODEL,
CLAUDE.md and ARCHITECTURE.md: a helper that outlives git's exit (a smart-HTTP
v0/v1 helper chain, an ssh ControlPersist master, a hostile process) is not
killed in any mode and may keep writing after the discard until it exits.

Tests, each failing without the fix (checked by mutation):
- the_group_is_signalled_only_while_the_leader_is_unreaped (truth table).
- a_reaped_leader_gets_no_group_signal_even_with_a_pipe_open, with a recording
  kill program, for a helper that holds the pipes and for none; the download
  is discarded in both.
- a_clean_exit_over_budget_discards_but_leaves_the_helper_alone: no group
  signal after a clean exit, the helper survives, the download is discarded.
- a_helper_holding_the_pipes_after_exit_is_an_error: the helper now outlives
  the call in serve mode too.

* docs: serve --store sc+git::, the filtered read and the watchdog, with compaction

Merges PR 2's docs and demo with PR 1's compaction on sc+git:: (#370). Both
now hold: `sc remote compact` works on a carrier, `sc serve --store sc+git::`
works behind a server-side private-branch gate, the manifest read is one
filtered fetch, and every fetch runs under a download watchdog.

ADR-0060 gains "Serving a Git store", the fetch-based read with its
abbreviation guard, and the watchdog with its exact limits: the group is
signalled only while git is unreaped, so a helper that outlives git's exit is
not killed and may keep writing after the discard. The planned fix, a
group-leading `sh` supervisor, is noted there and as ROADMAP follow-on 10.
THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md, README and the sc-cli-reference skill
carry the same facts. ROADMAP's `sc+git::` follow-ons mark (2) and (9) built and
the manifest-read part of (7) done, next to PR 1's list.

The carrier demo keeps PR 1's compaction step (4) and adds the serve step (7),
the traced filtered-read step (8) and the zero-residue step (9), on the bare
repository and sc-forge legs.

* fix(objio): an oversized key family degrades to one fetch per key

Sol review round 4 on serve --store sc+git::.

A carrier whose log/, checkpoints/ or packs/*.idx family passed the 2 GiB family
download budget could never be read, including by `sc remote compact`, whose
cold refresh reads the same families, so the store could not compact itself out
of that state.

When a whole-family fetch ends by passing its budget (the watchdog already
killed and discarded it), get_family now falls back to the watched single-key
path for the requested key. Each exact-key fetch has its own value cap (256 MiB,
2 GiB for a .pack). The instance remembers that the family is too big, prints
one note on stderr, and reads that family's other keys by single fetches from
then on, with no further family attempts; each family is judged on its own. A
family fetch that fails for any other reason is an error as before, and a
hostile value in an oversized family still fails its own cap. The budget breach
is told from other failures by a flag the watchdog sets (Watch.over), not by
message text.

The family-budget error text no longer says it needs compaction: an oversized
family is no longer a dead end. GitBucket::cap_family_fetches_at is a hidden
test hook so other crates can make a family "too big" without gigabytes.

Tests, each failing without the fix (checked by mutation):
- an_oversized_family_degrades_to_one_fetch_per_key: a counting shim shows one
  family attempt per family, later keys of it go straight to single fetches, and
  an absent key of an oversized family is absent.
- a_hostile_value_in_an_oversized_family_still_fails_its_own_cap.
- a_carrier_whose_families_exceed_the_family_budget_still_compacts (repo, over a
  real GitBucket with the family budget capped at one byte): both runs complete
  and the second deletes what aged.

ADR-0060, THREAT-MODEL and CLAUDE.md say the family budget is not a cliff.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants