Repository navigation
feat: compact sc+git:: stores so their refs stop growing - #370
Merged
Merged
Conversation
Bucket gains list_tags (keys with the tag get would report, one listing, no downloads) and delete_leased (delete each (key, tag) only if the key still holds that tag). The defaults are an error and an unconditional delete; GitBucket overrides both. GitBucket::delete_leased pushes ':refs/sc/K' with one --force-with-lease=refs/sc/K:<oid> per key (never '+', never --atomic) in chunks of at most 128 refs and 24 KiB of arguments. A key re-created since the caller's listing holds another oid and survives; after a refused push every ref of the chunk is re-read, and a key still at its lease is an error, so a hook refusal is never mistaken for a lost race. Contract suite covers the common part; the git tests cover a moved ref, a re-created ref, an update hook refusing one ref, bad keys and leases, and the command line (one lease per ref, no '+', under 32 KiB).
A Git host reports no trustworthy object age (committer time is writer-controlled), so v1 refused 'sc remote compact' there. Compaction now keeps its own clock. Age source: refs/sc/compact/sightings holds '<first_seen_unix_secs> <oid> <key>' per packs/ and checkpoints/ key, generation-aware by oid, strictly decoded (a bad record, or a time over five minutes ahead of the compactor, blocks the run before anything is written). It supplies ages only; the candidates still come from the listing minus the live chain and retained structure. It is written with a CAS before any delete decision uses it; a dry run never writes it. Sizes: the view keeps each live idx's declared pack length, so the merge plan never calls list_meta (which downloads every value on a carrier); a pack far over what its idx declares is left unmerged. Deletes: leased to the oid the one listing showed (Bucket::delete_leased), idx before bodies. Before every destructive batch, on every backend, the manifest is read with no cache and its tag compared (a carrier answers a conditional read from its 2 s window with no network call). The v1 refusals (repo compact(), refuse_git_compact, the CLI check) are removed. Tests run over a real GitBucket on file:// hubs with an injected clock and cover two runs, backdated, malformed, future-dated and manifest- naming records, dry run, a re-created key, a push landing before the manifest check, a key re-created before its leased delete, two compactors racing on the record, crashes at each delete call, a refusing host, and every operation boundary over an in-memory double with Git's tag semantics. The CLI scenario compacts twice and then clones.
ADR-0057 gets an addendum with the first-sighting record, idx sizing, leased deletes and the uncached manifest check; ADR-0060 drops "compaction refused", corrects the freshness-window sentence (a stale view is not harmless for a delete), and records that every pusher holds delete and that deleting a ref is not erasure. THREAT-MODEL, ROADMAP (follow-ons 1, 5a, 6, 7 addressed as far as they go; 3, 5b, 8 and S2 declined with reasons), CLAUDE.md, ARCHITECTURE.md, README and the sc-cli-reference skill follow. demo/run_git_carrier_demo.sh gains a compaction step in both legs (bare repo and sc-forge): two runs a grace window apart, printing the refs/sc/* count before and after, ahead of the keyless and keyed clones.
The crash test now uploads a pack that only the crashing run sees, so the run has something new to record, and asserts the record holds it at that run's time when the crash comes at each delete call. Also: the sweep comments and the exhaustion note say that log and checkpoint deletes wait for the uncached manifest check too (a busy remote can defer cleanup to a rerun), ADR-0057 and CLAUDE.md say the same, and two doc sentences are tightened.
Each behavior fix has a test that fails without it (checked by reverting it). - The manifest check is pinned: the late-push test now runs over a bucket that answers a conditional manifest read Unchanged from its freshness window, so a check that reuses the cached tag deletes the pack and fails the test. - A hash is deleted only when every present object of it has aged: a test ages a lone .pack, uploads a fresh .idx, and expects both to survive. - The sweep clock is read per attempt, so a key first listed after a replan is not stamped with the time the sweep began. - A retained log entry or checkpoint whose tag differs from its listed tag keeps every candidate pack (a carrier serves create-only keys from its cache). - Byte sums saturate (a hostile idx can declare u64::MAX, which panicked a debug build), including the merge planner's group sum. - The note for exhausted attempts names the age record when that is what lost; the carrier byte-count note appears whenever a carrier run deletes anything. Docs and comments: report counts are attempts, from_bucket takes the unleased path (Git carriers must use open), the stale list_meta comment, the late-push test comment, THREAT-MODEL (an understating idx is downloaded whole), ADR-0057, and the decoy-branch push failure recorded as a known issue in ROADMAP.
…re the body deletes Each fix has a test that fails without it (checked by reverting it). - The sweep clock on a carrier is read after the tagged listing returns, and that one instant is both the first-seen stamp and the age cutoff. Before, it was read at the start of the attempt: a process that stalled before listing stamped a pack uploaded during the stall with an older time, and the next run could delete it while its push was still staging. - A batch of pack deletes is two calls (idx files, then bodies). The uncached manifest check now runs before each: a writer can reuse a body that is still there (put_new leaves its oid unchanged), create the idx again and commit between the calls, and the body's lease still matched. A stop in that window keeps the bodies and says so. Byte counts are added per call so a stop reports only what was attempted. - The default Bucket::delete_leased is an error instead of an unconditional delete, so a lease is never silently dropped. Only the Git carrier overrides it; compaction deletes from the other backends with plain delete (tested). The lease contract checks move to leased_delete_suite, run on GitBucket only.
A lone aged .idx (body already gone) made the report count the declared length of a body that no longer exists.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
tonibergholm
approved these changes
Nov 21, 2026
tonibergholm
left a comment
Member
There was a problem hiding this comment.
Approved after agent reviews (Mythos, Astra, code-reviewer, sol-loop) and green CI on 2159061.
tonibergholm-codento
added a commit
that referenced
this pull request
Nov 21, 2026
… and cap fetch downloads (#371) * feat(objio): read a mutable key with one filtered fetch and watch every fetch The manifest read was `ls-remote` followed by `fetch`. It is now one `git fetch --depth=1 +refs/sc/K:refs/sc/K` under protocol v2, so the host filters its ref advertisement to that ref (verified with GIT_TRACE_PACKET against a local bare repository and against sc-forge over smart HTTP). A failed fetch falls back to an exact `ls-remote` to tell an absent ref from an error. Git resolves a refspec source that matches nothing exactly by its abbreviation rules, so a branch or tag spelled `refs/sc/K` after its own prefix could stand in for an absent key; the fetch carries five negative refspecs that keep the exact-name semantics of the old read. `git fetch` has no input-size limit, so every fetch now runs under a watchdog: `fetch.unpackLimit=1` puts the download into one `tmp_pack_*` file, the growth of `objects/pack` is polled every 50 ms, and git is killed over the budget (the key's value cap plus 0.1% and 1 MiB, 2 GiB for a family or batch) or after 120 s without progress. A fetch that ends between two polls is caught by a check after it exits. A breach removes the files the fetch added, every local ref and the lock files a killed git cannot release, and the error names the cap. The bound is disk per fetch, not RAM inside index-pack, not the total cache. Pipe readers are detached threads, so a helper that outlives a killed git cannot block the call. Serve mode (unix) runs git in its own process group and kills the group before reaping the leader; the CLI keeps Ctrl-C and ssh prompts by killing git alone. GitBucket::open_in puts the cache under a given directory and enables serve mode. GitBucket::reap_stale removes caches of dead owners only, so several `sc serve --stdio` processes can share one home. Tests: one filtered fetch per manifest read with the packet trace, decoys at all five abbreviations, oversized planted value in both modes and both check paths with a working instance afterwards, a runaway fetch killed with its descendants and locks, the no-progress deadline, a helper holding the pipes, the reaper, and a forge end-to-end check of the filtered advertisement. * feat(serve): serve --store sc+git:: behind a server-side private-branch gate `sc serve --store sc+git::<url>` now works over --stdio and --http. The daemon needs git to authenticate without prompting. The private-branch opt-in moves to the operator: `--allow-private-branches` (refused for any other store). Without it the server refuses, as InvalidArgument with no wire change, every object of a sealed-branch kind (Sealed, Manifest): - PutPack stops at the first such object before upload_pack, in any record order, so a mixed pack adds no refs/sc/*. - PutObject refuses before staging, so a later UpdateRef cannot flush it. - UpdateRef refuses a manifest tip even when its objects are already on the chain, so an existing private branch cannot be advanced or re-pointed. The flag lives on BucketTransport (set by open_for_serve) and is plumbed through WirePolicy and ServeLimits; false means gated. The serve cache lives under the serve home's .sc/tmp/ and `sc serve` reaps dead processes' caches at startup. Tests drive the hostile orders against a real bare repository and inspect it afterwards (no refs, no objects, no sealed object in any stored pack), over pipes and through a real HTTP listener with real clients, and through the binary. Disabling each of the three checks in turn fails a test. * test(objio): a discarded hostile value with a parent leaves the cache usable A depth-1 fetch records a value commit that has a parent as a shallow boundary, and a discarded download deletes that object. The new test checks that the same instance still reads a good key and refuses the hostile one again, in serve and CLI mode and on both check paths. The timing upper bounds of the watchdog tests move to 90 s (they only guard against a hang of 600 s) and wait_gone waits up to 20 s, so a loaded machine does not fail them. * fix(objio): reap serve caches by file lock, and harden the fetch watchdog Review fixes for serve --store sc+git::. Reaper liveness is now an advisory file lock, not a pid. Each bucket opened by open_in holds File::lock on <cache>/.owner.lock for its life; reap_stale removes a sc-git-bucket-* directory only when it can take that lock (never a symlink, a directory without a lock file, or another name). That drops the dependency on the `kill` binary and the own-pid exemption, and survives pid reuse, pid-1 containers and several pid namespaces on one host. The sweep holds the lock while it removes the directory, and an opener whose lock file vanished starts over. One host only; unreliable over NFS, as for DirBucket. Watchdog: - If the `kill` program for a process group cannot be run, serve mode falls back to killing git alone and warns once on stderr. - After a clean exit with a helper holding the pipes, both pipe receives share one deadline (not twice the idle time) and serve mode stops the group before returning the error. - transfer.bundleURI=false is pinned beside fetch.uriProtocols=. - The family budget error says a legitimate store over 2 GiB needs compaction. The serve gate's refusal now says "No private-branch object was uploaded": public objects of the same push may already be on the host as an orphan pack. Docs: ADR-0060, THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md and the skill state the lock semantics, the `kill` dependency, that only serve mode on unix is disk-bounded against a hostile HTTP host, that refs written by a family fetch are not measured, the 2 GiB family cliff, the unwatched ls-remote and push (ServerAliveInterval over ssh), that a fetch group outlives a killed sc serve, that an ssh user who runs sc serve is an operator, the per-connection cold cache cost times --max-connections, and the 2 ms to 50 ms poll back-off. forge e2e: the new test no longer splits raw_update from its doc comment and fetches with all five negative refspecs. Tests (each fails without its fix): reap_stale_goes_by_the_owner_lock_not_by_pid, a_sweep_never_removes_a_live_bucket_while_others_open (a smoke test of the create/sweep race), the_pipe_wait_after_exit_has_one_deadline, a_helper_holding_the_pipes_after_exit_is_an_error (group variant), a_missing_kill_program_falls_back_to_killing_git_and_warns_once, the_forced_config_pins_the_download_redirect_switches, a_family_fetch_over_budget_names_the_cap_and_compaction, the serve-gate wording in git_serve_tests, and serve_start_reaps_caches_nobody_holds. * fix(objio): absolute cache path, and one terminate-and-discard path for every breach Sol review round 1 on serve --store sc+git::. S1: git runs with the cache directory as its working directory and GIT_DIR set to the same path, so a relative serve home (`.`) was resolved from inside the cache and every Git-backed session failed ("not a git repository: ./.sc/tmp/sc-git-bucket-..."). open_full now makes the cache parent absolute (std::path::absolute) before any git launch, for the serve home and for the system temp directory alike. S2: the final budget check discarded the download without handling a surviving helper, and the pipe-timeout path stopped the group without discarding. Every breach (budget during the run, budget at the final check, no progress, a pipe held open after exit) now takes terminate_and_discard: stop git (the group in serve mode), reap it, keep watching until the pipes have closed and objects/pack has stopped changing or a 5 s settle deadline passes, then discard what the fetch added. Tests, each failing without its fix (checked by mutation): - a_relative_cache_parent_works (objio) and serve_store_sc_git_works_with_a_relative_serve_home (cli, serve home `.` with the server's cwd set to it, push and clone through it). - a_clean_exit_over_budget_still_stops_a_helper_and_discards (final check). - a_helper_holding_the_pipes_after_exit_is_an_error and a_silent_fetch_hits_the_no_progress_deadline now also assert that the added files are gone; a_runaway_fetch_is_killed_with_its_descendants covers the in-run budget path. Serve mode asserts no surviving helper. ADR-0060 and THREAT-MODEL describe the single breach path and the absolute cache path. * fix(objio): keep measuring after git exits, and signal the group only with evidence Sol review round 2 on serve --store sc+git::. S2: the final size check ran before the pipe wait, so a helper that outlived git could cross the budget during that wait and close both pipes before the deadline, and the watchdog returned success without discarding the oversized download. After git exits it now polls objects/pack for the one deadline both pipes share, and measures once more after both pipes have closed, before it returns success. A breach there takes terminate_and_discard like every other. S3: once the leader is reaped, an empty process group's id can be reused by an unrelated group of the same user, and stop() signalled -<pgid> anyway. The decision is now a small pure function, may_signal_group: free while the leader is unreaped (its pid pins the group id), and after the reap only while a pipe reader is still waiting (an open pipe has a live process holding its write end). With both pipes closed nothing is signalled; the watchdog only settles and discards. A try_wait error counts as an unknown reap state: no group signal. Tests, each failing without its fix (checked by mutation): - a_helper_that_crosses_the_budget_after_git_exits_is_caught (serve and CLI mode): the helper writes past the budget after git exited and closes its pipes. - the_group_is_signalled_only_when_its_id_is_pinned_or_a_member_is_proven (the decision's truth table). - a_reaped_leader_with_both_pipes_closed_gets_no_group_signal, with a recording kill program: no signal is sent, and the download is still discarded. - a_clean_exit_over_budget_still_stops_a_helper_and_discards now also asserts the one group signal in serve mode when a helper holds the pipes. ADR-0060 and THREAT-MODEL describe the post-exit measuring and the signalling rule. * fix(objio): never signal a git process group after its leader is reaped Sol review round 3 on serve --store sc+git::. An open pipe proves a live group member only at the instant it is looked at, and the external `kill` runs later, so once the leader is reaped any group signal can hit a reused group id of an unrelated process of the same user. Holding the pid without reaping needs waitid(WNOWAIT), which is unsafe and not allowed here. may_signal_group now takes only the leader's state and returns true only while the leader is unreaped (its pid pins the group id); after the reap the group is never signalled, whatever the pipes say. A breach found after the reap still settles for a bounded time and discards what the fetch added. The cost is stated in the function's doc comment, ADR-0060, THREAT-MODEL, CLAUDE.md and ARCHITECTURE.md: a helper that outlives git's exit (a smart-HTTP v0/v1 helper chain, an ssh ControlPersist master, a hostile process) is not killed in any mode and may keep writing after the discard until it exits. Tests, each failing without the fix (checked by mutation): - the_group_is_signalled_only_while_the_leader_is_unreaped (truth table). - a_reaped_leader_gets_no_group_signal_even_with_a_pipe_open, with a recording kill program, for a helper that holds the pipes and for none; the download is discarded in both. - a_clean_exit_over_budget_discards_but_leaves_the_helper_alone: no group signal after a clean exit, the helper survives, the download is discarded. - a_helper_holding_the_pipes_after_exit_is_an_error: the helper now outlives the call in serve mode too. * docs: serve --store sc+git::, the filtered read and the watchdog, with compaction Merges PR 2's docs and demo with PR 1's compaction on sc+git:: (#370). Both now hold: `sc remote compact` works on a carrier, `sc serve --store sc+git::` works behind a server-side private-branch gate, the manifest read is one filtered fetch, and every fetch runs under a download watchdog. ADR-0060 gains "Serving a Git store", the fetch-based read with its abbreviation guard, and the watchdog with its exact limits: the group is signalled only while git is unreaped, so a helper that outlives git's exit is not killed and may keep writing after the discard. The planned fix, a group-leading `sh` supervisor, is noted there and as ROADMAP follow-on 10. THREAT-MODEL, CLAUDE.md, ARCHITECTURE.md, README and the sc-cli-reference skill carry the same facts. ROADMAP's `sc+git::` follow-ons mark (2) and (9) built and the manifest-read part of (7) done, next to PR 1's list. The carrier demo keeps PR 1's compaction step (4) and adds the serve step (7), the traced filtered-read step (8) and the zero-residue step (9), on the bare repository and sc-forge legs. * fix(objio): an oversized key family degrades to one fetch per key Sol review round 4 on serve --store sc+git::. A carrier whose log/, checkpoints/ or packs/*.idx family passed the 2 GiB family download budget could never be read, including by `sc remote compact`, whose cold refresh reads the same families, so the store could not compact itself out of that state. When a whole-family fetch ends by passing its budget (the watchdog already killed and discarded it), get_family now falls back to the watched single-key path for the requested key. Each exact-key fetch has its own value cap (256 MiB, 2 GiB for a .pack). The instance remembers that the family is too big, prints one note on stderr, and reads that family's other keys by single fetches from then on, with no further family attempts; each family is judged on its own. A family fetch that fails for any other reason is an error as before, and a hostile value in an oversized family still fails its own cap. The budget breach is told from other failures by a flag the watchdog sets (Watch.over), not by message text. The family-budget error text no longer says it needs compaction: an oversized family is no longer a dead end. GitBucket::cap_family_fetches_at is a hidden test hook so other crates can make a family "too big" without gigabytes. Tests, each failing without the fix (checked by mutation): - an_oversized_family_degrades_to_one_fetch_per_key: a counting shim shows one family attempt per family, later keys of it go straight to single fetches, and an absent key of an oversized family is absent. - a_hostile_value_in_an_oversized_family_still_fails_its_own_cap. - a_carrier_whose_families_exceed_the_family_budget_still_compacts (repo, over a real GitBucket with the family budget capped at one byte): both runs complete and the second deletes what aged. ADR-0060, THREAT-MODEL and CLAUDE.md say the family budget is not a cliff.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An
sc+git::store (#369) only grew.sc remote compactrefused it, so every push added at least threerefs/sc/*refs for good. Everyls-remoteand push then downloaded a longer ref list, and a cold read fetched every log entry and checkpoint. Compaction was refused because it deletes by age, and a Git host has no trustworthy age: the committer date is set by the writer and predates publication.This PR lets
sc remote compactwork onsc+git::remotes:compact/sightings, with one line per key: the first time the compactor saw it, on the compactor's own clock, plus the commit id it saw. A changed commit id counts as a new sighting, so a re-uploaded pack starts its grace period over. The clock is read after the listing returns. The record is parsed strictly, a bad record blocks compaction, and a dry run never writes it. The record only supplies ages: a key the live chain or retained structure names is never deleted, whatever the record says.--force-with-lease=refs/sc/K:<oid>with:refs/sc/K, never+), so a key re-created after the listing survives.Bucket::delete_leasedhas no default that drops the lease, andBucket::list_tagslists keys with their tags without downloading values.ADR-0057 has an addendum, ADR-0060 and THREAT-MODEL are updated, and ROADMAP records the remaining follow-ons.
Proof:
demo/run_git_carrier_demo.shnow compacts twice in both legs, a bare repo and a real sc-forge over smart HTTP. Therefs/sc/*count goes 7 → 12 → 5: the first run only records sightings, the second deletes what has aged. A keyless clone and a keyed clone still read everything afterwards. New tests over a realGitBucketcover:Each new rule was also broken on purpose, and its test failed. fmt, clippy
-D warningsandcargo test --workspacepass.The plan was reviewed by Fable, Mythos (security), Astra and code-reviewer. The diff was reviewed by Mythos and code-reviewer and with sol-loop.
Still open, as follow-ons:
Reviewed with sol-loop: 3 rounds with gpt-6.2-sol on high. Sol verdict: approved
Created with Claude Opus 5.5 in Claude Code (T3 Code), with Claude Sonnet 5.5 subagents writing most of the implementation.