Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions .claude/skills/sc-cli-reference/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -431,8 +431,8 @@ cargo run --bin sc -- remote add <name> sc+git::<git-url> [--allow-private-branc
# `git ls-remote` lists them; the same repo can also
# hold a `--git` mirror of public branches. Anyone
# with push to the repo holds bucket put + delete;
# deleting a ref is not erasure. `serve --store`
# refuses it in v1; `remote compact` works (below);
# deleting a ref is not erasure. `remote compact`
# and `serve --store` work (below);
# clone --filter refused as for any bucket. Untested
# against real GitHub
cargo run --bin sc -- remote compact <name> [--dry-run] [--prune-expire 24h] [--allow-short-grace]
Expand Down Expand Up @@ -468,6 +468,20 @@ cargo run --bin sc -- serve --http <addr> <home> --store sc+wal://<dir>|sc+s3://
# all content lives in the bucket (instance is
# disposable); <home> still needs `.sc/` for tokens,
# the TLS identity and scratch space
cargo run --bin sc -- serve --http <addr> <home> --store sc+git::<git-url> [--allow-private-branches]
# the same in front of a Git repository (ADR-0060):
# git must authenticate WITHOUT prompting (ssh agent,
# key without passphrase, credential helper). Without
# --allow-private-branches (refused for any other
# store) the server refuses every private-branch
# object and a ref at a manifest before anything
# reaches the host; only the operator can lift it.
# The git cache lives under <home>/.sc/tmp/, locked
# while a session lives; a start reaps caches whose
# lock is free (unix, one host, not NFS). Every
# fetch runs under a download watchdog (disk per
# fetch; 120 s no-progress deadline); serve mode
# needs a `kill` on the PATH to kill git's group
bash demo/run_git_carrier_demo.sh # sc+git:: proof (P39): a bare Git repo as the host;
# protected path + secret + private branch pushed
# (private branch refused without the opt-in), plain
Expand Down
21 changes: 15 additions & 6 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1372,10 +1372,15 @@ length. Git is the system binary (`SC_GIT` overrides) with a scrubbed environmen
transport allowlist (`https`, `http`, `ssh`, `file`; `ext`, `fd` and `git` forced off), a
URL allowlist and no userinfo on `http(s)` URLs (credentials go through git's credential
helper); a private bare repo in `sc-git-bucket-<pid>-<random>` caches fetched
objects and is removed on `Drop`. Scaling: every `ls-remote` and push downloads the
whole ref advertisement (the host does not filter it server-side), and a cold read fetches
every `log/` and `checkpoints/` entry behind the last checkpoint; refs grow by at least three
per push until `sc remote compact` runs.
objects and is removed on `Drop`. A mutable key (the manifest) is read with one `git fetch`
of exactly `refs/sc/K` under protocol v2, so a v2 host lists that ref alone, and every fetch
runs under a watchdog that kills git past its download budget (disk bytes per fetch: a
key's value cap, 2 GiB for a family) or after 120 s without progress (only serve mode on
unix kills git's whole process group, through the system `kill`, and only while git is
unreaped; elsewhere, and for any helper that outlives git's exit, the helpers that write the
pack may keep writing). Scaling: `ls-remote` and push download the whole ref advertisement,
and a cold read fetches every `log/` and `checkpoints/` entry behind the last checkpoint;
refs grow by at least three per push until `sc remote compact` runs.

`sc remote compact` works on a carrier (ADR-0057 addendum). A Git host reports no
trustworthy object age (committer time is writer-controlled), so the compactor keeps its
Expand All @@ -1394,8 +1399,12 @@ A private branch goes to an `sc+git::` remote only under the per-remote opt-in
`--allow-private-branches` (`sc remote set <name> --allow-private-branches` for an existing
remote), since its name, recipient ids and sizes become readable by
every reader of the repository and the ciphertext cannot be taken back.
`sc serve --store` (a daemon cannot answer prompts, and it would bypass the opt-in)
refuses an `sc+git::` store in v1.
`sc serve --store sc+git::<url>` serves a carrier, and the opt-in moves to the operator
(`--allow-private-branches`): without it `BucketTransport` refuses objects of the
sealed-branch kinds in `PutPack` and `PutObject` before anything is uploaded or staged, and
an `UpdateRef` to a manifest, so no private-branch ciphertext reaches the Git host through
the server (the daemon needs non-interactive git auth, and keeps its git cache under the
serve home's `.sc/tmp/`).
sc-forge accepts exactly the `refs/sc/` prefix (ADR-0058 addendum). The dependency rule
is amended: `objio` may spawn the system `git`, and only `gitio::bridge` and objio's
git backend spawn it; `gix` stays in `gitio`. Proven by `crates/cli/tests/git_bucket.rs`,
Expand Down
34 changes: 31 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ the code, those win.
| P37c | Merge, cherry-pick and rebase on a partial clone: a tree-recursive engine (`merge_trees`) carries a subtree by id when the sides agree and no protection rule applies, reads only where they differ or a rule intersects, and on a lazy, online clone fills what it asks for by id outside the store lock (a non-lazy clone gets the gap error with `sc backfill <path>`); a carried protected subtree keeps the complete input wrap union; a conflict inside the view is marked and resolved like on a full clone (`sc conflicts`/`sc resolve`/`merge --abort`/`sc commit`; the out-of-view part is grafted from the decided root), while a conflict outside the view, or a result that must be sealed outside it, is refused before any ref, merge-state or working-tree change (fetched objects stay). A subtree identical on all three sides is carried even under a protection rule. Cherry-pick and rebase use the same engine (`replay_commit`): the fold's accumulator may hold absent subtrees, the view is filled before the final write, and a stopped pick or rebase keeps its carried subtrees in the decided root. Mode-only edits now merge and a file/directory collision fails before any write, on full clones too. | [0049](docs/adr/0049-merge-on-partial-clones.md) |
| P37d | Agent sessions on a partial clone: `sc ws fork`/`run`/`harvest`/`abandon` and `sc work` work; a workspace or agent sees the host's sparse view only (`sc work` no longer gets a complete tree there), the view closure is filled (lazy, online) or refused with the gap error before anything is written, a workspace commit grafts the out-of-view subtrees from the base without reading them, and harvest lands through the ADR-0049 merge (any conflict falls back to a `work-<i>` branch as on a full clone) | [0050](docs/adr/0050-sessions-on-partial-clones.md) |
| P38 | Gitea-compatible forge over bare Git and smart HTTP: recorded API subset, scoped auth, protected merges, maintenance and container operations; 24 recorded scenarios and six first-consumer acceptance scenarios pass. Hosted repos have none of the `.sc/` features; SSH and Actions remain outside v1. | [0058](docs/adr/0058-sc-forge-gitea-compatible-git-forge.md) |
| P39 | `sc+git::<git-url>` remotes: the bucket WAL stored as one parentless commit per key at `refs/sc/<key>` in any Git repository (system `git` via `objio::GitBucket`, lease CAS, no `+` refspec), so private branches, protected paths and secrets travel through a Git host (local bare repo, sc-forge; GitHub untested); private-branch push needs per-remote `--allow-private-branches`; `remote compact` works on it (first-sighting age record, leased deletes; ADR-0057 addendum) while `serve --store` refuses it in v1; sc-forge accepts `refs/sc/` | [0060](docs/adr/0060-git-carried-bucket-remotes.md) |
| P39 | `sc+git::<git-url>` remotes: the bucket WAL stored as one parentless commit per key at `refs/sc/<key>` in any Git repository (system `git` via `objio::GitBucket`, lease CAS, no `+` refspec), so private branches, protected paths and secrets travel through a Git host (local bare repo, sc-forge; GitHub untested); private-branch push needs per-remote `--allow-private-branches`; `remote compact` works on it (first-sighting age record, leased deletes; ADR-0057 addendum); `sc serve --store sc+git::` serves it and refuses private-branch objects unless the operator passes `--allow-private-branches`, and every fetch runs under a download watchdog; sc-forge accepts `refs/sc/` | [0060](docs/adr/0060-git-carried-bucket-remotes.md) |

## Standing boundaries & gotchas

Expand Down Expand Up @@ -228,7 +228,7 @@ transport-adjacent. The rest, imperatively:
error on it and it never reaches the P18 bridge. Pushing a private branch needs
the per-remote opt-in `--allow-private-branches` (`remote add`/`clone`;
enforced in `repo`, refuses before any upload); protected paths and secrets
need none. `sc serve --store` refuses it (deferred: no prompts in a daemon).
need none. `sc serve --store` serves it behind a server-side gate (next bullet).
`sc remote compact` works on it with its own age source (a Git host has no
trustworthy object age): a first-sighting record at
`refs/sc/compact/sightings` (`<secs> <oid> <key>`, generation-aware, strict
Expand All @@ -245,7 +245,35 @@ transport-adjacent. The rest, imperatively:
and forks may keep objects by id). Public content is plaintext, and private
branch names, recipient ids and sizes are readable by every reader of the
repository. A killed process can leave `sc-git-bucket-<pid>-<random>` in the
temp dir. Not tested against real GitHub.
temp dir (`sc serve` keeps it under the serve home's `.sc/tmp/`, locks
`<cache>/.owner.lock` for the bucket's life, and reaps at startup the caches
whose lock is free, unix). Not tested against real GitHub.
- **`sc serve --store sc+git::<url>` (ADR-0060) refuses private-branch objects
unless the operator passes `--allow-private-branches`** (refused on any other
store). The gate is `BucketTransport.refuse_sealed`, set by `open_for_serve`:
`put_pack` stops at the first object of a sealed-branch kind (`Sealed`,
`Manifest`, `transport::is_sealed_branch_object`) before `upload_pack`,
`put_object` refuses before staging, and `update_ref` refuses a manifest tip
even when it already sits on the chain; the error is `InvalidArgument`
(`EC_OTHER`, no wire change) and a client cannot lift it. The daemon needs
non-interactive git auth. A mutable-key read is one `git fetch` of exactly
`refs/sc/<key>` under protocol v2 (the host filters the advertisement; negative
refspecs stop git's abbreviation rules from letting a branch or tag stand in
for the key). Every fetch runs under a watchdog (`fetch.unpackLimit=1`, the
growth of `objects/pack` polled from 2 ms backing off to 50 ms): killed over the
budget (the key's value cap plus 0.1% and 1 MiB, 2 GiB for a family; a family
over its budget is discarded and read one fetch per key, remembered per
instance, so a big store can still compact) or after 120 s with no progress,
what it added removed. The bound is disk per fetch, not
RAM inside `index-pack`, not the total cache, and not the refs a family fetch
writes. Only serve mode on unix bounds disk against a hostile HTTP host: it runs
git in its own process group and kills the group through the system `kill`,
but only while git is unreaped (after the reap the group id may have been
reused, so it is never signalled; `may_signal_group`). Without a `kill` it
warns once and kills git alone. The CLI and Windows kill git alone, and the
helpers that write the pack may keep writing; so may any helper that outlives
git's exit, in any mode, until it exits. Only fetches are watched; `ls-remote`
and push are not.
- **`DirBucket` (`sc+wal://<dir>`) is a single-host, single-owner backend (#264).**
Its CAS takes an OS advisory file lock (`File::lock`: `flock` on unix,
`LockFileEx` on Windows) on `<root>/.cas-lock`, a file that is never removed and
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,9 @@ branches, protected paths and secrets travel through a Git host that a plain
`git clone` never sees into (ADR-0060). Pushing a private branch there needs
`--allow-private-branches` (its name and recipient ids become readable by
everyone who can read the repository). `sc remote compact` works on it (run it
from one host; the first run only records when it first saw each object);
`serve --store` is not supported on it yet. Proven by
from one host; the first run only records when it first saw each object).
`sc serve --store sc+git::<git-url>` serves it and refuses private branches
unless the operator passes `--allow-private-branches`. Proven by
`demo/run_git_carrier_demo.sh`; not tested against GitHub itself.

## How Phase 1 works
Expand Down
35 changes: 21 additions & 14 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -704,10 +704,11 @@ across every phase.
`refs/sc/*`, and the same repository can also mirror public branches through a
`--git` remote. A private-branch push needs the per-remote opt-in
`--allow-private-branches`; `sc remote compact` works on `sc+git::` (a
first-sighting age record, leased deletes; ADR-0057 addendum) while
`sc serve --store` refuses it in v1; sc-forge accepts the `refs/sc/` prefix. No
format, wire or crypto change. Proven by `demo/run_git_carrier_demo.sh`.
(ADR-0060.)
first-sighting age record, leased deletes; ADR-0057 addendum); `sc serve --store
sc+git::` serves it behind a server-side private-branch gate (the operator's
`--allow-private-branches`) and every fetch runs under a download watchdog;
sc-forge accepts the `refs/sc/` prefix. No format, wire or crypto change. Proven
by `demo/run_git_carrier_demo.sh`. (ADR-0060.)

## Active

Expand Down Expand Up @@ -1028,16 +1029,22 @@ P28 review follow-ons, the port de-flake, the clippy allow-list) are recorded in
come from the idx; `list_meta` itself still downloads every listed value); (7)
in part: compaction folds small packs and deletes what nothing names, which
bounds the ref growth of at least three per push and the log/checkpoint
entries a cold read fetches, but between runs every `ls-remote` and push still
downloads the whole ref advertisement (the host does not filter it by prefix
server-side, a packet trace showed the full list). Still deferred: (2)
`sc serve --store sc+git::...` (a daemon cannot answer ssh or credential
prompts, and it would sit between clients and the private-branch opt-in); (4)
validation against real GitHub (per-ref and per-push limits, rulesets, hosts
that refuse unknown ref namespaces), untested so far; (9) a download budget for
fetches into the temp cache (`git fetch` has no input-size limit, so an
oversized value is written before the read caps refuse it). Declined, with
reasons: (3) rollback and rewind of `refs/sc/manifest` by anyone with push
entries a cold read fetches, and the manifest read is one `git fetch` that a
protocol v2 host answers with that ref alone, but between runs `ls-remote` and
push still download the whole ref advertisement; (2) `sc serve --store
sc+git::...`, which refuses private-branch objects unless the operator passes
`--allow-private-branches` (the daemon needs non-interactive git auth); (9) a
download budget: every fetch into the cache runs under a watchdog that kills
git over its budget (disk bytes per fetch; not RAM inside `index-pack`, not the
total cache). Still deferred: (4) validation against real GitHub (per-ref and
per-push limits, rulesets, hosts that refuse unknown ref namespaces), untested
so far; (10) a group-leading supervisor for the serve-mode watchdog: run git
under a std-only `sh` leader (`sh -c 'git "$@"; echo $? > <status file>; read
_' sh <args>`, stdin piped from sc) so the process group id stays pinned after
git exits, and the watchdog can group-kill a helper that outlives git (the
post-exit gap ADR-0060 documents: a smart-HTTP v0/v1 helper chain, an ssh
`ControlPersist` master) before it closes the leader's stdin and reaps it.
Declined, with reasons: (3) rollback and rewind of `refs/sc/manifest` by anyone with push
access is the cross-cutting signed-ref effort already tracked under P34
follow-on (6) and ADR-0059 follow-on (2), now reachable by more people; a
client-side "highest manifest seq seen" floor was reviewed and judged a false
Expand Down
Loading
Loading