Skip to content

perf: cut an sc+git:: push from 19 git round trips to 10 - #373

Merged
tonibergholm-codento merged 5 commits into
mainfrom
feat/sc-git-round-trips
Nov 22, 2026
Merged

tonibergholm-codento merged 5 commits into
mainfrom
feat/sc-git-round-trips

Conversation

@tonibergholm-codento

Copy link
Copy Markdown
Collaborator

A push to an sc+git:: remote was slow on GitHub: a first small push took about a minute, at one HTTPS round trip per git call. Counted with a logging SC_GIT against a local bare repo, a first push into an empty carrier made 19 network git calls and a second push made 11. Ten of the first push's calls were five identical "is there a manifest?" probes, each a fetch plus an ls-remote.

This PR cuts that to 10 and 8, without loosening any CAS or compaction check:

  • One push for the pack and its idx. New Bucket::put_new_batch (the default loops put_new). GitBucket sends one porcelain push per bounded chunk: at most 64 MiB of values, with the same item and command-line bounds as delete_leased. Each ref gets its own create-only --force-with-lease, never a + refspec or --atomic. A ref not reported landed is classified by one exact-name ls-remote, as a single put is today.
  • One presence check before the manifest CAS. New Bucket::exists_batch checks every pending pack and idx in one ls-remote. It never answers from the cache.
  • No read-back after a won CAS. The transport builds its view from the manifest it just wrote, at the tag the CAS returned. It re-reads when that fails or when an S3-compatible host returns no ETag.
  • Only has_object trusts a recent "empty". It answers from a 2 s memory of an empty remote. list_refs, head_branch, update_ref, get_pack and compaction still read the manifest every time.

A batch that fails can leave any subset of its keys created. Those are orphans that sc remote compact sweeps, as a failed pair of single pushes leaves today.

ROADMAP records (11) as built. It also declines the decoy-branch issue: a refs/heads/refs/sc/K shadows refs/sc/K in git's push destination matching, and git cannot turn that off. Only someone who can already delete refs/sc/* can plant one, and reads compare exact ref names. The old-pack download in check_ref_move is noted as a separate bandwidth item.

Proof:

  • The call counts come from the logging shim, run against main and this branch, and a clone afterwards reads both commits.
  • New tests pin the exact git call sequence of a first and second push. Other new tests cover the batch outcomes (landed, already present, refused by a hook, a lost acknowledgement, a retry), the chunk bounds, the memo staying inside has_object, and the view after a CAS.
  • Each new rule was broken on purpose, about 30 times in all, and a test failed each time.
  • fmt, clippy -D warnings, cargo test --workspace (1954 passed) and demo/run_git_carrier_demo.sh pass.

The plan was reviewed by Mythos (security), Astra and code-reviewer, and the diff by code-reviewer.

Reviewed with sol-loop: 2 rounds with gpt-6.2-sol on high. Sol verdict: approved

Created with Claude Opus 5.5 in Claude Code (T3 Code), with a Claude Sonnet 5.5 subagent writing most of the implementation.

… read-back after a push (sc+git::)

A push to a sc+git:: carrier paid one git round trip per key. This cuts the
network calls of a first push from 19 to 10 and of a second from 11 to 8.

- Bucket gains put_new_batch and exists_batch (defaults loop put_new and
  exists; both in the contract suite). GitBucket overrides them: one
  `git push --porcelain` per chunk (64 MiB of values, the leased delete's item
  and command-line bounds), a create-only lease per ref, never `+`, never
  `--atomic`; the per-ref porcelain lines are read whatever git's exit status.
  Keys not reported landed are classified together by one ls-remote of their
  exact refs; if that fails the push refusal is returned, never false.
  exists_batch is one ls-remote per chunk, exact refname compare, never
  answered from the local state.
- BucketTransport uploads pack and idx with one put_new_batch and checks every
  pending pack and idx with one exists_batch before the manifest CAS.
- After a won manifest CAS the view is built from the manifest just written
  (build_view, split out of refresh_once) instead of reading it back; a build
  failure falls back to a plain refresh.
- has_object (and only it) answers "absent" for 2 s after a manifest read that
  found the remote empty, instead of probing once per object.
Tighten the batch tests after breaking each rule on purpose (the per-ref
porcelain lines read on a non-zero exit, check_key before any git call, what
landed being cached), and record the measured counts in ROADMAP and ADR-0060.
ROADMAP also declines the decoy-branch push failure and notes the old-pack
download in ref_guard::check_ref_move as out of scope. ADR-0046 notes the 2 s
empty-remote memo of has_object.
- A batch push names every rejected ref, and the first refusal of a batch
  is the one reported.
- After a won CAS with an empty tag (an S3-compatible host that sent no
  ETag), re-read instead of building a view on it.
- The memo tests no longer depend on setup finishing inside the 2 s window.
- ADR-0060 says which backends pay the per-object manifest probe.
@coderabbitai

coderabbitai Bot commented Nov 22, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: dcf808f8-326e-4610-8301-cb2fbb5e644a
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-22T09:10:53.100711Z 1d5ffe0 New commits
🔒 Security Review ✅ Completed 2026-11-22T09:13:24.778352Z 1d5ffe0 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@tonibergholm-codento
tonibergholm-codento merged commit 1e69c05 into main Nov 22, 2026
11 checks passed
@tonibergholm-codento
tonibergholm-codento deleted the feat/sc-git-round-trips branch November 22, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants