Prototype: git fast-import based git export - #60
Open
marcarl wants to merge 2 commits into
Open
Conversation
Extract plan_init_commit() and plan_temporal_commits() so the legal-content logic (temporal filtering, title processing, commit message wording) is no longer entangled with git side effects. Behavior is unchanged; this is what lets a fast-import based exporter reuse the same logic instead of forking it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
batch_export_to_git.py creates every commit via git add/git commit subprocess calls, once per document and once per date per document. Each call rewrites the index and touches the working tree, which dominates wall-clock time at ~50 000 författningar (see ADR-003). fast_import_writer.py streams commits (backdated author/committer dates, inline file blobs) straight into a single `git fast-import` process instead - no working tree, no index, no per-commit subprocess. fast_import_export.py orchestrates it: build every commit's plan up front by reusing plan_init_commit/plan_temporal_commits, sort all of them globally by date across every document, then write them in one fast-import pass and push once. As a side effect, the resulting branch's parent chain now matches real chronological order instead of being grouped document-by-document. Prototype only - not wired into batch_export_to_git.py or run against real SFS data / the se-lex/sfs remote yet. Validated against scratch repos in test/test_fast_import_writer.py and test/test_fast_import_export.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a fast, git-fast-import based Git export pipeline and refactors existing commit generation to share pure “commit plan” logic across exporters.
Changes:
- Introduces
FastImportWriterand a new fast-import export orchestrator that globally date-sorts commits before writing. - Refactors
generate_commits.pyto exposeplan_init_commit/plan_temporal_commits(plus dataclasses) for reuse across exporters. - Adds unit/integration tests for the fast-import writer and event planning.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/test_fast_import_writer.py | Integration tests that validate git fast-import round-trips into a real scratch repo. |
| test/test_fast_import_export.py | Unit tests for commit-event planning without touching git/network. |
| exporters/git/generate_commits.py | Refactors commit creation into reusable plan functions + dataclasses; reuses plans in temporal export. |
| exporters/git/fast_import_writer.py | New fast-import stream writer + date conversion utilities. |
| exporters/git/fast_import_export.py | New end-to-end exporter that builds + globally sorts commit events, then streams to git fast-import. |
| exporters/git/init.py | Exposes new plan functions and dataclasses from the git exporters package. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+66
to
+73
| def _quote_path(path: str) -> str: | ||
| """Quote a path per fast-import's C-style path quoting. | ||
|
|
||
| Always quoting (rather than only when a special character is present) | ||
| is valid per the grammar and avoids having to special-case spaces, etc. | ||
| """ | ||
| escaped = path.replace("\\", "\\\\").replace('"', '\\"') | ||
| return f'"{escaped}"' |
Comment on lines
+93
to
+97
| process = subprocess.Popen( | ||
| ["git", "fast-import", "--stats" if verbose else "--quiet"], | ||
| cwd=repo_dir, | ||
| stdin=subprocess.PIPE, | ||
| ) |
Comment on lines
+162
to
+168
| def close(self, timeout: int = 3600) -> None: | ||
| """Flush and close the stream, waiting for `git fast-import` to finish.""" | ||
| self._stream.close() | ||
| if self._process is not None: | ||
| returncode = self._process.wait(timeout=timeout) | ||
| if returncode != 0: | ||
| raise RuntimeError(f"git fast-import misslyckades med exit code {returncode}") |
Comment on lines
+145
to
+147
| year_dir = md_file.parent.name | ||
| filename = md_file.name.replace("-markers", "") | ||
| relative_path = Path(year_dir) / filename |
Comment on lines
+206
to
+216
| writer = FastImportWriter.to_repo(repo_dir, verbose=verbose) | ||
| for i, event in enumerate(events): | ||
| raw_date = to_raw_git_date(event.date) | ||
| writer.commit( | ||
| branch=branch_name, | ||
| message=event.message, | ||
| author_date=raw_date, | ||
| committer_date=raw_date, | ||
| changes=[FileChange.write(event.path, event.content)], | ||
| from_ref=from_ref if i == 0 else None, | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prototype of an alternative to
batch_export_to_git.py's git export pipeline, usinggit fast-importinstead of per-document/per-changegit add+git commitsubprocess calls (see ADR-003 for background on the backdated-commit approach; SFS has ~50 000 författningar, each potentially with many amendment commits).exporters/git/fast_import_writer.py— streams commits directly into agit fast-importprocess (inline blobs, backdated author/committer dates, no working tree or index touched).exporters/git/fast_import_export.py— orchestrator: builds every commit's plan up front (initial + all temporal changes, across every document), sorts them globally by date, writes them in a single fast-import pass, pushes once.exporters/git/generate_commits.pyrefactor — extractedplan_init_commit()/plan_temporal_commits()so the actual legal-content logic (temporal filtering, title processing, commit message wording) is shared between the existing subprocess-based exporter and this new one, instead of duplicated. No behavior change.Side effect: because commits are globally date-sorted before being written, the branch's parent chain now matches real chronological order across documents, rather than being grouped document-by-document with backdated timestamps layered on top (as today's exporter produces).
Not done: not wired into
batch_export_to_git.py's CLI, not run against real SFS data or these-lex/sfsremote, no throughput benchmark yet vs. the current exporter. ADR-003 intentionally left untouched until this is validated for real and a decision is made to adopt it.Test plan
python -m pytest test/ -v— all 386 tests pass, including newtest/test_fast_import_writer.py(round-trips commits through a realgit fast-importagainst scratch repos: dates/timezones, message/content, parent chaining, deletes, unicode, rooting a new branch onto an existing tip) andtest/test_fast_import_export.py(JSON/marker-markdown → CommitEvent routing)ruff check/ruff formatclean on all touched/new filesbatch_export_to_git.pyon realistic volume🤖 Generated with Claude Code