Skip to content

feat: back up GitHub Issues with metadata archives and offline browsing - #23

Draft
ImSingee wants to merge 3 commits into
masterfrom
feat/github-issues-backup
Draft

ImSingee wants to merge 3 commits into
masterfrom
feat/github-issues-backup

Conversation

@ImSingee

@ImSingee ImSingee commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds opt-in GitHub Issues backups to the existing source/task/cron workflow, with end-to-end Web, RPC, and CLI access.

  • Separate local-only metadata Git archives at metadata/<source-id>/<GitHub-repository-id>.git, with canonical raw JSON, inert readable Markdown, checksummed manifests, immutable content-change snapshots, and verified ZIP exports.
  • Paginated issues/comments/timeline/labels/milestones collection; PR filtering; independent issue/comment watermarks; full reconciliation; interrupted-run checkpoints; atomic Git publication followed by a rebuildable SQLite index.
  • Failed pages, 403/404/410, or repository identity changes retain the last successful backup and do not advance successful watermarks. A completed full crawl marks missing issues unavailable without claiming deletion.
  • Source opt-in, source/sync-all/cron child tasks, interrupted-job restart, authenticated ConnectRPC/export routes, and Issues browsing with search, filters, history, manual sync/full reconcile, and export.
  • CLI: issues sync, issues export, issues verify, and issues reindex. See the usage, format, security, recovery, and scope guide.

Scope boundaries

Attachments are links only; no attachment bytes are backed up or rendered remotely. This does not back up PR reviews, Discussions, Projects, every reaction member, or edits/events GitHub no longer exposes. There is no automatic metadata remote push or GitHub restore/import. ZIP exports contain one selected snapshot; preserve the bare metadata repository for complete observed history.

Existing source tokens and API authentication are reused. No new credentials or write permissions are required. SQLite Issues indexes are rebuildable from Git once the existing repository inventory is available. The source-code archive layout is unchanged.

Validation

Passed locally:

  • go test ./... and go vet ./...
  • Race tests for githubapi, issuearchive, issuesync, issuetask, issueservice, and source-task integration
  • pnpm lint, pnpm check:types, all 16 frontend unit tests, and pnpm build
  • Buf lint/code generation and sequential Drizzle/schema/sqlc regeneration (no second schema migration)
  • Real CLI verify/reindex/export against a seeded archive; ZIP integrity check
  • Archive cross-compilation for Windows/amd64 and macOS/arm64 (not native execution)

Real production frontend + Go/SQLite/Git acceptance fixture, with only upstream GitHub replaced by a local synthetic server:

  • Current issue list, pagination, search, closed filter, empty results, description/comments/timeline, historical detail, historical ZIP, exact large integer IDs
  • No img/script/iframe execution from hostile archived content
  • Real queued sync and 404 failure with retained commit, 32 issues, 32 comments, history and export
  • Source rename preserves the enabled Issues setting

GitHub Actions passed on 266a91e for both pull_request and push: lint, type checking, Go tests, all 28 frontend tests (16 unit + 12 Chromium browser), and the production build. The API client also passed 500 shuffled test repetitions and 10 shuffled race repetitions after isolating test-only cooldown state.

Local Vitest Browser Mode remained unavailable because of this environment's Chromium installation/launch restrictions; the browser suite was verified in CI. The live source-toggle-off step was not executed after the environment denied that configuration change; automated browser/backend tests cover explicit false and disabling queued work. UI acceptance screenshots are retained locally and were not published.

Review notes

  • Git remains authoritative; publishing a snapshot and replacing the SQLite index are deliberately separate, recoverable steps.
  • Reconciliation runs every seven days when a sync runs, or on demand. Long rate-limit waits fail visibly and retry on the next manual/scheduled run.
  • The current history picker shows the latest 100 commits. Very large snapshots are bounded and materialized in memory; streaming and attachment capture are follow-up work.
  • Buf plugins are pinned to the repository's existing generated versions; dependency versions are not upgraded.

Draft for review; do not merge yet.

Copy link
Copy Markdown
Owner Author

Issue 同步 Acceptance 补充

测试提交:266a91e11b78884270f13faaf536c7f5e8c57fd9。

Acceptance flows 与原始证据

本轮是 partial:7 个 flow、9 项完整检查因 UI 阻塞未通过,Acceptance 显示 uncertain(0 passed / 9 uncertain),未宣称 UI 验收通过。在独立配置、SQLite/Git 目录与合成 GitHub 上游中,通过真实生产 HTTP 处理器和任务队列验证了这些后端路径:

  • 旧配置缺省关闭,启用保存不自动同步。
  • 32 条 Issue / 32 条评论的分页同步、PR 排除、大整数保真、列表搜索/筛选与归档内容。
  • since 增量请求,以及 Issue 更新时间不变、仅评论改变时仍同步;无变化快照数 3→3,旧快照 ZIP 保留旧内容。
  • 实际 404 失败不替换内容/commit、不推进两个成功游标;恢复后成功。
  • 完整对账将缺失项标为 unavailable,保留历史查看/导出。
  • 单来源、Sync All、19:39:00 UTC 实际 cron 均产生成功的 Issue 子任务,代码与元数据归档独立。
  • 关闭后已 queued 的任务因 disabled 被拒绝,上游请求数 90→90;新排队请求被拒绝,历史和旧 ZIP 仍可读。

剩余阻塞:云浏览器无法连接隔离 shell 服务,且没有受支持的服务暴露路径。因此本轮未完成 UI 开关/按钮、分页搜索交互、历史选择与下载、安全 HTML 渲染/外部资源请求检查,也未取得规定的截图或视频。API 证据不替代这些 UI 检查。

未改产品代码或生产来源;未写入真实 GitHub Issues。reaction 参与者明细不在本轮范围。

发布后读回:9/9 项有文本证据,共14条附件/内联证据;产品截图0/9,规定视频0/2。该上传状态不代表验收完成。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant