Repository navigation
fix(syncer,daemon): avoid full-volume ignore and cache work on targeted local syncs - #336
Conversation
…sient scan retries
runbear-bot
left a comment
runbear-bot
left a comment
There was a problem hiding this comment.
Bobb's bot(Totoro) Review
Conclusion: not-ready
Blocker
internal/syncer/syncer.go:1155의 새RetryPaths기록은walkFolderPaths콜백 내부의d.Info/Readlink/blob 읽기 실패만 다룹니다. 하지만internal/syncer/walk.go:90-92는filepath.WalkDir의 디렉터리 순회 오류를return nil로 삼켜 콜백을 호출하지 않습니다.ScanEvent{Directory:true}는 같은 사이클에deleteScopes를 설정하므로, 일시적인 디렉터리 열거 실패 시 해당 디렉터리의 캐시 자식들이 unseen으로 남아 삭제 op로 기록될 수 있고 peer로 전파됩니다. 디렉터리 순회 실패를 retry 대상으로 보존하고 삭제 pass를 막아야 합니다. 이 finding은 변경된 retry 경로에 inline으로 남겼습니다.
Verification
git diff --check: passed- 부모 source probe: passed — traversal error swallow, directory delete scope, callback-only retry recording을 현재 소스에서 확인
- Go test/gofmt: not run (
gotoolchain unavailable;0 / 0 / 0) - 독립 OMX draft도 같은 P1 blocker를 확인했습니다.
Reviewed head: 93ebca8b094d4cc897d0c77fb47aac9fa4ecd244.
| seen := make(map[string]bool, len(cache)) | ||
| retrySeen := map[string]bool{} | ||
| var retryPaths []string | ||
| recordRetry := func(rel string) { |
There was a problem hiding this comment.
디렉터리 순회 오류는 이 callback에 도달하지 않아 recordRetry가 호출되지 않습니다. ScanEvent{Directory:true}가 만든 deleteScopes와 결합되면 일시적인 WalkDir 실패가 캐시 자식들의 삭제 op로 오인될 수 있습니다. 순회 오류를 retry로 전파하고 해당 범위의 삭제 pass를 막아야 합니다.
There was a problem hiding this comment.
Implemented in 52b18dc. Non-ENOENT WalkDir errors now report through walkFolderPathsWithErr to scan, which records the failed path in RetryPaths and blocks the delete pass for that path and its cached descendants (or skips the delete pass altogether when the mount root itself cannot be read). When the daemon re-queues RetryPaths, it sets ScanEvent.Directory from os.Stat so recovered directories are re-walked as directories. Verified with go test -race -v ./internal/syncer -run TestTargetedScanRecordsRetryPathsOnTransientReadFailure and go test -race -v ./internal/daemon -run TestWorkerRequeuesRetryPathsFromTargetedScan.
runbear-bot
left a comment
runbear-bot
left a comment
There was a problem hiding this comment.
Bobb's bot(Totoro) Review
Conclusion: ready-with-notes
- 기존 디렉터리 순회 오류 finding은 현재 소스에서 오류 범위의 삭제 방지와 retry 경로가 추가된 것을 확인했습니다. 런타임 재현은 하지 못했습니다.
- 참고: ignore 규칙을 추가할 때 이미 등록된 watcher는 즉시 제거되지 않아 해당 트리의 이벤트가 계속 발생할 수 있습니다. escaped
/규칙의 anchoring 동작도 기존과 달라 보이므로 호환성 테스트를 권합니다. 둘 다 실행 재현은 못 했습니다. git diff --check: 통과. Go 도구가 없어 syncer/daemon 테스트는 실행 불가 (0 / 0 / 0); PR 작성자의 테스트 결과는 별도 보고이며 독립 검증하지 않았습니다.
Reviewed head: 52b18dcc8ed4b26007f6ff148c56166674de4d8a.
TL;DR
.bdriveignorerules) now upload within one scan interval instead of stalling for minutes..bdriveignorematching avoids backtracking across every byte offset of every path, andbackfillOpsandmaterializeskip ignore checks for files that already match the local cache..git,.bdrive, nested mounts, and ignored directories, skips loading the state cache during echo suppression when no pre-fence events are pending, and retries targeted scan paths once after a transient read failure.Summary
User-requested fix after v0.18.2 (
844e708dc3a5a85843f8da4cf9d4949bd931ff54). Even with targeted watcher scans enabled, local file edits in large workspaces appeared stuck because post-scan stages still ran full-volume ignore regex evaluation and repeated journal folds on every cycle, while the macOS watcher watched ignored trees and doubled open file descriptors duringENOENTrebuilds. This PR removes the per-cycle full-volume bottlenecks and hardens watcher pruning, echo suppression, and transient scan retries.Root cause
In
internal/syncer/ignore.go, unanchored.bdriveignorepatterns compiled with a leading(^|.*/)alternation, which prevented Go'sregexppackage from using a literal prefix or one-pass execution and forcedtryBacktrackacross every byte position of every path. Ininternal/syncer/syncer.go,backfillOpsandmaterializeevaluatedFilter.Skipon every target entry before checking whether the entry already matchedcache[rel], andcycleLockedparsed and replayed all device journals up to three times per cycle. Ininternal/daemon/daemon.go,suppressMaterializationEchoesloaded the full state cache JSON on every check after the first watcher event even when the pending queue was empty,startLocalFolderWatcherregistered watches on.git,.bdrive, nested mounts, and ignored directories, andrebuildWatcheropened a replacement watcher before closing the previous one onEMFILE/ENFILE.Design
compileininternal/syncer/ignore.gonow compiles unanchored rules with(?:^|/)and(?:/.*)?$, extracts literal segment and prefix prefilters at compile time, and evaluates rules in reverse order so the last matching rule returns immediately. Ininternal/syncer/syncer.go,backfillOpsandmaterializecheck whether a regular file's cached blob and mode already match the replayed target (and the path is not read-only) before callingFilter.Skip.cycleLockedscans.bdriveignoreops directly whenhadCachedIgnoreis true withoutmountJoining, and reuses the initialAllOps()slice across conflict detection and materialization wheneverpullandloadScopedid not modify peer journals on disk.In
internal/daemon/daemon.go,suppresssnapshots pre-fence candidate events underpendingMu, evaluateskeepoutside the lock with lazysync.Oncecache loading, and drops only entries whose sequence number did not advance while unlocked.startLocalFolderWatcherprunes reserved directories, nested mounts outsiderootList, andFilter.PruneDirdirectories both at startup and on runtime directory creation, re-walking the root when.bdriveignorechanges. IfrebuildWatcherhitsEMFILEorENFILE, it closes the old watcher first to release file descriptors before retrying. If a targetedScanHintwalk encounters a transient stat, readlink, or blob-hash error,Session.scanrecords the path inResult.RetryPathsandrunVolumeWorkerre-queues it once for the next scan interval.Changes
internal/syncer/ignore.go: add precomputed literal/segment matchers (pattern.match), replace(^|.*/)with(?:^|/), and evaluateignoredByin reverse order.internal/syncer/syncer.go: check cache convergence beforeFilter.SkipinbackfillOpsandmaterialize, reuse loaded journal ops acrosscycleLocked, and returnResult.RetryPathsfor transient targeted-scan read failures.internal/daemon/daemon.go: evaluate echo suppression outsidependingMuwith lazy cache loading, prune reserved/nested/ignored directories instartLocalFolderWatcher, close the old watcher before retrying anEMFILE/ENFILErebuild, and re-queueResult.RetryPathsonce.internal/syncer/targeted_safety_test.go,internal/daemon/watchpoll_test.go,internal/daemon/watch_safety_test.go: add unit and integration coverage for QA-01 through QA-09.architecture/cli-sync.md,CHANGELOG.md: update the sync engine class diagram andUnreleasedrelease notes.Architecture changes
In
architecture/cli-sync.md,Resultgains+RetryPaths []stringfor transient targeted-scan read failures, andlocalFolderWatcheradds a dependency onFilter(PruneDir on watched trees).flowchart LR Session["<b>Session</b><hr/>+Cycle(ctx) Result<br/>+CycleWithScanHint(ctx, hint) Result"] Result["<b>Result</b><hr/>+LocalOps +PulledOps<br/>+Conflicts +Adopted +Pruned +Materialized<br/>+Reverted []string<br/>✅ +RetryPaths []string<br/>+Inbound []store.InboundEvent<br/>+Pushed +Offline +OfflineErr<br/>+ReadOnly +NoAccess +AccessErr"] Filter["<b>Filter</b><hr/>+Skip(rel) bool<br/>+SkipUp(rel) bool<br/>+PruneDir(rel) bool<br/>+AcceptRules(text)"] volumeWorker["<b>volumeWorker</b><hr/>-string expectedRemote<br/>-context.CancelFunc cancel<br/>-chan wakeCh<br/>-chan done"] localFolderWatcher["<b>localFolderWatcher</b><hr/>-chan wake<br/>-chan err<br/>+close()<br/>+take(root) localWatchBatch<br/>+fence() sequence<br/>+suppress(root, fence, keep)"] Session --> Result Session --> Filter : SkipUp on scan, Skip on materialize volumeWorker --> Session : one Cycle per tick volumeWorker --> localFolderWatcher : local filesystem wake and safety fallback localFolderWatcher -.->|"✅ PruneDir on watched trees"| FilterQA checklist
.bdriveignorematcher agrees with the reference regex matcher across anchored, unanchored,dirOnly,*,**,?, escaped characters, and!negations, and completes 5,000 deep paths across 20 rules under 500msTestFilterFastEquivalenceAndLargeScalebackfillOpsandmaterializeskipFilter.Skipfor converged cache entries while preserving read-only drift reverts and respecting ignored read-only pathsTestMaterializeAndBackfillSkipFilterForConvergedCacheTestWorkerRemoteWakePreservesPendingLocalEdit,TestWorkerRemoteTimerPreservesQueuedLocalChangesTestWorkerSuppressesLateMaterializationEchoesENOENTrecovers without stopping the watcher, andEMFILE/ENFILEduring rebuild releases the old watcher before retryingTestWatcherRebuildsOnChildENOENT,TestWatcherRecoversFromAsyncChildENOENT.git,.bdrive, and.bdriveignore-pruned directories are skipped at startup and runtime creation, and removing a rule watches the newly un-ignored directoryTestLocalFolderWatcherPrunesIgnoredAndReservedDirssuppressMaterializationEchoesskipsLoadCachewhen no pre-fence events are pending, evaluateskeepoutsidependingMu, and preserves events merged whilekeeprunsTestSuppressMaterializationEchoesSkipsCacheLoadWhenNoPendingEventsScanHintpopulateResult.RetryPathsand are retried on the next scan interval without waiting for the 60s safety scanTestTargetedScanRecordsRetryPathsOnTransientReadFailure,TestWorkerRequeuesRetryPathsFromTargetedScango test -race ./internal/syncer ./internal/daemonVerification
Commands run:
go vet ./...: passedgo test -race -v ./internal/syncer -run 'TestFilterFastEquivalenceAndLargeScale|TestMaterializeAndBackfillSkipFilterForConvergedCache|TestTargetedScanRecordsRetryPathsOnTransientReadFailure': passed (2.49s)go test -race -v ./internal/daemon -run 'TestSuppressMaterializationEchoesSkipsCacheLoadWhenNoPendingEvents|TestLocalFolderWatcherPrunesIgnoredAndReservedDirs|TestWorkerRequeuesRetryPathsFromTargetedScan': passed (2.10s)go test -count=1 ./internal/syncer ./internal/daemon ./cmd/... ./internal/config ./internal/journal ./internal/remote ./internal/store ./internal/autostart ./internal/agenthooks ./internal/secrets ./internal/templates: passedOld vs new (same test code):
main(b7c6f2dc8651)93ebca8b094d)TestFilterFastEquivalenceAndLargeScale)fast filter workload took 4.33345875s-race)TestMaterializeAndBackfillSkipFilterForConvergedCache)nestedMiss=map[locked:true]TestLocalFolderWatcherPrunesIgnoredAndReservedDirs)pruned directory .git was added to watcherTestSuppressMaterializationEchoesSkipsCacheLoadWhenNoPendingEvents)suppress held pendingMu while evaluating keepTestTargetedScanRecordsRetryPathsOnTransientReadFailure)RetryPaths = [], want "retry.txt"TestWorkerRequeuesRetryPathsFromTargetedScan)timed out waiting for targeted retry uploaded the file; cycles=2TestLocalWatchNotExist).bdriveignorerules): single-passFilter.Skipover 66,492 paths dropped from 97.12s to 0.33s (and 0 calls on converged cache entries); live daemon propagation completed in 2.08s (existing-edit), 3.41s (new-file), 3.76s (new-directory), 3.97s (atomic-replace), and 3.88s (delete).Risks and upgrade impact
bdrivecommand or agent hook runs. On-disk journal and state cache formats are unchanged.Not run / not changed / not added
internal/autostartruntime tests (compile-only on darwin/linux).fsnotifystill useskqueueon macOS for non-ignored directories; switching to a cgo FSEvents backend was rejected to preserve pure-Go builds (CGO_ENABLED=0).