perf(sessions): skip unchanged host bookkeeping writes on idle passes - #3079
Open
devin-ai-integration[bot] wants to merge 1 commit into
Open
devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
The daemon reruns the history pass every idle minute. Each pass rewrote every provider's host-coverage row and the Kimi and Pi discovery frontiers with a bumped mtime even when nothing changed: nine commits per idle project pass, plus the same on the user store, each adding WAL frames that readers and checkpoints reread. revise_host_record now writes such a record only when its byte_offset or file_id changes; mtime stays the revision that lets a changed record move backwards past the monotonic cursor guard. The Codex callers' own unchanged-coverage guard is gone, since the shared write owns it. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
Contributor
Author
There was a problem hiding this comment.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
host-coverage://<provider>/v1row and the Kimi and Pi discovery frontiers, even when nothing had changed. That was 9 WAL commits per idle project pass, plus about the same onuser-sessions.db. The daemon runs this pass every minute (HISTORY_IDLE_RECHECK_INTERVAL).revise_host_record(runtime/source.rs) writes a record only when itsbyte_offsetorfile_idchanges. Those are the only fields readers check.mtimeis still bumped on a real change, which lets a record movebyte_offsetbackwards past the monotonic cursor guard.Motivation
Refs #2521, Refs #2658.
I re-measured after #2777 and #2948. The capture path already reads only the objects a commit names. In
load_current_mutations,SourceObjectCoverageV1::Completeis reached only for complete-snapshot-absence sources. Next I looked at an idle daemon with nothing being ingested, using an isolated profile and project. Temporary SQL probes, since removed, plusparse_offsetssnapshots showed what changed each minute:host-coverage://{claude,cursor,kimi,opencode,pi}/v1andhost-frontier://{kimi,pi}/discovery/v1were rewritten on both stores with onlymtimeincremented. Every such commit adds WAL frames that readers and checkpoints then reread for as long as the daemon is idle. #2658 lists this idle traffic as the thing that makes whole-daemon bytes per message unmeasurable.Changes
crates/tracedecay-sessions/src/runtime/source.rs:persist_host_provider_coveragenow goes through the newrevise_host_record, which skips unchanged records.hosts/kimi.rs,hosts/pi.rs: the discovery frontier writes userevise_host_record. Reaching the end on an already-reset frontier no longer writes.ingest/project_provider.rs,ingest/user_provider.rs: removed the Codex callers'stored_coverage != Some(coverage)guard. The shared write now owns that check for every provider.crates/tracedecay-session-runtime/tests/session_store_read_cost.rs: newidle_history_pass_commits_nothing. It runs the production project ingest pass twice over an empty host home on a registered store. It asserts that the first pass commits and the second appends no WAL commit frames.Test plan
bash scripts/require-exact-test.sh cargo test -p tracedecay-session-runtime --features test-helpers --test session_store_read_cost -- idle_history_pass_commits_nothing --exact: 1 passed. Without thetracedecay-sessionschange it fails withleft: Some(9) right: Some(0).cargo test -p tracedecay-sessions --lib -- runtime::hosts runtime::ingest runtime::source: 353 passed.cargo clippy -p tracedecay-sessions -p tracedecay-session-runtime --all-targets --features tracedecay-session-runtime/test-helpers -- -D warnings: clean.cargo fmt --all -- --check: clean.HOME/TRACEDECAY_DATA_DIRand an enrolled project. Before the change, coverage and frontier rows were rewritten every idle minute. After it,parse_offsetsand both stores' WALmxFramewere byte-identical across 130 s (two idle passes).Not covered: operator-scale read volume (the 2 TB / 3 h in #2521) is not re-measured here. The per-message store-growth target in #2658 is unchanged by this PR.
Checklist
CHANGELOG.md: not updated; release-please generates it from commits.envfiles includedLink to Devin session: https://app.devin.ai/sessions/868821e98d0743dc83648344702f3479
Open in Devin Desktop: https://app.devin.ai/desktop/session/868821e98d0743dc83648344702f3479?variant=devin
Requested by: @ScriptedAlchemy