fix(datastore): make legacy Android bucket merge linear instead of quadratic

[TimeToBuildBob]

Fixes ActivityWatch/aw-android#261 (blank web view + "ActivityWatch isn't responding" on v0.14.0).

Root cause

migrate_test_bucket_names (merge path from #661, first wired into the app by aw-android#244, shipped in v0.14.0) moved disjoint legacy events with a correlated NOT EXISTS subquery per legacy event. The subquery has no lower bound on starttime, so for every legacy event SQLite rescans all earlier events in both buckets: O(n²).

Every user who ran an older release (which wrote to aw-watcher-android-test_<host>) and then v0.14.0b2 (which created aw-watcher-android_<host>) has both buckets, so the merge path runs for them on the first v0.14.0 start. With a couple of years of history it keeps the single datastore worker thread busy for hours. Everything else queues behind it:

• every web UI API call (GET /api/0/settings/ never answers) → blank white WebView
• every main-thread JNI datastore call (widget refresh, heartbeats, getBuckets) → ANR

Play vitals confirm the shape: the top ANR clusters are the main thread parked in aw_datastore::worker::Datastore::get_buckets → crossbeam_channel::recv, triggered from the widget refresh broadcast, WebWatcher, SystemJobService and the LOG_DATA alarm. And because a single overlapping cutover heartbeat leaves the merge "partial", the app re-runs the same migration on every start.

Evidence

Benchmark of the shipped SQL on synthetic disjoint data (desktop CPU, both buckets the same size):

events per bucket	merge time
5k	0.65 s
10k	2.6 s
20k	10.4 s
40k	41 s

Time quadruples per doubling. A phone with ~300k events is in the hours range, and phones are several times slower than this.

Reproduced on an Android 16 emulator with the v0.14.0 release APK and a 150k+15k event fixture: Migrating 'aw-watcher-android-test' bucket names… is logged at service start, the web UI's first request GET /api/0/settings/ is matched and never answered, /api/0/info times out, and the WebView stays white.

Fix

One SELECT id, starttime, endtime, bucketrow … ORDER BY starttime over both buckets, then a sweep that keeps still-open events in a min-heap keyed by endtime. Each overlapping pair is examined exactly once, so the pass is O(n log n). Movable ids are reassigned with batched UPDATE … WHERE id IN (…). Overlap semantics are unchanged (strict a.start < b.end AND b.start < a.end); all six existing merge tests pass untouched.

Tests

• test_migrate_test_bucket_names_merges_large_history_quickly: 100k legacy + 10k destination events plus one overlapping cutover event; asserts the merge finishes under a 30 s wall-clock bound (it takes well under a second in a debug build; the old query would take minutes) and that exactly the overlapping pair stays behind.
• test_migrate_test_bucket_names_zero_duration_event_at_shared_start_stays: pins the strict-overlap edge case the sweep has to special-case.

Note: cargo clippy --all-targets -D warnings with a local rustc 1.90 flags a pre-existing unnecessary_mut_passed in legacy_import.rs:200, unrelated to this change; not touched here to keep the hotfix minimal.

Follow-ups (aw-android side, separate)

• Bump the submodule in aw-android and cut v0.14.1.
• BackgroundService.onStartCommand ran twice on one launch in the repro, so the migration coroutine is queued twice; harmless once the migration is fast, but worth guarding.
• Main-thread JNI datastore calls (getBuckets from heartbeat paths and the widget provider) will ANR whenever the worker is busy for >5 s; they should move off the main thread.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the previously reported quadratic behavior for stacked histories has been addressed and its thread was resolved.

Summary

• Uses epoch-based overlap marking to keep heavily overlapping histories linear apart from heap operations.
• Preserves strict overlap behavior for zero-duration events.
• Adds regression tests for large disjoint histories, zero-duration boundaries, and heavily stacked histories.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Load both buckets ordered by start time] --> B[Drain events ending before current start]
    B --> C{Current event has positive duration?}
    C -->|Yes, open set nonempty| D[Mark current overlap and advance epoch]
    C -->|No| E[Check zero-duration event against earlier-start open events]
    D --> F[Push event with current epoch]
    E --> F
    F --> G{More events?}
    G -->|Yes| B
    G -->|No| H[Drain heap and finalize epoch-based overlap marks]
    H --> I[Select non-overlapping legacy IDs]
    I --> J[Move IDs to destination in batches]

Loading

_{Reviews (2) · Last reviewed commit: "fix(datastore): mark overlapping open ev..."}

[codecov]

Codecov Report

❌ Patch coverage is 95.34884% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.14%. Comparing base (656f3c9) to head (a3c39b3).
⚠️ Report is 105 commits behind head on master.

Files with missing lines	Patch %	Lines
aw-datastore/tests/datastore.rs	93.15%	5 Missing ⚠️
aw-datastore/src/datastore.rs	98.21%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #679      +/-   ##
==========================================
+ Coverage   70.81%   80.14%   +9.32%     
==========================================
  Files          51       67      +16     
  Lines        2916     6059    +3143     
==========================================
+ Hits         2065     4856    +2791     
- Misses        851     1203     +352     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

Verified on device (emulator) with the CI-built binaries-android x86_64 library from this PR (commit e34c809, debug profile) packaged into a v0.14.0 debug APK, against the same 150k + 15k event two-bucket fixture that wedges the shipped v0.14.0 indefinitely (still stuck after 5+ minutes when I stopped it):

07:58:59.828  aw_datastore::datastore: Migrating 'aw-watcher-android-test' bucket names to 'aw-watcher-android'
07:59:02.093  aw_datastore::datastore: Partially merged 'aw-watcher-android-test_…' into 'aw-watcher-android_…'; 2 overlapping event(s) remain in the legacy bucket
07:59:02.149  BackgroundService: Watcher bucket migration result: Migrated 0 'aw-watcher-android-test' bucket(s)

2.3 s for the merge in an unoptimized build, GET /api/0/info answered 200 within 5 s of launch, and the second (duplicate) migration the service queues on the same launch completed in 0.2 s. On the desktop the same fixture merges in 0.5 s (debug profile).

Two observations from the run, both pre-existing and out of scope here:

• BackgroundService.onStartCommand ran twice on one launch, so the migration is queued twice (harmless now that it is fast; I'll guard it on the aw-android side).
• When the datastore worker dies (I hit this with an unreadable seeded DB), Java_…_getBuckets does unwrap() on the SendError/RecvError (aw-server/src/android/mod.rs:198) and panics across JNI on the main thread. Worth a follow-up to return an error JSON like migrateHostname does.

[TimeToBuildBob]

CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.