fix(datastore): apply privacy filters on save and startup

[TimeToBuildBob]

Summary

Privacy filter rules saved from the webui never actually applied. DatastoreWorker starts with an empty PrivacyFilterEngine, insert/heartbeat paths do filter against that engine, and RefreshPrivacyFilter reloads settings.privacy_filters — but nothing called refresh after save, and startup never loaded the persisted key. The UI could save drop/redact rules all day; matching events were stored unchanged.

Fixes #659.

What changed

• Reload the in-memory engine when settings.privacy_filters is written or deleted (SetKeyValue / DeleteKeyValue).
• Load the same key at worker startup so rules survive a server restart.
• Keep RefreshPrivacyFilter as an explicit path (HTTP settings endpoints still call it as belt-and-suspenders).
• Datastore tests: a drop rule saved via set_key_value actually drops a matching insert (no explicit refresh), and the same rule still applies after reopen.

Test plan

• cargo test -p aw-datastore --test datastore
• Save a drop rule in the webui Privacy Filters settings, generate a matching window event, confirm it is not stored.
• Restart the server with that setting present, confirm matching events still drop.

[greptile-apps]

Greptile Summary

The PR keeps the in-memory privacy filter synchronized with its persisted setting during startup and settings changes, while restoring the prior filter state if a transaction fails.

• Loads persisted privacy rules before processing events.
• Reloads rules when the privacy setting is written or deleted.
• Restores the transaction-start engine after a failed commit.
• Adds tests covering immediate activation, deletion, and restart persistence.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the current rollback path restores the transaction-start filter engine, and non-absence loading errors preserve the active rules.

Important Files Changed

Filename	Overview
aw-datastore/src/worker.rs	Synchronizes privacy-filter state with settings and restores the pre-transaction engine on commit failure; the previously reported rollback and recovery issues are addressed.
aw-datastore/tests/datastore.rs	Adds coverage for activation after saving, deactivation after deletion, and loading persisted rules after reopening.
aw-server/src/endpoints/settings.rs	Retains explicit privacy-filter refreshes after successful HTTP setting writes and deletions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Start worker] --> B[Load persisted privacy filters]
    B --> C[Begin transaction]
    C --> D[Snapshot active engine]
    D --> E[Process settings and event commands]
    E --> F{Commit succeeds?}
    F -- Yes --> G[Keep updated engine]
    F -- No --> H[Restore transaction-start engine]
    G --> C
    H --> C

Loading

_{Reviews (4): Last reviewed commit: "fix(datastore): restore privacy engine f..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 91.02564% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.75%. Comparing base (656f3c9) to head (1b3aaa2).
⚠️ Report is 97 commits behind head on master.

Files with missing lines	Patch %	Lines
aw-datastore/src/worker.rs	77.77%	4 Missing ⚠️
aw-server/src/endpoints/settings.rs	66.66%	2 Missing ⚠️
aw-datastore/tests/datastore.rs	98.14%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #662      +/-   ##
==========================================
+ Coverage   70.81%   79.75%   +8.93%     
==========================================
  Files          51       66      +15     
  Lines        2916     5680    +2764     
==========================================
+ Hits         2065     4530    +2465     
- Misses        851     1150     +299     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TimeToBuildBob]

macOS CI failure was the known sync_roundtrip flake, not this PR

Run 33137074103 failed only on macOS, in aw-sync:

thread panicked at aw-datastore/src/datastore.rs:129:6:
Failed to upgrade database when adding data field to buckets:
  SqliteFailure(..., Some("duplicate column name: data"))

test_own_data_does_not_return_via_peer and test_push_does_not_reexport_synced_buckets both call round_trip(), which builds temp db paths from pid + name + nanos. Run in parallel on macOS they can land in the same clock tick, open the same SQLite file, and race the v1→v2 migration. Nothing to do with privacy filters — it hits any PR in this repo.

That is exactly what #655 fixes (green, mergeable, open since 2026-08-24). To unblock this branch's CI the same fix was carried here, and cfbfad9 makes aw-sync/tests/sync_roundtrip.rs byte-identical to #655's version — so #655 and #662 land in either order with no conflict, and both keep the timestamp component (the counter alone resets on PID reuse, and these tests never delete their temp dbs).

Also in this push: 95898ce addresses the Greptile P1 — a failed batched commit rolled SQLite back but left the in-memory privacy engine holding the rolled-back rules.

Local: cargo test -p aw-sync --test sync_roundtrip 3/3, cargo test -p aw-datastore 13/13, fmt clean, no new clippy warnings.

Merging #655 is still worth doing on its own — it fixes the flake on master, where this PR's copy does not help.

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

Rebased onto current master (2ded7d7). Conflicts were only in aw-sync/tests/sync_roundtrip.rs — the two tmp_db uniqueness commits already landed via #666, so I dropped them. Privacy-filter commits applied cleanly. cargo test -p aw-datastore --test datastore: 19 passed.

[TimeToBuildBob]

Greptile 3/5 leftover: a rolled-back delete plus a failed durable reload could keep the uncommitted (empty) engine, so later inserts ran unfiltered.

1b3aaa2 snapshots the engine when the SQLite transaction opens — before any SetKeyValue/DeleteKeyValue reload — and restores that snapshot on commit failure. Recovery no longer depends on a second read succeeding.

cargo test -p aw-datastore --test datastore: 19 passed. Local in-band review 5/5.

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.