Skip to content

fix: 复用来源缓存加速记忆页面 Agent 筛选 - #800

Merged
origami-chen merged 3 commits into
MemTensor:v1.2.6from
Orsted290:fix-agent-filter-cache-indexes
Nov 1, 2026
Merged

origami-chen merged 3 commits into
MemTensor:v1.2.6from
Orsted290:fix-agent-filter-cache-indexes

Conversation

@Orsted290

Copy link
Copy Markdown
Contributor

问题定位

记忆页面切换 Cursor、Codex 等 Agent 时,菜单立即切换,列表仍等待成员查询。前端按子栏目保留当前筛选视图,返回已访问 Agent 也会重新创建视图;后端虽然已准备 panel_memory_facts,成员筛选仍从 memories 反复解析历史 JSON、归一化来源并排序。

修改内容

  • 管理面板的受支持筛选复用 panel_memory_facts.source,按不同来源值计算别名匹配,再通过来源及排序索引查询成员。正文与标签组合搜索继续按需读取原表,其他 repository 过滤保留旧路径。
  • 保留现有 dirty 增量维护和实时 raw 删除、脱敏、会话可见性检查。旧缓存就绪时安装索引,空缓存回填结束后一次建索引,避免每个小批次反复维护索引拖慢首次回填。
  • 保留菜单、布局、固定分页、刷新、取消与迟到响应处理。返回已访问 Agent 仍发请求,但不再全量解析历史来源 JSON。
  • 补充来源别名/优先级、Memmy/电脑历史归类、其他来源、正文/标签组合搜索、排序、删除、增量更新、旧缓存升级及中断回填后的重开测试。

测试结果

基线为 330c393e。使用初始 47,482 条合成记忆、真实 React 页面、Backend/Memory HTTP 和 SQLite worker。计时从点击到全部实际卡片标题匹配本次响应,再等待两个动画帧;未添加 SQL 计时包装、未增加超时。OS 磁盘缓存未人为清空,空缓存指 panel facts 尚未回填。

场景 基线 优化后
缓存就绪,首次 Cursor(两轮) 约 356 ms 38–47 ms
返回 Cursor 328–361 ms 32–40 ms
空缓存,菜单出现后立即筛选 Cursor 3288 ms 948 ms
  • 40 组实际页面的 ID、顺序、标题、页码与基线一致;另在增量后的大库比较 12 组完整成员查询,ID 与顺序全部一致。
  • UI 验证覆盖筛选后翻页、刷新、25 条来源修改后的增量刷新、后台写入与统计并发,以及快速切换后等待迟到响应仍显示最终正确内容。
  • npm run memory:test 最终完整补跑:269 个文件通过、2 个跳过;2867 项通过、2 项跳过。此前一轮有一次并发捕获 HTTP ECONNRESET,未改代码、超时或重试,单独重跑该文件 4 项通过,再完整补跑通过;该偶发连接错误根因未定位。
  • 前端相关 4 个文件、72 项测试通过;npm run lint、npm run typecheck、git diff --check 通过。
  • 最终 unsigned macOS arm64 DMG 已隔离安装:完整应用首次 Cursor 229 ms,随后切换约 37–40 ms;分页、刷新、快速切换及退出重启核验通过。重启后的 47,482 条源记忆内容摘要与原合成库一致。用户手动验收通过。

性能边界:空缓存首次“全部”加载因一次建索引从约 776 ms 增至 929 ms;后台并发下部分切换仍约 315 ms,不声称所有场景都是几十毫秒。上述不同缓存条件分开计量,未将原报告的首次 2.9 秒直接折算为固定加速比例。

请重点核对来源匹配、raw 可见性及索引建立时机。

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #800
Task: 58dce77eb02fe301
Base: v1.2.6
Head: fix-agent-filter-cache-indexes

🔍 OpenCodeReview found 5 issue(s) in this PR.

⚠️ Review warnings (1)

  • bounded analysis finalized with explicit task_done after reviewing all group files

1. Memory/src/storage/repositories.ts (L1268)

SELECT DISTINCT source FROM panel_memory_facts is re-executed on every panel-members request that carries a source filter (which is the common case), and it has to scan the whole facts table (or the whole idx_panel_memory_source index) just to rediscover a handful of producer values that only change when a memory write touches source. On a large memory set this adds an O(#facts) scan to a management read. Consider materializing the distinct producers during the cache fill (e.g. a small panel_memory_sources table maintained alongside panel_memory_facts) and rewriting this as f.source IN (SELECT source FROM panel_memory_sources WHERE ...), so the per-request cost stays proportional to the number of producers rather than the number of facts.


2. Memory/src/storage/repositories.ts (L1246)

This key allow-list duplicates the MemoryFilter contract and is rebuilt on every call. Two consequences: (1) a typo or a newly added filter key silently disables the cached path (correctness stays safe, but the optimisation silently disappears), and (2) the memory_layer/status clause construction just below re-implements buildMemoryWhere's addArrayClause inline, so the fast path and the fallback buildMemoryWhere can drift apart unnoticed. Suggest hoisting the set to a module-level constant typed against keyof MemoryFilter and reusing a shared arrayInClause(column, value) helper for the layer/status clauses.

💡 Suggested Change

Before:

    const cachedKeys = new Set(["panelSource", "panelVisible", "memoryLayer", "status", "agentId", "excludedAgentIds", "tags"]);

After:

    const cachedKeys = new Set(["panelSource", "panelVisible", "memoryLayer", "status", "agentId", "excludedAgentIds", "tags"]);

3. Memory/src/storage/repositories.ts (L1250)

The cached path silently drops the deleted_at IS NULL predicate that buildMemoryWhere always enforces (line 7832), so soft-deleted rows are excluded only as a side effect of fillNativeFacts pruning facts (DELETE ... WHERE NOT EXISTS(SELECT 1 FROM memories s WHERE s.id=... AND s.deleted_at IS NULL)). Any path that leaves a fact behind for a soft-deleted memory (a fill that was never re-run for that id, a rebuild of memories during a migration, a fact written by a future projection) makes this branch return memories the authoritative path considers deleted, and the two will disagree on page membership. Please either document the invariant explicitly or add a cheap guard (e.g. join/EXISTS on memories.deleted_at IS NULL) so the equivalence with buildMemoryWhere is enforced by the query itself rather than by the pruning side effect.


4. Memory/src/storage/repositories.ts (L1249)

This turns a read-model method into one that may write, and it duplicates the call the only production caller already makes (panel-read.ts:638). Two consequences: (1) on a readonly connection ensurePanelCache throws Panel cache changed before read snapshot instead of degrading, so any future readonly caller of panelMemberIds (tests, snapshot readers, CLI) now fails where the previous implementation merely read panel_memory_facts; (2) in the writable service path a full, synchronous drain of panel_memory_dirty can be triggered inside a request. Consider keeping ensurePanelCache at the read-model boundary and asserting !panelCachePending(db, ["memories"]) here instead.


5. Memory/src/storage/panel-cache.ts (L86)

Index availability now depends on the dirty queue being empty at the exact moment this runs. If a fill never reaches a fully drained queue — the background prepare path in reader-worker.ts snapshots ids via panelStatisticsDirtyIds and stops, so concurrently inserted dirty rows keep the table non-empty — the two indexes are never (re)created and the new f.source IN (...) / ORDER BY created_at query silently falls back to a full scan of panel_memory_facts. Consider using an explicit signal for "bulk backfill finished" (e.g. a flag in panel_cache_meta, or creating the indexes once in the migration when the facts table already exists) rather than inferring it from an empty queue.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: NO TEST SCOPE

Automated tests were not run because the changed files do not map to an executable test scope.

Details: No executable test scope maps to the changed files. Automated tests were not run; add env.yaml source_mapping + execution for this path family, then rerun. Changed files: Memory/src/storage/panel-cache.ts, Memory/src/storage/repositories.ts, Memory/tests/service/read-model/panel-source-visibility.test.ts, Memory/tests/storage/panel-members-cache.test.ts
Manual review or env.yaml source_mapping/execution coverage is required before merge.

Branch: fix-agent-filter-cache-indexes

@origami-chen
origami-chen merged commit cab8dba into MemTensor:v1.2.6 Nov 1, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants