[Engine] Automatic disk-backed conversation cache with size cap and auto-eviction, for subagent workloads on low-RAM machines #3891

Area

[Engine]

What do you want to do?

Keep conversations alive across turns when several of them alternate on a server that handles one request at a time, without the client calling any API and without the box having RAM to spare for parked snapshots. In short: an opt-in, automatic disk tier under the conversation cache that is configured in the run config and manages itself.

Who needs it: people running agent clients with subagents (e.g. Claude Code with 4 subagents sharing one server) on machines with limited RAM, where one concurrent request is all the memory allows.

Requested behavior

• Automatic. When the active conversation changes, the engine parks the outgoing one and, when a conversation returns, restores it, with no /slots/0?action=save|restore calls from the client.
• Disk as a tier, not only RAM. Park to a directory on SSD/NVMe; the RAM budget can be small or zero.
• Self-managing size. Config options along the lines of a directory, a total size cap (MiB), and optionally a maximum entry age or entry count. Least-recently-used entries are evicted when the cap is reached; stale or orphaned files are cleaned up at start and stop. No manual cleanup.
• Cheap switches. Restoring from disk should be much cheaper than re-reading a long prompt (see the numbers below).
• Opt-in and off by default, so nothing changes for existing setups.

Why: what the docs say happens today

Reading docs/DETAILS.md and docs/BATCHING.md (0.2.3.1):

• Without "parallel": N Strata serves one request at a time; others wait in the queue.
• The engine holds one live session plus up to 6 checkpoints in RAM that are shared by all conversations and rotate by least recent use. The oldest (the system-prompt root) is kept for good.
• "one conversation cached at a time (switching between two chats re-reads the other one unless the opt-in cache above is enabled, or each conversation keeps its own batch slot)".
• So with several subagents taking turns, each one probably re-reads its own conversation from the shared system-prompt root, unless one of its checkpoints survived. (This is my reading of the docs; I have not measured the re-read cost on my setup.)

The existing options each need something these machines lack:

• --conversation-cache-mib parks in RAM only. The docs themselves describe stalls from paging when the budget is large (Deterministic deadlock at the completion handshake after the last block: host at layer step 48 with the GPU already rang - 5/5 stalls identical, on 0.1.41 and 0.1.43 (RTX 4080 / Windows / IQ3_XXS native pack / 256K) #2049).
• "parallel": N needs VRAM per slot and a 262144 context is already tight.
• Disk sessions (--slot-save-path, POST /slots/0?action=save|restore) work, but only through explicit calls: single slot 0, no erase action (the caller manages the files), they run between requests while the queue is held, and they return 501 with parallel requests. A client such as Claude Code will never call them, so they cannot help here without a wrapper in front of the server.

For scale, the docs measure a 63,025-token conversation on a 12 GB card with NVMe: a 1.20 GB file, save 0.65 s, restore 1.05 s, against 25.5 s for the cold first turn. Single runs, different hardware and model than mine, but it shows why a disk tier is attractive when RAM is short.

My setup

Linux, AMD/HIP (container), about 62 GiB RAM with about 8 GiB available while the model runs, container capped at 57 GiB; Qwen3.8-Flash-Next IQ3_S, --max-context 262144 --kv int8, no parallel, no conversation-cache budget. Clients: Claude Code and similar agents with several subagents.

Related

• [Engine] Measurements for a disk tier under the conversation cache (single GPU, Linux) #3485: measurements for a disk tier under the conversation cache (the same direction; this issue asks for the feature to be automatic, self-managing and configurable, aimed at the low-RAM subagent case)
• [Engine] Share identical KV prefix blocks across concurrent conversations #1962: share identical KV prefix blocks across concurrent conversations
• Conversation cache slots are consumed by successive turns of the same subagent, evicting the parent conversation #342: subagent turns evicting the parent conversation's cache (closed)
• [Engine] session files (SAVE/RESTORE) refuse multi-GPU layer-split runs — request per-device session files #3744: session files refuse multi-GPU layer-split runs

What do you do today instead?

Nothing automatic. The options are accepting the re-reads, running fewer concurrent subagents, or putting a client-side wrapper in front of Strata that drives the slot API, which means identifying conversations from the outside and blocking the queue while it saves and restores.