Repository navigation
feat(*): compress trace payloads, cold logs and message packs - #1149
Conversation
gloryfromca
left a comment
gloryfromca
left a comment
There was a problem hiding this comment.
Blocking: confine pack-index paths and make every supported trace reader decode the new compressed storage.
I found four concrete failures inline: a crafted pack index can escape the trace root, legacy turn matching still consumes compressed bytes as JSON, the viewer accepts Node versions that cannot decode the newly default storage format, and sessions_spawn synthesis bypasses the compressed reader.
Coverage: I read the complete github/main...HEAD diff, relevant trace/trajectory/viewer callers, commit history, AGENTS.md and the Trajectory/Message Blob/Artifact Shell contracts in CONTEXT.md. I checked backward compatibility, RPC/frontend boundaries, dependency and test-file rules, and reviewed the test diff for weakened assertions. I also exercised the confinement and legacy-turn failures directly.
Verification: uv run pytest tests/test_action_executor.py tests/test_cli_cron_commands.py tests/test_cli_tracing_commands.py tests/test_cron_service_dedup.py tests/test_cron_tool.py tests/test_rpc_cli_dispatch.py tests/test_rpc_console.py tests/test_subagent_acp.py tests/test_tracing_api.py tests/test_tracing_message_pack.py tests/test_tracing_store_compression.py tests/test_trajectory_conversation.py tests/test_trajectory_index.py tests/test_utils_mirrors.py tests/integration/test_tracing_viewer_e2e.py -x passed (1,221 tests). A narrower tracing/trajectory run also passed (250 tests). The underlying source-language and large-file checks passed when invoked directly; make is unavailable here. uv run pytest -x could not collect because datrie is absent, and uv sync --all-packages --all-extras could not build it because this environment has no gcc.
360c324 to
db0e3f6
Compare
gloryfromca
left a comment
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
The new revision fixes all four prior blockers: pack entries are confined and range-validated in both readers, compressed turn inputs are decoded with a bound, the viewer enforces Node 22.15, and spawn synthesis now uses the stored-payload decoder with an end-to-end regression. I replied to and resolved all four threads.
I reviewed the delta from c37bd62b0022, the complete github/main...HEAD result, the newly carried channel/onboarding changes and their callers, relevant history, backward compatibility, the AGENTS.md constraints, the Trajectory/Message Blob/Artifact Shell contracts in CONTEXT.md, architecture boundaries, and the test changes for weakened coverage. I found no new plain error.
Verification: the broad changed-area run passed, 1,688 tests in 34.47s. The focused six-suite run passed, 518 tests in 12.85s. check_source_language.py, check_large_files.py, and check_commit_messages.py all exited 0 over the branch range.
db0e3f6 to
f465f9e
Compare
gloryfromca
left a comment
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
The production tree is unchanged from the clean db0e3f6ccd28 revision. This delta adds 13 focused tests for compressed-store failure branches; I reviewed those tests against the exercised implementations and found no weakened assertion or behavior change. The four previously settled fixes remain present.
Coverage: I checked the db0e3f6ccd28..HEAD delta, the resulting github/main...HEAD diff, affected compression/packing/trajectory/viewer callers and history, backward compatibility, AGENTS.md and the relevant CONTEXT.md contracts, architecture boundaries, and test integrity. No new plain error surfaced.
Verification: the broad affected-area run passed, 1,702 tests in 34.83s; the focused compression/packing/trajectory/viewer run passed, 78 tests in 6.26s. Source-language, large-file, and commit-message checks all exited 0.
gloryfromca
left a comment
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
The production tree is unchanged from the clean f465f9e24166 revision. This delta only extends test_tracing_compression_edges.py with pack-stream, validation, bundle-degradation, and turn-input failure coverage. I checked the new assertions against the exercised callers and found no weakened test or behavior change; the previously settled fixes remain intact.
Coverage: I reviewed the f465f9e24166..HEAD delta, the resulting full branch diff, affected compression/packing/bundle/selection callers and history, backward compatibility, AGENTS.md and relevant CONTEXT.md contracts, architecture boundaries, and test integrity.
Verification: the focused compression/packing/trajectory/viewer run passed, 89 tests in 6.19s. Source-language, large-file, and commit-message checks all exited 0.
|
Windows compatibility review — PR #1149 @ Scope: the full base-to-head diff at this head (34 files, +2466/−130): zstd framing for stored payloads ( Result: no Windows-specific defect found.
Verification on native Windows 11, detached worktree
Coverage gaps: the JS viewer's behaviour was checked by reading only — |
a731ba8 to
57447ef
Compare
gloryfromca
left a comment
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
This revision rebases onto the current github/main and adds targeted fixes for edge cases found after the prior clean head: unknown-size zstd frames are measured from the frame start, Node 23.0-23.7 is rejected while 23.8+ is accepted, packed slices have a total decode bound and stream over the cache budget, UTF-8 cuts degrade to the placeholder, and repacking preserves a standalone message when its indexed pack is missing or truncated. The new Message Pack contract matches the implementation. I found no new plain error, and all four threads I previously opened remain resolved.
Coverage: I reviewed the prior-head delta, the resulting github/main...HEAD diff, the affected codec/pack/artifact/CLI callers and history, backward compatibility and corruption behavior, AGENTS.md and the updated CONTEXT.md contract, architecture boundaries, and the tests for weakened assertions. At this late round, no candidate met the introduced + ordinary-use + no-workaround blocker bar.
Verification: the full affected tracing/trajectory set passed, 946 tests in 27.17s; the focused changed paths passed, 180 tests in 9.77s. Source-language, large-file, and commit-message checks all exited 0.
|
Windows compatibility re-review — PR #1149 @ Scope: the branch is rebased onto current main; the delta since the previously reviewed head Result: no Windows-specific defect.
Verification on native Windows 11, detached worktree
Coverage gaps: the JS viewer's behaviour is still checked by reading only — |
Artifacts and content-addressed messages are written as zstd frames under the paths they already had. A reader tells the encoding from the frame, so a store written before compression stays readable and a half-migrated one stays coherent. `raven tracing compress` migrates a store in place, re-linking each artifact onto the compressed blob; `--dry-run` reports the real saving without writing.
Rotation is the one moment a log is closed for good, so the rotated file is rewritten as a zstd frame under the name it already has. Readers of the span log chain decode the frame and keep working in plaintext byte offsets, which is the only coordinate a log cursor and the shard index record. `raven tracing compress --logs` migrates archived logs in place. ACP frame journals stay plaintext: a span records a plaintext byte range into one, and re-encoding it would invalidate that audit record.
One small file per message costs a filesystem block each, so a store of tens of thousands of messages is mostly block slack. Packing concatenates message texts into multi-megabyte zstd frames and keeps an address to (pack, offset, length) index; a reader that finds no file reads its slice. `raven tracing pack-messages` writes the packs, commits the index, and only then removes the files, so a crash at any point leaves every address readable. The writer skips an address the index already holds.
A trace store can be imported from another machine, so a compressed payload is untrusted input. Reading one now stops at MAX_PLAINTEXT_BYTES (256 MiB, far above the largest captured payload) and treats an oversized or corrupt frame as unreadable instead of allocating it. The same bound applies to the viewer's zstd decode through maxOutputLength. resolve_payload also gets a per-shell total budget, so a shell naming many addresses cannot pull in more than one bounded amount.
A pack index ships with an imported store and is untrusted. A name that is not a generated pack basename is refused before it is joined under the pack directory, and the range fields must be non-negative integers. The viewer applies the same validation before it opens a pack.
A turn input written after compression is a zstd frame, but the legacy turn matcher handed its raw bytes to json.loads and found nothing. It now uses the bounded plaintext decoder, so pre-Turn-Trace-Mark sessions still join their conversation messages.
The viewer reads zstd-compressed payloads by default, and zstd landed in Node's zlib in 22.15. The launcher accepted any Node 22 and then showed empty payloads on 22.0 through 22.14; it now refuses those versions and names the real floor.
sessions_spawn synthesis read tool artifacts with a raw text read, so a compressed input or output artifact always parsed to null and the dispatch was silently omitted. It now uses the stored-payload decoder, which picks the encoding from the frame.
Exercise the index write failure, an unremovable message file, an orphan pack sweep, an already-compressed rotated log, a corrupt blob, and the readers that must degrade to missing inside a confined root.
Exercise the streaming slice, an out-of-range index entry, relative message paths, an address that was never packed, hidden files, an unremovable orphan, the bundle import fallback, and the turn-input OSError and oversize branches.
plaintext_size() read the first 18 bytes to ask the frame header for the content size and, when the field was absent, decoded what was left on the handle: a slice starting 18 bytes into the frame. read_plaintext() then saw no zstd magic, treated those bytes as plaintext, and returned the remaining file size as the uncompressed length. A span log written without a content size was catalogued with that wrong size, so SpanLogScanner reported generation_changed on every other pass and SessionIndex reset its entries. Rewind before the fallback decode. Test: tests/test_tracing_compression_edges.py adds a direct length check (12000-byte payload measured as 11 before the fix) and a scanner pass pair (second scan reported generation_changed before the fix).
resolve_payload() guarantees that a message whose blob cannot be read
resolves to the placeholder, never to an exception. Its packed fallback
called _packed_text() outside the try that enforces that: read_message()
returns at most `limit` bytes, and a budget that ended inside a multi-byte
character made decode("utf-8") raise UnicodeDecodeError, which escaped
resolve_payload.
_packed_text() now returns None for that decode failure, so the address
degrades like a missing one.
Test: tests/test_tracing_message_pack.py adds a packed non-ASCII message
under a budget cut inside a character; it raised before the fix and resolves
to the placeholder now.
zstd landed in Node's zlib in 22.15 and 23.8. The viewer's floor compared the version tuple against (22, 15, 0), which accepts 23.0 through 23.7. The launcher let those run, zlib.zstdDecompressSync is absent there, and every compressed artifact was shown as empty. The floor is now explicit per major: 22.15+, 23.8+, or 24 and later. Test: tests/test_cli_tracing_commands.py rejects 23.0 and accepts 23.8.
…budget Two mismatches against the stated decode bound: - _stream_slice() decompressed up to the index's `offset` with no bound; only the slice length was capped. An imported index could name a huge offset and make one read decompress the whole pack. An entry whose offset plus wanted length exceeds MAX_PLAINTEXT_BYTES is now refused, so the skip is bounded too. - _cached_pack() returned the plaintext of every pack it could decode, even one larger than CACHE_BYTES; the module docstring and the PR description say such a pack is streamed per read. It now returns None above the budget and the caller streams the slice it needs. The header is read first, so an oversized pack is not decoded whole just to be refused. Tests: a crafted offset is refused without calling _stream_slice(), and a pack above a shrunken cache budget is streamed and still resolves.
pack_messages() deleted every message file whose sha1 the index held without checking that the named pack still provided the slice. A store copied incompletely, or with a pack deleted or truncated outside this module, keeps a readable message file as the only copy; a run that also packed a new message deleted it, and the address became unreadable. The deletion pass now keeps a file unless the pack decodes whole and its decoded length covers the entry's offset and range. The decoded length, not the frame header's recorded size, because a truncated frame still carries the original size. One read and one decompression per distinct pack, cached for the run however many entries name it. Tests: a missing pack and a truncated pack each leave the restored file in place and still resolving; the healthy case still deletes every packed file.
CONTEXT.md said the message store had no reclaim path and that compact names two content stores. This branch adds `raven tracing pack-messages` and a third store, so the Message Blob entry now points at the Message Pack and names all three. The new term carries its format, migration order, untrusted-index validation and read cache.
The session index identifies a span log by (st_dev, st_ino), which is what lets a rotation rename keep the cursor. Compressing a closed log writes a new file and renames it over the old name, so the same plaintext came back under a new inode. The scanner dropped the old cursor and read the new file from offset 0: every record without trace and span ids was listed a second time under a new position key, a rotation in the middle of a budgeted scan reset the whole index (new epoch), and a spilled trace whose position named the old inode was counted as lost. On ext4 the freed inode is handed to the next file created, usually the new active log, so the old key did not even vanish: it came back holding a shorter file, which read as a truncation and reset the index. Before treating an unknown file as new, the scanner now matches it against the cursors whose key is gone or now holds less plaintext than they consumed: the same sha1 of the first 64 plaintext bytes, and at least as much plaintext as was consumed. A match carries the cursor over at its offset, the key's new holder is read as a new file, and a recovery task holding the old key follows it to the new one. codec.read_head() reads only those first bytes, so a match does not decode the whole archive. Tests: tests/test_trajectory_index.py adds four cases that failed before the change: a log the store compresses after rotation is not read again (a broken record was listed twice), compressing a log mid-scan keeps the epoch (it moved from -2 to -3), a compressed log whose inode is reused keeps the epoch (a hard link keeps the old inode alive so the reuse is the same on every file system), and a spilled trace whose first file was compressed loses nothing (unresolved_dropped was 1).
…ated Every append opens the active log, writes a line and closes it. A writer that opened the file just before another writer's rotation renamed it still writes into that inode a moment later. Compressing the just-rotated log at rotation replaced the inode, so that line went into the unlinked file and was lost. Rotation now compresses the newest older archive of the same kind instead: nothing holds a log one rotation old. One file per rotation keeps the cost where it was (about 9 ms to find it among 2191 archives); an older plain backlog is left to `raven tracing compress --logs`. The re-encoded file keeps its mtime, because the viewer orders logs by mtime and keeps the last write of a span, and an archive compressed a rotation later must not sort after the logs written since. Tests: tests/test_tracing_store_compression.py replaces the test that pinned compression of the just-rotated file with one that pins the previous archive compressed, its mtime unchanged and the newest archive plain, and adds a writer holding the active log across a rotation, whose span was lost before.
…mpress --logs `raven tracing compress --logs` rewrote every plain rotated log, including the one just rotated, which a writer that opened the active log a moment before the rename may still append its last line to. The store stopped compressing that log at rotation for this reason, and the manual migration now follows the same rule: the newest rotated log of each kind is counted and left for the store's next rotation, and every older one is compressed. A compressed log also keeps its mtime now. The viewer orders logs by mtime and keeps a span's last write, and choosing the newest log of each kind reads the same mtimes: with the migration's own time on every rewritten file, a second run took an already compressed log for the newest and compressed the real newest one. The command prints how many logs it left for the next rotation. Tests: tests/test_tracing_store_compression.py adds a store with two rotated logs of each kind, where the newest of each stays plain and the older ones keep their mtimes; it failed before the change. The tests that planted one rotated log and expected it compressed now plant two and assert that the older one is compressed, so none of them passes with nothing compressed. The two index tests that used the migration to compress the one log they need now re-encode it with the store's per-file compression.
The store heals a missing message blob: the next turn that sends the message writes it again. address_items() skipped that write for any address the pack index held, without checking that the pack still existed, so a store copied without its packs kept those addresses unreadable however often the message was sent again. _is_packed() now also requires the pack file the index names to exist. That is one stat per packed address. A truncated pack is not detected here: proving it would mean decoding packs on every write, and a conversation's messages are spread over all packs, so that is left to pack_messages(), which decodes before it deletes anything. Test: tests/test_tracing_message_pack.py deletes the packs and sends the same messages again; the files were not written before the fix and resolve now.
find_node() returns the first candidate at or above the TUI's floor (22.0), in the order RAVEN_NODE, venv, PATH, Raven's private runtime. The viewer then refused that node when it had no zstd, so a PATH node in 22.0-22.14 or 23.0-23.7 hid a zstd-capable node later in the order, such as the runtime the installer puts under ~/.raven/runtime, and the launcher reported that no usable Node was found. find_node() takes an optional acceptance predicate, defaulting to the TUI's floor, and the viewer passes its zstd check, so the search continues past a node the viewer cannot use. It still returns the best node it saw when none qualifies, for the error message. Test: tests/test_cli_tracing_commands.py puts a 22.12 node on PATH and a 22.20 runtime under RAVEN_HOME; the launcher refused before the fix and picks the runtime now. The two viewer floor tests stub find_node with the new argument.
The command table described `raven tracing pack-messages` as packing message blobs into "archives". CONTEXT.md lists "archive" under the Message Pack entry's Avoid list, because it names the rotated-log directory that `raven tracing compress --logs` works on. Both language versions now say Message Pack.
57447ef to
93cfa8f
Compare
gloryfromca
left a comment
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
Reviewed the six commits added after the previously clean head and confirmed the rebased first 16 commits are patch-equivalent. The new changes correctly carry trajectory cursors across in-place log re-encoding, avoid rewriting the only rotated log that may still have an open writer, preserve log ordering metadata, heal messages whose indexed pack is missing, let the viewer's zstd requirement drive Node selection, and use the canonical Message Pack term.
Coverage included AGENTS.md and CONTEXT.md rules, the base-to-head diff and new-head delta, affected callers and history, plaintext/compressed backward compatibility, test changes for weakening, and the storage/viewer architecture constraints. I found no weakened tests.
Verification: uv run pytest over the affected tracing, trajectory, RPC, CLI, and subagent suites: 954 passed; viewer integration suite: 28 passed. Source-language, large-file, and commit-message checks all passed. The four earlier findings remain resolved.
|
Windows compatibility re-review — PR #1149 @ Scope: the branch is rebased onto current main (which now carries the merged #1141); the delta since the previously reviewed head Windows angle: the cursor fix is deliberately filesystem-independent — it fingerprints plaintext rather than relying on inode-reuse timing (the test keeps the old inode alive through a hard link so the reuse case is the same on every filesystem), which is also what makes it correct on NTFS, where compressing a file stages and renames a new file and therefore gives it a new file index. Verification on native Windows 11, detached worktree
Coverage gaps: the JS viewer's behaviour is still checked by reading only — its integration test is deselected by the suite's marker. The full suite was not rerun. |
A pack can be left short by an interrupted copy while the index still names its slices. The writer only checked that the pack file existed, so an address whose slice no longer decoded kept resolving to the missing-blob stand-in however often its message was sent again; running `raven tracing pack-messages` again could not help either, because the message files were deleted when the pack was written. The writer now asks whether the pack still decodes and covers the slice. The decoded length is streamed and cached per pack, so the first check of a pack costs one decompression and every later address is a stat plus a lookup. A frame cut short decodes to a shorter length rather than raising, so an entry before the cut stays packed and only the entries past it are written again. Test: tests/test_tracing_message_pack.py truncates a pack to its magic and re-sends the same messages. The files were not written before the change and resolve now.
gloryfromca
left a comment
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; suggestions only, and they are marked inline.
The truncated-pack recovery is correct: the writer now republishes only addresses whose indexed slice is no longer covered, while retaining readable slices and caching the decoded length. I reviewed the delta, its store caller and pack lifecycle, history, backward compatibility, AGENTS.md/CONTEXT.md rules, the untrusted-store architecture, and the test changes; no tests were weakened.
Nonblocking follow-up: raven/tracing/message_pack.py's new _decoded_length() should stop and refuse the pack once total exceeds MAX_PLAINTEXT_BYTES. It currently streams an imported pack to EOF from the ordinary TraceStore.address_items() path, so a crafted small zstd frame can force gigabytes of decompression work on the first matching write, bypassing the codec's imported-store resource ceiling. This requires a hostile/corrupt imported pack and has a manual escape, so it does not meet the late-round blocker bar.
Verification: 955 affected tracing, trajectory, RPC, CLI, and subagent tests passed; 28 viewer integration tests passed. Source-language, large-file, and commit-message checks passed. Existing reviewer-owned threads remain resolved.
Summary
Three storage layers of the trace store, each with an idempotent, opt-in migration:
raven tracing compressmigrates an existing store in place and moves each artifact hard link onto the one compressed blob;--dry-runcompresses in memory to report the real saving and writes nothing.raven tracing compress --logs, which leaves the newest rotated log of each kind for that next rotation; a compressed log keeps its mtime, which is how the viewer orders logs. The span-log readers decode by the frame and keep working in plaintext byte offsets, which is the only coordinate the session index cursor and the viewer shard sidecars record. Compression gives a log a new inode, so the session index recognises it by the sha1 of its first 64 plaintext bytes and carries the cursor over instead of reading it again.sha1 -> (pack, plaintext offset, length)index.raven tracing pack-messageswrites the packs, commits the index atomically, and only then removes the message files; a reader that finds no file reads its slice.A fourth commit bounds what a stored frame may decode to, after the automated security review flagged decompression bombs. A later review round fixed six defects, each with a regression test that failed before its change: a frame without a content-size field was measured from the wrong offset (a span log then reset the session index on every other pass); a per-shell budget cut inside a multi-byte character raised instead of degrading; the viewer accepted Node 23.0-23.7, which have no zstd; the pack reader's streaming skip was unbounded and packs past the cache budget were decoded whole rather than streamed; the pack cleanup deleted a message file whose pack no longer decoded; and CONTEXT.md still said the message store had no reclaim path. A third round fixed four more the same way: compressing a rotated log replaced its inode, so the session index read it again from offset 0 (a record without ids was listed twice, a rotation mid-scan reset the index, as did ext4 handing the freed inode to the next log, and a spilled trace was counted as lost); compressing the log just rotated dropped a line another writer had opened the file to append, at rotation and in
raven tracing compress --logsalike; the writer did not write a packed message again when its pack was gone or cut short, so a store copied incompletely never healed; and a PATH Node without zstd hid a zstd-capable one later in the search order, such as the runtime under~/.raven/runtime. The command table also stopped calling packs archives.Why: the live store on this machine was 1357.7 MB, most of it block slack and uncompressed JSON. On a clone of that store the archive logs fell from 94.0 MB to 8.2 MB, and 40,056 messages fell from 225 MB of disk to 12 packs (23.6 MB) plus a 3.9 MB index. Projected whole-store result: about 548 MB, roughly 60 percent saved.
Key decisions:
artifact_pathwhen it is written and those records are the audit trail, so a migration cannot rename anything. Readers sniff the zstd magic, which is also what keeps a store written before this PR readable and a half-migrated one coherent.subagent.external.frames.start/end), and feat(*): redraw the trajectory as an agent waterfall with external agents' steps #1147's frames replay reads exactly those offsets. Re-encoding a journal would silently invalidate that audit record, so neither the migration nor rotation touches it.Relationship to #1147: the two branches overlap in four files (
CONTEXT.md,raven/trajectory/conversation.py,raven/trajectory/details.py,tests/test_subagent_acp.py), none of them touched by the third-round fixes, and a merge-tree dry run against its head1950ad79has no conflict. #1147 does not touchpyproject.tomloruv.lock. This branch was rebased onto main; the only conflict was main'sopenaidependency from #1108, resolved by keeping both entries and re-locking. A second rebase, onto7c12b93f(#1141), had no conflict.Deployment: deploy this code before running either migration. The bundled viewer needs Node 22.15 or newer (23.8 on the 23.x line) for the zstd decoder, the same requirement its compressed-artifact path already has. Code older than this PR cannot read a compressed payload, and the write path compresses new artifacts by default, so the deployment is one-way once anything new is written; the migrations themselves are manual and opt-in.
Measurements on a clone of the live store (2026-10-29):
compressed 0, already 33.Type
Verification
Commands and results:
uv run pytest tests/test_tracing_store_compression.py tests/test_trajectory_index.py tests/test_cli_tracing_commands.py tests/test_tracing_api.py tests/test_subagent_acp.py tests/test_trajectory_conversation.py tests/test_trajectory_details.py tests/test_trajectory_selection.py tests/test_trajectory_bundle.py tests/test_trajectory_export.py tests/test_rpc_cli_dispatch.py tests/test_utils_mirrors.py tests/test_node_server.py tests/test_tracing_message_pack.py -q -n 4: 1127 passed, 1 skipped.57447ef8and passes now, and removing each fix again (nine mutants: no cursor adoption, adopting only for a key that vanished, no moved-key lookup in the rescan, compressing the just-rotated log, no mtime kept at rotation, the migration compressing the newest log, the migration not keeping mtimes, an index-only packed check, the TUI floor in the viewer's Node search) turns exactly its own tests red.uv run pytest tests/test_trajectory_index.py tests/test_tracing_store_compression.py tests/test_tracing_compression_edges.py tests/test_tracing_message_pack.py tests/test_cli_tracing_commands.py -q: 188 passed, and the same files pass on each of the six fix commits' own tree.tests/test_tracing_message_pack.pyfailed before the change (the message files were not written again) and passes now; the decoded length is streamed and cached per pack, so a healthy store pays one decompression per pack rather than one per address. The affected tracing, trajectory, CLI and RPC set passed: 1050 tests. Full suite at the fixed head: 37157 passed, 8 failed - the same known local baselines as before.uv run pytest tests/integration/test_tracing_viewer_e2e.py -m integration -q: 28 passed.uv run pytest tests/integration/test_kernel_wheel_smoke.py -q: 1 passed.7c12b93f. The eight are the known local baselines on this box (five proxy-environment tests, the Excel parser, the install script, and the node runtime probe); the same eight IDs also fail at57447ef8here, and none of the new tests is among them. The new index tests also pass with the test temp directory on ext4, which reuses a freed inode at once, and on tmpfs, which does not.uv run ruff checkanduv run ruff format --checkon the changed trees: clean.uv run --frozen --all-extras ty check raven/tracing raven/trajectory raven/cli/tracing_commands.py raven/cli/tui_commands.py raven/rpc/methods/cli_dispatch.py: all checks passed.uv run --frozen --all-extras lint-imports: 10 kept, 0 broken.scripts/check_commit_messages.py origin/main..HEAD,commitlint --from origin/main --to HEAD,scripts/check_source_language.py origin/main...HEADandscripts/check_large_files.py origin/main...HEAD: exit 0.Risk
Security impact considered
Backward compatibility considered
Rollback path is clear for risky changes
Security: local files only; no new network surface, credentials, or permissions. A trace store can be imported from another machine, so decompression is bounded rather than trusted: every decode path stops at 256 MiB (including the pack reader's offset skip), far above the 32.9 MB largest captured payload, and treats an oversized or corrupt frame as unreadable, the viewer passes
maxOutputLengthto its zstd decode, andresolve_payloadcarries a per-shell total budget so a shell naming many addresses cannot pull in more than one bounded amount. The pack index is validated (40-hex addresses, integer offsets) before any path is built.Backward compatibility: readers tell the encoding from the frame, so plain and half-migrated stores read. Older code cannot read a migrated payload, which is why both migrations are manual and the deployment order is stated above. ACP journals are not migrated at all.
Performance: measured read overhead is sub-millisecond; the pack cache is capped at 32 MB per process and the index is 3.9 MB for 40,000 messages. A rotation compresses at most one earlier log, and finding it among 2,191 archives took 9 ms.
Rollback: the write path stores every new artifact compressed, so a revert is only clean if no artifact has been written by this version. After any new write or a migration there is no decompress command in this PR; any current reader still returns the original plaintext, and a decompress command can be a follow-up if one is wanted.
Related Issues
N/A. The relationship to #1147 is described in Summary.