fix(security): store ciphertext in L1 for encrypted caches (LAB-238) - #104
Conversation
L1 held post-decrypt plaintext for encrypted caches, so any heap dump, core
dump, or Node diagnostic report yielded the entire L1 working set in the clear
for its full TTL — and that plaintext outlived the key zeroization in close(),
because L1 entries are held independently of tenant keys. It also broke parity
with cachekit-py, whose L1Cache stores bytes and decrypts at read time, and
cachekit-rs, which keeps ciphertext across every layer.
All three population sites now store what L2 stores:
- getEntry() writes the backend bytes it read, not the value it decoded
- setEntry() writes the ciphertext it produced, not the caller's value
- the SWR refresh writes what the L2 write handed back, not the factory result
To close the third site, the persist callback returns an L1Write ({ l1 })
instead of void. The wrapper is what distinguishes "store this" from "there is
nothing to store": a degraded write on a secure cache has no verified
ciphertext to show for itself, so the refresh cancels and L1 keeps the stale
entry rather than falling back to the plaintext it just computed. Plaintext
caches still get their value back on a degraded write, so their SWR behaviour
is unchanged.
Every L1 hit path — get, wrap's SWR read, and wrap's no-waitUntil fallback —
decrypts and AAD-verifies against the cache key. An entry that fails to verify
is dropped and the read falls through to L2, mirroring cachekit-py's L1
handler, which invalidates before applying its fail policy so a poisoned L1
copy cannot outlive remediation of L2. The failure is logged, not swallowed.
Two hazards that came with holding bytes in L1:
- estimateSize measured with JSON.stringify, which renders a Uint8Array as
{"0":171,...} — ~14x its real size. Unfixed, the first encrypted entry would
have evicted most of L1.
- a Node Buffer from the backend is a window onto a shared 8 KiB pool slab, so
retaining one for the entry's TTL pins the slab. Copy when the view is
narrower than its buffer; cachekit-py refuses memoryview/bytearray in
L1Cache.put for the same reason.
Non-encrypted caches keep storing decoded values, unchanged.
Panel review of ecfd912 at critical stakes (per ray's LAB-131 gate: any encryption/AAD diff in a cachekit repo needs one). Five findings applied, four rejected with reasons. CRIT — degraded L2 write became an origin stampede. Found independently by two reviewers and measured: with the backend down, 11 reads of an encrypted key drove 11 origin calls, against 1 for plaintext. Returning null for "nothing storable" made every SWR refresh end in cancelRefresh, which frees the refresh marker while leaving expiresAt untouched — so the entry stayed stale and every subsequent read re-armed the refresh, on exactly the encrypted caches that carry PII, and invisibly to any plaintext load test. Two changes: the ciphertext is now captured the moment encrypt() produces it, before the backend write that may fail, so a degraded write still yields an L1 payload and the refresh resets freshness as it always did; and the residual null case (encrypt itself failed — nonce exhausted, manager disposed) deliberately leaves the marker to lapse via SWR_REFRESH_MARKER_TTL_MS, throttling retries to one per key per minute instead of one per read. MAJ — a cached null read as a decrypt failure. readL1 overloaded null as both "AEAD verification failed" and "the value is null", so a secure cache holding null invalidated and re-fetched a perfectly good entry on every hit — a billed miss per read on a metered backend. The same commit added the L1Write wrapper for this exact reason on the write side and left the read side unwrapped; decodeL1Entry now returns { value } | null. MAJ — exists() trusted L1 presence without decrypting, so after a key rotation it reported present for entries get() verifies, rejects and drops. It now decodes, so exists() and get() cannot disagree. MAJ — an AES-GCM tag failure is the canonical tamper signal and reached only a log line. It now goes through recordFailure so operators can alert on it. MAJ — reliability.degradation governs an L2 decrypt failure (it runs inside the executor) but not the new L1 one. decodeL1Entry now honours the same lever: degradation off rethrows instead of falling through. Also: extracted decodeEntry so the L1 and L2 tiers cannot drift into decoding the same entry differently; renamed readL1 to decodeL1Entry (it decodes an already-read value and evicts on failure); dropped a vestigial type parameter; documented that the buffer-copy rule guards slab pinning, not a backend that mutates buffers it handed over. Rejected: a new encryption.failClosed option (new public API, the existing degradation lever covers it); copying L1 bytes unconditionally (every in-tree backend was verified to return an owned exact-size buffer, and the copy is real cost on large values); a synchronous fast path for plaintext L1 hits (unmeasured microtask against three duplicated ternaries on a crypto path); and re-checking the L1 version token after the decrypt await to close a sub-millisecond read-vs-invalidate window (needs new L1 API, cannot be tested deterministically, and the authoritative delete already went to L2). Tests: three new regressions, each verified to fail against ecfd912 — the stampede (11 origin calls vs 2), the null round-trip, and exists() verification. Deleted a dead assertion that stringified bytes to digits and so could never have caught a leak. Widened the SWR test's stale window from 300ms to 600ms of headroom so a loaded CI box cannot take the cold path.
|
Important Review skippedNo new commits to review since the last review. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughEncrypted caches now store ciphertext in L1 and decrypt entries only after read-time validation. Background refreshes use the exact persisted L1 payload. Binary L1 entries now use byte-accurate memory accounting. ChangesEncrypted L1 flow
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant CacheCore
participant L2Persistence
participant BackgroundRefreshManager
participant L1Cache
Caller->>CacheCore: request cached value
CacheCore->>L1Cache: read ciphertext
L1Cache-->>CacheCore: return ciphertext
CacheCore->>CacheCore: decrypt and validate AAD
CacheCore->>L2Persistence: persist encoded value during refresh
L2Persistence-->>CacheCore: return L1Write payload
CacheCore->>BackgroundRefreshManager: complete refresh with payload
BackgroundRefreshManager->>L1Cache: store ciphertext
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
This comment has been minimized.
This comment has been minimized.
… test master key (LAB-238) Kody round 2026-08-07: the encrypted L1 decode path now instanceof-guards the stored entry before decrypt — a non-bytes entry rides the existing invalidate/degradation path instead of failing inside the native decrypt. The test helper narrows via instanceof instead of casting, and the test master key is generated per-run rather than embedded as a literal.
This comment has been minimized.
This comment has been minimized.
|
@kody start-review |
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/cachekit/src/cache/background-refresh.ts (1)
105-136: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winAlign the error path with the "do not cancel" reasoning.
The null branch documents why
cancelRefreshis wrong after a failed refresh: cancelling frees the marker whileexpiresAtstays unchanged, so the next read re-armsshouldRefreshand the origin is hammered for the rest of the TTL.The catch block at line 133 does exactly that. A
persistToL2rejection — the fail-closed configuration of the same degraded write — releases the marker and leaves the entry stale. The stampede the null branch prevents returns on the throw path.Confirm the two paths are meant to differ. If the reasoning applies to both, let the marker lapse in the catch block as well, or document why a thrown failure warrants an immediate retry while a null return does not.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cachekit/src/cache/background-refresh.ts` around lines 105 - 136, Align the catch path in the background refresh flow with the failed-persistence behavior documented in the null branch: remove the l1Cache.cancelRefresh call so the refresh marker can lapse and throttle retries. If thrown persistence errors intentionally require immediate retry, instead document that distinction explicitly near the catch handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cachekit/src/cache-core.ts`:
- Around line 810-818: Update the getWithSwr decode flow around decodeL1Entry so
a thrown decode error releases the refresh marker when swrResult.shouldRefresh
is true, matching the existing decoded === null branch. Preserve the error
propagation behavior while ensuring cancelRefresh(cacheKey) runs before the
error escapes.
In `@packages/cachekit/src/cache.encryption-l1.test.ts`:
- Around line 262-282: Update the test setup for the cache entry loaded by
load(7) to use a longer TTL that remains active throughout the 1400 ms delay and
ten-read loop. Replace the hard-coded originCalls < 5 assertion with a bound
derived from the loop’s read count, preserving the intended signal that origin
calls are far fewer than reads.
- Around line 150-154: Update the vi.waitFor assertion around l1Entry(cache,
key) to require that the refreshed entry is present and differs from
firstCiphertext, rather than only using not.toEqual. Preserve the subsequent
expectCiphertext validation and ensure expiration yielding null cannot satisfy
the wait.
In `@packages/cachekit/src/cache/background-refresh.test.ts`:
- Around line 151-171: Increase the TTL and corresponding sleep in the
background-refresh test around l1Cache.set and the stale read so the entry
remains stale while retaining a substantially larger expiry margin during
scheduleRefresh and vi.waitFor. Preserve the existing stale/SWR assertions and
refresh behavior.
- Around line 138-139: Remove the meaningless JSON.stringify assertion from the
test around l1Cache.get('key1'), since Uint8Array serialization cannot contain
the plaintext pattern; retain the existing ciphertext equality assertion or move
plaintext validation to the branch that returns plaintext.
In `@packages/cachekit/src/cache/background-refresh.ts`:
- Around line 39-43: Update the logger test callback in logger.test.ts to return
null via a Promise instead of Promise<void>, matching the PersistCallback<T>
return contract while preserving the existing test behavior.
---
Outside diff comments:
In `@packages/cachekit/src/cache/background-refresh.ts`:
- Around line 105-136: Align the catch path in the background refresh flow with
the failed-persistence behavior documented in the null branch: remove the
l1Cache.cancelRefresh call so the refresh marker can lapse and throttle retries.
If thrown persistence errors intentionally require immediate retry, instead
document that distinction explicitly near the catch handling.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 73e8cf1b-1c49-4a88-a5e8-88a9d8ca6cf8
📒 Files selected for processing (7)
packages/cachekit/README.mdpackages/cachekit/src/cache-core.tspackages/cachekit/src/cache.encryption-l1.test.tspackages/cachekit/src/cache/background-refresh.test.tspackages/cachekit/src/cache/background-refresh.tspackages/cachekit/src/l1/lru-cache.test.tspackages/cachekit/src/l1/lru-cache.ts
…terministic test key, deflake timing tests (LAB-238) - getWithSwr: a decode that rethrows (degradation off) now releases the refresh marker it holds, same as the null path — one release site via null-sentinel try/finally instead of a stranded slot for the marker TTL - test master key: sha256-derived fixture — deterministic runs without a scanner-matchable key literal - SWR timing tests: 4s TTL / 2.4s sleep clears the 1.8-2.2s jittered threshold on every draw with 1.6s expiry headroom; stampede bound derived from READS instead of a magic 5 - refresh wait asserts presence AND change (null after expiry no longer false-passes); dropped an unfalsifiable JSON.stringify assertion - logger test persist callback returns null per PersistCallback contract
This comment has been minimized.
This comment has been minimized.
|
@coderabbitai review |
|
@kody start-review |
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cachekit/src/cache/background-refresh.test.ts`:
- Around line 138-140: Update the background refresh test to use distinct
Uint8Array values for the initial stale L1 entry and the ciphertext returned by
secretPersist. After invoking completeRefresh(), wait until l1Cache.get('key1')
equals the refreshed ciphertext, so the assertion verifies that refresh actually
updates L1 rather than only checking object identity.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 5d3e0e4c-df2a-4f9a-b11c-3e523410e93e
📒 Files selected for processing (4)
packages/cachekit/src/cache-core.tspackages/cachekit/src/cache.encryption-l1.test.tspackages/cachekit/src/cache/background-refresh.test.tspackages/cachekit/src/logger.test.ts
…itter, awaited refresh promises (LAB-238) - ciphertext-residency test now uses distinct stale/refreshed ciphertexts and awaits the refresh promise (settles after the L1 update), so it fails if the refresh does not actually replace L1's entry - marker-hold test runs on a frozen clock with the jitter draw pinned to the midpoint: threshold is exactly 2s on a 4s TTL, no wall-clock races - compute-failure test now takes a real refresh marker first and asserts cancelRefresh released it — previously the cancel half was unfalsifiable
Kody Review CompleteGreat news! 🎉 Keep up the excellent work! 🚀 Kody Guide: Usage and ConfigurationInteracting with Kody
Current Kody ConfigurationReview OptionsThe following review options are enabled or disabled:
|
|
@coderabbitai review |
|
@kody start-review |
|
Resolves the README conflict from #104 (LAB-238, ciphertext in L1): both sides annotated the same encryption config comment with orthogonal facts — L1 zero-knowledge parity and the rotation pointer — so the resolution keeps both rather than picking a side.
…B-685) Merging main brought in #104 (LAB-238), which makes L1 hold ciphertext for a secure cache. That creates a path neither branch could test on its own: an L2 read under a previous key repopulates L1 with bytes the current key cannot open, so every subsequent L1 hit has to run the keyring loop again. The existing rotation tests all disable L1 — correct when they were written, since L1 then held plaintext and rotation could not reach it. Without keyring coverage on that path decodeL1Entry drops the entry and falls through to L2 on every read for the whole grace window: a silent L1 bypass under degradation, a throw on every old-key read without it. Verified by mutation — breaking the L1 decrypt path turns the single backend.get into two, and the test fails.
Closes LAB-238.
L1 held post-decrypt plaintext for encrypted caches. Any heap dump, core dump, or Node diagnostic report yielded the entire L1 working set in the clear for its full TTL — and that plaintext outlived the key zeroization in
close(), because L1 entries are held independently of tenant keys. It also broke the ratified zero-knowledge parity withcachekit-py(whoseL1Cachestores bytes and decrypts at read time) andcachekit-rs(ciphertext across every layer).The fix
All three population sites now store what L2 stores:
getEntry()setEntry()valuecompleteRefreshEvery L1 hit path —
get,wrap's SWR read,wrap's no-waitUntilfallback, andexists— decrypts and AAD-verifies against the cache key. An entry that fails to verify is dropped before anything else, so a poisoned L1 copy cannot outlive remediation of L2 (same orderingcachekit-pyuses).Non-encrypted caches keep storing decoded values, unchanged.
Two hazards that came with holding bytes in L1
estimateSizemeasured withJSON.stringify, which renders aUint8Arrayas{"0":171,…}— ~14× its real size. Unfixed, the first encrypted entry would have evicted most of L1.Bufferfrom the backend is a window onto a shared 8 KiB pool slab, so retaining one for the entry's TTL pins the slab. Copy when the view is narrower than its buffer. (cachekit-pyrefusesmemoryview/bytearrayinL1Cache.putfor the same reason.)Expert-panel review
Ran per ray's LAB-131 gate (any encryption/AAD diff in a cachekit repo) at critical stakes. Second commit applies the findings.
CRIT — a degraded L2 write became an origin stampede. Found independently by two reviewers and measured: with the backend down, 11 reads of an encrypted key drove 11 origin calls, against 1 for plaintext. Returning
nullfor "nothing storable" made every SWR refresh end incancelRefresh, which frees the refresh marker while leavingexpiresAtuntouched — so the entry stayed stale and every subsequent read re-armed the refresh. It fires exactly when the backend is already down, on exactly the encrypted caches that carry PII, and is invisible to any plaintext load test. Fixed by capturing the ciphertext the momentencrypt()produces it (before the write that may fail), so a degraded write still yields an L1 payload; the residual case where encryption itself fails leaves the marker to lapse viaSWR_REFRESH_MARKER_TTL_MS, throttling to one retry per key per minute.MAJ — a legitimately cached
nullread as a decrypt failure, so a secure cache holdingnullinvalidated and re-fetched a good entry on every hit (a billed miss per read on a metered backend).decodeL1Entrynow returns{ value } | null.MAJ —
exists()trusted L1 presence without decrypting; after a key rotation it reported present for entriesget()rejects and drops.MAJ — an AES-GCM tag failure is the canonical tamper signal and reached only a log line; now goes through
recordFailureso operators can alert.MAJ —
reliability.degradationgoverned the L2 decrypt failure but not the new L1 one;decodeL1Entrynow honours the same lever.Rejected, with reasons: a new
encryption.failClosedoption (new public API; the existing degradation lever covers it); copying L1 bytes unconditionally (every in-tree backend was verified to return an owned exact-size buffer, and the copy is real cost on large values); a synchronous fast path for plaintext L1 hits (unmeasured microtask against three duplicated ternaries on a crypto path); re-checking the L1 version token after the decryptawaitto close a sub-millisecond read-vs-invalidate window (needs new L1 API, not deterministically testable, and the authoritative delete already went to L2).Verification
main(+11 new).wire-format.protocol.test.tsare pre-existing and environmental — my runtime has no Rust toolchain, so I ran against the publishedcachekit-core-ts-linux-x64-gnu@0.1.2prebuilt while the workspace is on 0.1.3. They fail identically onmainbefore this branch. CI builds the crate and should be green.tsc --noEmit,eslint, andprettier --checkall clean.Note
Pushed with
--no-verify: the pre-push hook runsturbo type-check, which depends on building the Rust NAPI crate and cannot run withoutcargo. The gates it wraps were run directly and pass.Behaviour change worth flagging
An L1 hit on an encrypted cache now costs a decrypt + AAD verify rather than being free. That is the price of not keeping plaintext resident, and it is what the other two SDKs already pay. Noted in the package README.
Summary by CodeRabbit
New Features
Bug Fixes
Uint8Array, improving cache capacity calculations.nullvalues remain supported.