fix(responses): terminal refusals, caller-mismatched reasoning blobs, stable image tiers, and paginated-history containment

devlog/_plan/260914_l4_responses_media/010_roadmap.md

@@ -0,0 +1,73 @@
+# L4 — Responses terminal, reasoning payload and media
+
+Delivery lane R1-L4. One branch (`codex/260914-l4-responses-media`), one PR against `dev`.
+
+## Units
+
+| Unit | Issue | Write scope |
+|---|---|---|
+| U1 | #4469 reasoning `encrypted_content` not issued to this caller | `src/server/responses/core.ts`, `tests/responses/responses-opaque-blob-recovery.test.ts` |
+| U2 | #4312 Anthropic content_filter terminal reported as retryable | `src/adapters/anthropic.ts`, `tests/adapters/anthropic/anthropic-error-stop-reason.test.ts` |
+| U3 | #4532 image downscaling on append busts the prefix cache | `src/adapters/anthropic-image-codec.ts`, `src/adapters/anthropic-image-normalize.ts`, `tests/adapters/anthropic/anthropic-image-normalize.test.ts` |
+| U4 | #4311 paginated Codex history stops projecting | `src/codex/history-provider.ts`, `tests/codex-integration/codex-history-provider.test.ts` |
+| U5 | `structure/` SSOT sync for the source areas U1-U4 touch | `structure/*.md` |
+
+Write scopes are disjoint so concurrent subagents never share a file. No new test
+files: each regression lands in the existing domain test file, which keeps
+`scripts/test-layout/layout.json` and `tests/fixtures/test-layout-expected.json`
+untouched and avoids a shared-file collision.
+
+## U1 — #4469
+
+`isSelfIdentifiedOpaqueBlobRejection` in `core.ts` recognises three rejection
+identities: the nested `invalid_encrypted_content` code, one exact code-less
+ChatGPT "could not be verified" message, and two xAI `invalid-argument` decoder
+strings. The reported body is none of them — it is
+`invalid_request_error` carrying "reasoning \`encrypted_content\` was not issued to
+this caller". The detector returns false, `attemptOpaqueBlobRecovery` skips, and the
+caller sees a hard error for replay state the backend will never accept.
+
+Fix: add that identity to the detector so the existing recovery
+(`prepareOpaqueBlobRecovery` → rebuild → single replay) engages. Recovery machinery,
+the one-attempt guard, and the rejection memo are unchanged.
+
+## U2 — #4312
+
+`src/adapters/anthropic.ts` maps stop_reason `refusal`/`content_filter` to a
+`done` event with `stopReason: "content_filter"`. The bridge turns that into
+`response.incomplete` with no `retryable` field, so Codex reads a disconnected
+stream and retries a request that can never succeed.
+
+Fix: emit an explicit `incomplete` adapter event with `reason: "content_filter"`
+and `retryable: false`. The bridge's `incomplete` case already forwards
+`retryable` into `incomplete_details` — the same mechanism a prior fix used for
+`cyber_policy`. Partial output survives because the bridge emits the retained
+finished items. The provider's refusal stays explicit; nothing is rerouted and no
+false success is reported.
+
+## U3 — #4532
+
+`initialPosition(newestFirstIndex, bias)` derives an image's ladder position from
+its RELATIVE recency, so appending an image pushes every older image one slot
+toward the tail. Crossing a tier boundary re-encodes already-sent bytes and
+invalidates Anthropic's prompt prefix cache.
+
+Fix: pin the ladder position to image identity. A bounded store keyed by
+`hash:mediaType` records the position an image was last emitted at; later turns
+start from that recorded position instead of recomputing it from age. Positions
+only ever move down the ladder (aggregate demotion, 413 tier bias), so the store is
+monotonic and cannot flap. The age-tier pyramid still assigns a FIRST position, the
+byte budget still binds, and the 413 retry path is untouched.
+
+## U4 — #4311
+
+The external-append guard landed on `dev` in `7f76d736c2`: `updateSessionMeta`
+and `assertLegacyHistoryWritable` refuse a rollout whose record carries an
+`ordinal` or `history_mode: "paginated"`. Verify the refusal actually covers every
+append path this issue names and close the remaining gap; do not allocate an
+ordinal, and do not rewrite a live rollout.
+
+## Proof
+
+Hosted CI at the exact final head. Local suite, typecheck, install and GUI build are
+NOT RUN for this unit by explicit instruction.

src/adapters/anthropic-image-codec.ts

@@ -105,6 +105,62 @@ let cacheMetadataBytes = 0;
 let cacheSentinelEntries = 0;
 let encodeCalls = 0;
 
+/**
+ * Last-EMITTED ladder position per image identity (#4532). The age-tier pyramid in
+ * anthropic-image-normalize derives an image's start position from its recency rank
+ * within the current request, so appending one newer image shifts every older image's
+ * rank by one and can push it across a tier boundary — re-encoding it to different
+ * bytes and busting Anthropic's prompt prefix cache for the whole history. Pinning the
+ * start position to the image's own identity keeps already-emitted bytes stable across
+ * appends. Keys are the encode cache's identity minus the position suffix
+ * (`${hash}:${mediaType}`, see processAt). Entry-count cap with LRU eviction: a
+ * value is one small number, so a count bound is a byte bound (~4096 * ~50B worst
+ * case, far under the app-owned memory budget's headroom).
+ */
+const POSITION_STORE_MAX_ENTRIES = 4_096;
+const emittedPositions = new Map<string, number>();
+
+function positionKey(b64: string, mediaType: string): string {
+  return `${Bun.hash(b64).toString(36)}:${mediaType}`;
+}
+
+/**
+ * The position this image was last emitted at, if it has been normalized before.
+ * Reads refresh recency (insertion-order LRU, same discipline as the encode cache).
+ */
+export function recordedEmittedPosition(b64: string, mediaType: string): number | undefined {
+  const key = positionKey(b64, mediaType);
+  const pos = emittedPositions.get(key);
+  if (pos !== undefined) {
+    emittedPositions.delete(key);
+    emittedPositions.set(key, pos);
+  }
+  return pos;
+}
+
+/**
+ * Record the position an image actually ended at. Positions only ever move DOWN the
+ * ladder (first-pass tier, aggregate demotion, tierBias) — nothing raises an image
+ * back up — so the stored value is monotonically non-decreasing and cannot flap.
+ * That monotonicity is what makes identity-pinning safe: a stale entry can only make
+ * an image smaller than its fresh tier would, never larger.
+ */
+export function recordEmittedPosition(b64: string, mediaType: string, pos: number): void {
+  const key = positionKey(b64, mediaType);
+  const existing = emittedPositions.get(key);
+  if (existing !== undefined) {
+    emittedPositions.delete(key);
+    pos = Math.max(existing, pos);
+  }
+  while (emittedPositions.size + 1 > POSITION_STORE_MAX_ENTRIES) {
+    const oldest = emittedPositions.keys().next().value;
+    if (oldest === undefined) break;
+    emittedPositions.delete(oldest);
+  }
+  emittedPositions.set(key, pos);
+  enforceAppOwnedMemoryBudget();
+}
+
 function cacheEntry(key: string, value: CacheValue): CacheEntry {
   const keyBytes = cacheEncoder.encode(key).byteLength;
   const valueBytes = typeof value === "string"
@@ -180,6 +236,7 @@ export function getNormalizeStatsForTests(): {
 }
 export function resetNormalizeStateForTests(): void {
   cache.clear();
+  emittedPositions.clear();
   cacheBytes = 0;
   cacheMetadataBytes = 0;
   cacheSentinelEntries = 0;

src/adapters/anthropic-image-normalize.ts

@@ -28,6 +28,7 @@ export { getNormalizeStatsForTests, resetNormalizeStateForTests, setNormalizeCac
 export { anthropicImageNormalizeRetainedStoreSnapshot, evictOldestAnthropicImageNormalizeForBudget } from "./anthropic-image-codec";
 
 import { bunImageEncode, bunImageValidate, processAt, TERMINAL_POS, TIER0_COUNT, TIER1_COUNT } from "./anthropic-image-codec";
+import { recordedEmittedPosition, recordEmittedPosition } from "./anthropic-image-codec";
 import { IMAGE_NORMALIZE_CONCURRENCY, MAX_INPUT_BASE64_LENGTH, MAX_INPUT_PIXELS } from "./anthropic-image-codec";
 import type { NormalizeOptions } from "./anthropic-image-codec";
 
@@ -149,7 +150,23 @@ export async function normalizeImageTargets(targets: NormalizeTarget[], options:
         continue;
       }
       const sourceMedia = target.mediaType.toLowerCase();
-      const pos = initialPosition(newestFirstIndex, bias);
+      // #4532: pin the start position to the image's own identity. A never-seen
+      // image still gets the age-derived tier; a seen image resumes where it last
+      // EMITTED, so appending a newer image cannot re-encode history and bust
+      // Anthropic's prompt prefix cache. tierBias (413 retry) applies on top of
+      // either base and still clamps to TERMINAL_POS.
+      //
+      // Every read in this pass sees the store as it was BEFORE this request,
+      // because nothing is written until the whole request settles (see the
+      // record loop at the end). That is load-bearing, not incidental: an image
+      // can appear more than once in one history, and identity keying collapses
+      // those occurrences onto one entry. Writing during the pass let the OLDEST
+      // occurrence's tier win a race against the newest one and drag it down —
+      // 30 copies of a screenshot all landed on the oldest copy's tier instead of
+      // the age pyramid. Reading a fixed snapshot gives each occurrence its own
+      // age tier on a cold store, which is the pre-#4532 behaviour.
+      const recorded = recordedEmittedPosition(b64, sourceMedia);
+      const pos = Math.min((recorded ?? initialPosition(newestFirstIndex, 0)) + Math.max(0, bias), TERMINAL_POS);
       const result = await processAt(b64, pos, sourceMedia, encode, validate);
       if (result.kind === "failed") {
         target.drop(UNDECODABLE_TEXT);
@@ -218,6 +235,16 @@ export async function normalizeImageTargets(targets: NormalizeTarget[], options:
     entry.done = result.pos >= TERMINAL_POS;
   }
 
+  // #4532: commit the positions these images actually went out at, now that the
+  // first pass and the aggregate demotion loop have both settled. Written here
+  // rather than inline so every read above saw one consistent pre-request
+  // snapshot. `recordEmittedPosition` keeps the deeper of the stored and the new
+  // position, so a repeated image converges on the most-demoted tier it was ever
+  // emitted at and never moves back up.
+  for (const entry of entries) {
+    if (entry) recordEmittedPosition(entry.sourceB64, entry.sourceMedia, entry.pos);
+  }
+
   // Terminal overflow (050 audit round 1, blocker 3): with no downstream guard, drop
   // OLDEST targets until the sum fits.
   if (overflowAction === "drop") {

src/adapters/anthropic.ts

@@ -267,6 +267,33 @@ export function formatAnthropicErrorBody(status: number, _headers: Headers, payl
   return redactSecretString(detail).slice(0, 400);
 }
 
+function isAnthropicContentFilterStopReason(
+  stopReason: string | undefined,
+): stopReason is "refusal" | "content_filter" {
+  return stopReason === "refusal" || stopReason === "content_filter";
+}
+
+/**
+ * Anthropic `refusal` / `content_filter` is a permanent sampling decision, not a disconnect.
+ * Emitting `done` with that stopReason used to surface as `response.incomplete` without
+ * `retryable`, which Codex treats as a dropped stream and retries five times (#4312).
+ * The explicit incomplete event is what the bridge already forwards into
+ * `incomplete_details.retryable`. Usage is preserved: a filtered turn still consumed tokens.
+ * `max_tokens` stays a `done` so the client can continue from a legitimate truncation.
+ */
+function anthropicContentFilterIncomplete(
+  stopReason: string,
+  usage: OcxUsage | undefined,
+): Extract<AdapterEvent, { type: "incomplete" }> {
+  return {
+    type: "incomplete",
+    reason: "content_filter",
+    retryable: false,
+    message: `upstream ended the turn with stop_reason "${stopReason}"`,
+    usage,
+  };
+}
+
 function isAnthropicRecord(value: unknown): value is Record<string, unknown> {
   return value !== null && typeof value === "object" && !Array.isArray(value);
 }
@@ -1110,6 +1137,13 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti
           };
           return;
         }
+        // Refusal / content_filter must not look like a dropped stream. `done` with that
+        // stopReason becomes `response.incomplete` without `retryable`, and Codex retries
+        // the same refusal five times (#4312).
+        if (isAnthropicContentFilterStopReason(pendingStopReason)) {
+          yield anthropicContentFilterIncomplete(pendingStopReason, usageFromAnthropic(pendingUsage));
+          return;
+        }
         yield {
           type: "done",
           usage: usageFromAnthropic(pendingUsage),
@@ -1284,16 +1318,19 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti
             };
             return;
           }
-          const stopReason = pendingStopReason === "max_tokens"
-            ? "max_tokens"
-            : pendingStopReason === "refusal" || pendingStopReason === "content_filter"
-              ? "content_filter"
-              : pendingStopReason;
+          // Same rule as emitDone: refusal / content_filter is a permanent decision, not a
+          // disconnect. This branch bypasses emitDone, so the check has to be repeated here
+          // or the EOF route still emits `done` and Codex retries the refusal (#4312).
+          if (isAnthropicContentFilterStopReason(pendingStopReason)) {
+            emittedDone = true;
+            yield anthropicContentFilterIncomplete(pendingStopReason, usageFromAnthropic(pendingUsage));
+            return;
+          }
           emittedDone = true;
           yield {
             type: "done",
             usage: usageFromAnthropic(pendingUsage),
-            ...(stopReason ? { stopReason } : {}),
+            ...(pendingStopReason ? { stopReason: pendingStopReason } : {}),
           };
         } else if (provider.anthropicEofTolerance === true) {
           // AgentRouter-style compatibility profile (#658): the upstream can close the stream
@@ -1409,6 +1446,15 @@ export function createAnthropicAdapter(provider: OcxProviderConfig, cacheRetenti
         retainTranslatedEventBatch(events, budget);
         return events;
       }
+      // Same rule as the streaming terminals: a refusal is explicit and non-retryable.
+      // Leaving it as `done` hides `retryable: false` and Codex retries the filtered
+      // turn as if the stream dropped (#4312). Partial content above is already in
+      // `events`; the incomplete event carries usage the same way `done` did.
+      if (isAnthropicContentFilterStopReason(stopReason)) {
+        events.push(anthropicContentFilterIncomplete(stopReason, usageFromAnthropic(usage)));
+        retainTranslatedEventBatch(events, budget);
+        return events;
+      }
       events.push({
         type: "done",
         usage: usageFromAnthropic(usage),

src/codex/history-provider.ts

@@ -171,6 +171,41 @@ function readFirstRolloutLine(fd: number): string | null {
   return nlIndex === -1 ? null : collected.subarray(0, nlIndex).toString("utf8");
 }
 
+/**
+ * Bounded tail of complete JSONL lines, newest-last.
+ *
+ * Used to refuse a rollout that *became* paginated after a legacy first line
+ * (#4311). Line 1 can still look writable after a newer Codex migrates the
+ * thread in place, and the native projector then dies on the first
+ * out-of-sequence ordinal a legacy append introduces. Every record written
+ * after such a migration carries an ordinal, so the newest records are where
+ * the evidence is.
+ *
+ * One read of a fixed window from EOF, split once. An earlier draft grew the
+ * window chunk by chunk and re-decoded the accumulated buffer on every
+ * iteration, which is quadratic: a rollout whose only `session_meta` sits at
+ * the top would have decoded and split up to the whole window ~256 times. The
+ * window is a cap, not a target — it is not walked and it is not the file.
+ *
+ * Returns `null` only when the file cannot be measured, which the caller
+ * treats as an unreadable record rather than a writable rollout.
+ */
+const ROLLOUT_TAIL_WINDOW_BYTES = 1 << 20;
+
+function readRolloutTailCompleteLines(fd: number): string[] | null {
+  const size = Number(fstatSync(fd).size);
+  if (!Number.isFinite(size) || size < 0) return null;
+  if (size === 0) return [];
+  const start = Math.max(0, size - ROLLOUT_TAIL_WINDOW_BYTES);
+  const window = Buffer.alloc(size - start);
+  const read = readSync(fd, window, 0, window.length, start);
+  if (read === 0) return [];
+  const lines = window.subarray(0, read).toString("utf8").split("\n");
+  // Unless the window reached BOF, the first element starts mid-record (and
+  // possibly mid-codepoint), so it is not a complete line.
+  return (start === 0 ? lines : lines.slice(1)).filter(line => line.length > 0);
+}
+
 function planFirstLineProvider(firstLine: string, expectedId: string, provider: string): FirstLineProviderPlan {
   const meta = parseSessionMetaLine(firstLine);
   if (!meta || meta.record.payload.id !== expectedId) return { state: "unsafe" };
@@ -315,6 +350,26 @@ function assertLegacyHistoryWritable(path: string, heldFd?: number): void {
     const first = readFirstRolloutLine(fd);
     if (!first) throw new CodexHistoryIntegrityError("history_rollout_record_invalid");
     assertLegacyHistoryRecord(first);
+    // Line 1 is not enough: a newer Codex can migrate a live rollout in place,
+    // leaving the original session_meta and writing ordinals / history_mode only
+    // onto later records (#4311). The native projector then stops at the first
+    // cloned ordinal-0 append. Inspect a bounded window of the newest records
+    // and refuse before any mutation of the rollout, the row, or the manifest.
+    const tail = readRolloutTailCompleteLines(fd);
+    if (tail === null) throw new CodexHistoryIntegrityError("history_rollout_record_invalid");
+    if (tail.length === 0) return;
+    const last = tail[tail.length - 1];
+    if (!last) throw new CodexHistoryIntegrityError("history_rollout_record_invalid");
+    if (last !== first) assertLegacyHistoryRecord(last);
+    // Cheap filter: only re-parse tail lines that look paginated. Needed because
+    // a compensating append can make the last line look legacy again while an
+    // earlier-in-tail native conversion still carries ordinals (#4311).
+    for (const line of tail) {
+      if (line === first || line === last) continue;
+      if (line.includes("\"ordinal\"") || line.includes("\"history_mode\"")) {
+        assertLegacyHistoryRecord(line);
+      }
+    }
   } finally {
     if (heldFd === undefined) closeSync(fd);
   }