Skip to content

Commit 3230a64

Browse files
committed
fix(run-store): let a resolved sub-redis-only run fall back to Postgres under a global redis-only
The read fallback gate refused Postgres whenever the global dial was redis-only, before checking the run's own resolved mode. Postgres suppression is org-scoped, so an org held at dual-write or redis-read while the global dial has advanced to redis-only keeps its whole log in Postgres, yet every read of its runs threw and the run hung with its data sitting readable in Postgres. Check the resolved per-run mode first: a run resolved below redis-only falls back to Postgres, a run resolved to redis-only still refuses, and only an unresolved run is governed by the global short-circuit. Two tests cover both directions and fail on the old ordering.
1 parent 43da771 commit 3230a64

2 files changed

Lines changed: 41 additions & 9 deletions

File tree

‎internal-packages/run-store/src/taskRunExecutionSnapshotStore.redisOnlyFallback.test.ts‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -438,3 +438,33 @@ describe("redis-only fallback resolves a cold run→org cache authoritatively",
438438
expect(h.delegateTouched).toContain("findLatestExecutionSnapshot");
439439
});
440440
});
441+
442+
// An org held BELOW a global redis-only (dual-write / redis-read) keeps all its rows in Postgres,
443+
// because suppression is org-scoped. The read gate must serve those rows, not refuse them as though
444+
// the global dial spoke for this run. A resolved per-run answer wins over the global short-circuit.
445+
describe("a run held below a global redis-only reads from Postgres, never strands", () => {
446+
it("resolves to dual-write while the global dial is redis-only, and falls back to Postgres", async () => {
447+
const h = harness({
448+
globalMode: "redis-only",
449+
anyOrgRedisOnly: () => true,
450+
readModeFor: (runId) => (runId === "run_held" ? "dual-write" : undefined),
451+
});
452+
453+
const result = await h.decorated.findLatestExecutionSnapshot("run_held", undefined, "env_held");
454+
expect(result).toEqual({ id: "pg_fallback" });
455+
expect(h.delegateTouched).toContain("findLatestExecutionSnapshot");
456+
});
457+
458+
it("still refuses Postgres for a run that itself resolves to redis-only under a redis-only dial", async () => {
459+
const h = harness({
460+
globalMode: "redis-only",
461+
anyOrgRedisOnly: () => true,
462+
readModeFor: (runId) => (runId === "run_ro" ? "redis-only" : undefined),
463+
});
464+
465+
await expect(
466+
h.decorated.findLatestExecutionSnapshot("run_ro", undefined, "env_ro")
467+
).rejects.toThrow("redis brownout");
468+
expect(h.delegateTouched).not.toContain("findLatestExecutionSnapshot");
469+
});
470+
});

‎internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1124,21 +1124,23 @@ export class TaskRunExecutionSnapshotStore extends DelegatingRunStore {
11241124
* at `redis-only` throws while everyone else still falls back.
11251125
*/
11261126
async #fallbackAllowed(runId?: string, environmentId?: string): Promise<boolean> {
1127-
// The global dial preserves today's behaviour exactly: at redis-only Postgres holds nothing.
1128-
if (this.mode === "redis-only") return false;
1129-
1130-
// The run's own org from the sync cache, checked BEFORE the census: a run whose org resolves to
1131-
// redis-only must throw even when anyOrgRedisOnly (a separate, independently-lagging source) has
1132-
// not yet caught the enable, or the writing-process window strands it. A concrete non-`redis-only`
1133-
// mode falls back normally, even when some OTHER org is `redis-only`.
1127+
// A RESOLVED per-run answer is authoritative and wins over the global dial. Postgres suppression
1128+
// is org-scoped, so an org resolved BELOW redis-only has its whole log in Postgres even when the
1129+
// GLOBAL dial has advanced to redis-only; refusing the fallback there would strand a run whose
1130+
// rows are sitting readable in Postgres. A run resolved TO redis-only still refuses (even when
1131+
// anyOrgRedisOnly has not caught the enable); a concrete non-`redis-only` mode falls back, even
1132+
// when a DIFFERENT org is `redis-only`. So this check precedes the global short-circuit below.
11341133
if (runId !== undefined) {
11351134
const resolved = this.modeResolver?.readModeFor?.(runId, environmentId);
11361135
if (resolved === "redis-only") return false;
11371136
if (resolved !== undefined) return true;
11381137
}
11391138

1140-
// Org unresolved (cold run→org cache, or no runId). If no org is `redis-only`, this run cannot
1141-
// be either, so Postgres is a valid answer.
1139+
// Org unresolved (cold run→org cache, or no runId). At a global redis-only we cannot prove
1140+
// Postgres holds this run, so refuse rather than serve an empty history as though it were real.
1141+
if (this.mode === "redis-only") return false;
1142+
1143+
// If no org is `redis-only`, this run cannot be either, so Postgres is a valid answer.
11421144
if (this.modeResolver?.anyOrgRedisOnly?.() !== true) return true;
11431145

11441146
// Some org IS redis-only and the sync cache cannot tell if it is this one. Resolve the run's org

0 commit comments

Comments
 (0)