docs: exploratory testing pass for persistence under a full disk (2026-10-10)

docs/exploratory-testing/2026-10-10-queue/2026-10-10-report.md

@@ -0,0 +1,125 @@
+# Queue exploratory-testing report — 2026-10-10
+
+## Outcome
+
+Two confirmed bugs, both in `--persist` mode when the `PERSIST` volume runs out
+of space. Both cause loss of items the API acknowledged with `200`:
+
+- [#160](https://github.com/jonbaldie/queue/issues/160): short writes are
+  treated as success and corrupt `persist.dat`. Acknowledged items are lost after
+  a crash restart, or by a graceful shutdown that exits 0.
+- [#161](https://github.com/jonbaldie/queue/issues/161): enqueue and dequeue
+  return `500` when the persist write fails, but the in-memory queue has already
+  changed. Rejected items are delivered later, and dequeued items vanish.
+
+The ordinary worker flow and Docker volume recreation behaved as documented.
+
+## Setup and evidence
+
+- Build: `main` at `1c2c3e9`. `deno cache --frozen main.ts`, then
+  `deno compile --frozen` with the Dockerfile's permission flags, output to the
+  gitignored `target/queue`. Docker image `qx20261010-queue:local` built from the
+  repository `Dockerfile`.
+- Environment: Deno 2.7.6 (`aarch64-apple-darwin`), Docker 29.4.0 (Linux VM).
+- Small volumes: a 2 MiB HFS+ RAM disk
+  (`hdiutil attach -nomount ram://4096` + `diskutil erasevolume HFS+`) on macOS,
+  and a 2 MiB `tmpfs` named volume in Docker. This is an intervention that
+  simulates a full disk. Nothing else was changed.
+- Evidence (this directory):
+  - [run-log.txt](./run-log.txt): build commit and dependency install.
+  - [journey1_workers.ts](./journey1_workers.ts),
+    [journey1-transcript.txt](./journey1-transcript.txt)
+  - [journey2_diskfull.sh](./journey2_diskfull.sh),
+    [journey2_snapshot_nearfull.sh](./journey2_snapshot_nearfull.sh),
+    [journey2_enospc_500.sh](./journey2_enospc_500.sh),
+    [journey2-transcript.txt](./journey2-transcript.txt)
+  - [journey3_docker.sh](./journey3_docker.sh),
+    [journey3-transcript.txt](./journey3-transcript.txt)
+
+## Journeys
+
+### 1. Publishers and polling workers (compiled binary, default config)
+
+Goal: 3 producers enqueue 120 jobs while 3 polling workers drain the queue.
+Every job is delivered exactly once, in order per producer.
+
+- Default config: 99 `200`, 94 `429`. Only 41 of 120 jobs were enqueued and
+  delivered. Producers on the same host share the default 100 requests/minute
+  per-IP budget, and polling consumers spend it too.
+- Variation, `RATE_LIMIT_REQUESTS=100000`: 120 sent, 120 received, 0 duplicates,
+  per-producer order kept, final length 0.
+- Requests with the wrong token also use up the budget (they get `429` before
+  auth runs). The `429` response has no `Retry-After` header.
+
+### 2. Persistence when the `PERSIST` volume fills up (compiled binary)
+
+Goal: items acknowledged with `200` under `--persist` survive a restart. A full
+disk should cause visible failures, not silent loss.
+
+- Crash after a disk-full episode: 3 items acknowledged, 1 recovered after
+  SIGKILL (2/2 runs). **Confirmed, #160.**
+- Variation, graceful SIGTERM after the same episode: all 3 recovered (1/1 run).
+  The snapshot is rebuilt from memory.
+- Graceful SIGTERM with about 100 KiB free: exit 0 and `Goodbye!`, but
+  `persist.dat` drops from 200123 to 4096 bytes. 1 of 2 items recovered (2/2
+  runs). **Confirmed, #160.**
+- Disk completely full: `500` enqueues still add the item (length 5 when 2 were
+  accepted). Each `500` dequeue removes an item (length 5 → 4 → 3). After space
+  returns, the 3 rejected items are delivered and the 2 accepted ones never are
+  (2/2 runs). **Confirmed, #161.**
+
+### 3. Docker deployment per README
+
+Goal: the README's Docker `--persist` command with a named volume keeps items
+across container *recreation*.
+
+- 3a: enqueue 2 items, dequeue 1, `docker rm -f`, recreate with the same volume.
+  Length 1, the remaining item was delivered, then `204`. Pass.
+- 3b: the default `CMD` (no `--persist`) with `-v ...:/data` loses items across
+  restart. This matches "Persistency is opt-in", so it is not a bug. See the
+  usability notes.
+- 3c: graceful `docker stop` with a nearly full 2 MiB tmpfs `/data`. Exit 0, and
+  `persist.dat` drops from 200123 to 102400 bytes. 1 of 2 items recovered. This
+  reproduces #160 on Linux.
+
+## Rejected candidates
+
+- In runs 1–2 of `journey2_enospc_500.sh`, `/length` returned `Internal Server
+  Error` after a failed dequeue. Cause: the harness wrote the server log onto the
+  full volume, so the log write failed. Runs 3–4 kept the log elsewhere and
+  `/length` worked. This is a harness artifact and is annotated in the transcript.
+
+## Unresolved
+
+- None.
+
+## Usability observations
+
+- The README does not mention the rate limit (`RATE_LIMIT_REQUESTS`, default 100
+  requests/minute per client IP), `QUEUE_DEPTH_LIMIT`, `QUEUE_COUNT_LIMIT`,
+  `/peek`, or `/queues`. A single-host setup that follows the README's
+  publisher/subscriber advice hits `429` within a minute (journey 1).
+  *Suggestion:* document these in the README and add `Retry-After` to `429`.
+- The image sets `PERSIST=/data/` and declares a `/data` volume, but the default
+  `CMD` has no `--persist`. Mounting a volume therefore persists nothing
+  (journey 3b). The README's "Mount a named volume there" sentence does not
+  repeat that `--persist` is required. *Suggestion:* say so in that sentence.
+- The README calls `persist.dat` a "binary log", but it is newline-delimited JSON.
+
+## Limitations and unexplored areas
+
+- Disk-full was simulated with RAM disks and tmpfs, not with a real full disk.
+  Short-write sizes depend on the filesystem (4096 B on HFS+, 102400 B on tmpfs).
+- Docker Desktop port mapping makes host clients share one source IP. How that
+  affects per-IP rate limits behind a real NAT was not measured.
+- Not explored this pass: concurrent server instances sharing one `PERSIST`
+  directory, and signal storms during shutdown.
+
+## Cleanup
+
+- Removed Docker containers `qx20261010-c` and `qx20261010-c-holder`, volumes
+  `qx20261010-data` and `qx20261010-tmpfs`, and image `qx20261010-queue:local`.
+- Detached the RAM disk and deleted `target/queue` and the scratch server logs
+  in `TMPDIR`.
+- Docker's shared build cache was not pruned. Pruning it would also remove
+  cache entries that belong to other projects.

docs/exploratory-testing/2026-10-10-queue/journey1-transcript.txt

@@ -0,0 +1,17 @@
+## journey1 default rate limit
+{"rateLimit":"default","statuses":{"200":99,"204":1,"429":94},"sent":120,"received":41,"unique":41,"duplicates":0,"perProducerOrdered":true,"finalLength":"Too many requests"}
+## journey1 RATE_LIMIT_REQUESTS=100000
+{"rateLimit":"100000","statuses":{"200":240,"204":34},"sent":120,"received":120,"unique":120,"duplicates":0,"perProducerOrdered":true,"finalLength":"0"}
+## 429 response headers (RATE_LIMIT_REQUESTS=2)
+req 1 -> 200
+req 2 -> 200
+req 3 -> 429
+HTTP/1.1 429 Too Many Requests
+content-type: text/plain;charset=UTF-8
+vary: Accept-Encoding
+content-length: 17
+date: Sat, 10 Oct 2026 00:40:46 GMT
+
+Too many requests
+## unauthenticated requests consume the same budget
+wrong token -> 429

docs/exploratory-testing/2026-10-10-queue/journey1_workers.ts

@@ -0,0 +1,87 @@
+// Journey 1: producers and polling workers against the compiled binary.
+// Usage: deno run -A journey1_workers.ts <binary> <port> [RATE_LIMIT_REQUESTS]
+const [bin, portArg, rate] = Deno.args;
+const port = Number(portArg);
+const token = "qx-token-20261010";
+const base = `http://127.0.0.1:${port}`;
+const auth = { Authorization: `Bearer ${token}` };
+
+const env: Record<string, string> = { QUEUE_API_TOKEN: token, PORT: String(port) };
+if (rate) env.RATE_LIMIT_REQUESTS = rate;
+const child = new Deno.Command(bin, { env, stdout: "null", stderr: "piped" }).spawn();
+
+async function waitReady() {
+    for (let i = 0; i < 100; i++) {
+        try {
+            const r = await fetch(`${base}/health`);
+            await r.body?.cancel();
+            if (r.ok) return;
+        } catch { /* not up yet */ }
+        await new Promise((r) => setTimeout(r, 100));
+    }
+    throw new Error("server not ready");
+}
+
+const statuses: Record<number, number> = {};
+const count = (s: number) => statuses[s] = (statuses[s] ?? 0) + 1;
+
+async function produce(id: number, n: number) {
+    for (let i = 0; i < n; i++) {
+        const r = await fetch(`${base}/enqueue/jobs`, {
+            method: "POST",
+            headers: { ...auth, "Content-Type": "application/json" },
+            body: JSON.stringify({ payload: { producer: id, seq: i } }),
+        });
+        count(r.status);
+        await r.body?.cancel();
+    }
+}
+
+const received: { producer: number; seq: number }[] = [];
+async function consume(stopAfterEmpty: number) {
+    let empties = 0;
+    while (empties < stopAfterEmpty) {
+        const r = await fetch(`${base}/dequeue/jobs`, { headers: auth });
+        count(r.status);
+        if (r.status === 200) {
+            received.push(await r.json());
+            empties = 0;
+        } else {
+            await r.body?.cancel();
+            empties++;
+            await new Promise((res) => setTimeout(res, 20));
+        }
+    }
+}
+
+try {
+    await waitReady();
+    const PRODUCERS = 3, PER = 40;
+    await Promise.all([
+        ...Array.from({ length: PRODUCERS }, (_, i) => produce(i, PER)),
+        consume(10), consume(10), consume(10),
+    ]);
+    // Final drain in case consumers stopped early.
+    await consume(3);
+    const len = await (await fetch(`${base}/length/jobs`, { headers: auth })).text();
+    const keys = received.map((m) => `${m.producer}:${m.seq}`);
+    const unique = new Set(keys);
+    let perProducerOrdered = true;
+    for (let p = 0; p < PRODUCERS; p++) {
+        const seqs = received.filter((m) => m.producer === p).map((m) => m.seq);
+        if (seqs.some((s, i) => i > 0 && s < seqs[i - 1])) perProducerOrdered = false;
+    }
+    console.log(JSON.stringify({
+        rateLimit: rate ?? "default",
+        statuses,
+        sent: PRODUCERS * PER,
+        received: received.length,
+        unique: unique.size,
+        duplicates: received.length - unique.size,
+        perProducerOrdered,
+        finalLength: len,
+    }));
+} finally {
+    child.kill("SIGTERM");
+    await child.status;
+}