[sorenbs]

Documentation only. This proposes how queues that nobody uses stop costing sweeps and wake-ups, and records the decision to bill each wake-up attempt. Nothing is implemented here.

Why

While a queue has anything pending, it stays on its shard's roster. It is swept at its next deadline, or an hour out when no deadline applies (nq_roster_target). Nothing tells a queue in use from a forgotten one:

• A backlog nobody claims is swept about 96 times under the default 4-day retention.
• A drained queue is swept hourly for up to a day while its deletion debt waits for the collector's daily pass.
• An unanswered wake-up subscription is notified once a minute until its backlog expires. The backoff stops growing at 60 s, so that is about 1,440 notifications a day. Each one is also a state write and an outbound HTTPS request.

On 2026-11-02, a queue cell with 25 ms store latency served about 14 cold touches a second. A million queues touched once an hour needed 32 cells. A million forgotten queues swept once an hour are the same rate of cold work, and they earn nothing.

What changes

New: docs/QUEUES-IDLE.md, revision 9 of the queues design, with decisions D23-D28:

• D23: a queue is on the roster only for a real deadline. Deletion debt moves to its own index, which the collector reads.
• D24: the roster is keyed by deadline, so a tick reads only the queues that are due.
• D25: an operation on an overdue queue first runs one sweep step, and never more than one.
• D26: a queue with no request for 24 h is parked and swept at most once a day until nothing is pending. Its expired data is retired up to 24 h late.
• D27: a wake-up subscription goes dormant after 24 h without a useful claim. The next useful claim or send makes it active again.
• D28: each attempted wake-up notification is one billed queue operation at D14's rate, whatever the receiver answers. One that the project budget refuses is not billed.

Amended:

• QUEUES-DESIGN.md: §6.8, §8.1, §9.2 and §10 point to the new doc. §14 adds D23-D28, and D14 counts wake-up attempts.
• OBSERVABILITY-BILLING.md §5: a row for wake-up attempts.
• docs/README.md: an entry for the new doc.

Findings in the current code (not fixed here)

• queue_roster reads only the first 4,096 roster rows, ordered by hash, on every tick. A shard with more than 4,096 queues on its roster never sweeps, wakes or collects the debt of the rest.
• capture_debt reads the state row of every queue on the roster page every 30 s.
• sweep_due_queues awaits each notification in turn, so a receiver that takes its full 2 s delays every queue after it.
• The per-project notification budget in §9.2 (10 a second) is not in the code.
• §6.8 says probe_debt counts the roster's earliest deadline. The code doesn't yet, so a closed shard's queue timers wait until something else opens it.

Decisions

The owner approved the direction on 2026-11-03: catch-up on access instead of hourly sweeps, parking after 24 h, and stopping wake-ups after 24 h. The owner also decided D28. The details of D23-D27 are for review.

Breaking changes

None now. Once implemented, wake-up dormancy, the billing of wake-up attempts, parkedAt, and the late retirement of a parked queue's data are each client-visible. Each becomes an edge record when it lands.

🤖 Generated with Claude Code