diff --git a/docs/OBSERVABILITY-BILLING.md b/docs/OBSERVABILITY-BILLING.md index ea53a463..87a35e48 100644 --- a/docs/OBSERVABILITY-BILLING.md +++ b/docs/OBSERVABILITY-BILLING.md @@ -266,6 +266,7 @@ These may support future packaging, abuse detection, and support investigations | consumer pull | — | delivered payload sum | — | pull | | consumer redelivery | — | delivered payload sum again | — | pull/redelivery | | settle/extend | — | 0 | — | queue op | +| queue wake-up notification attempt (proposed, [QUEUES-IDLE.md](QUEUES-IDLE.md) §3.6) | — | 0 | — | queue op, plus `queue_wakeup_attempts` | | empty long-poll/HEAD | — | 0 | — | read op | | watch notification | — | 0 in v1 | — | watch op | | fork creation | 0 | — | 0 owned bytes | fork | diff --git a/docs/QUEUES-DESIGN.md b/docs/QUEUES-DESIGN.md index d425038c..229b9117 100644 --- a/docs/QUEUES-DESIGN.md +++ b/docs/QUEUES-DESIGN.md @@ -15,7 +15,11 @@ shard, all proven by Q0 before Q1 starts. Revision 7 specifies the transitions a review of revision 6 found open: retirement into collection (a coverage rule), auxiliary-row lifetimes, failure into redrive, active queues into deletion, drains into later wake-ups, timers with no debt, and -capacity reservations. §15 maps every review point. Every client-visible +capacity reservations. §15 maps every review point. Revision 9 (2026-11-03, +proposed, not implemented) is [QUEUES-IDLE.md](QUEUES-IDLE.md): sweeps only +at real deadlines, a roster ordered by deadline, catch-up on access, parking +after 24 h without a request, dormant wake-ups and billed wake-up attempts +(D23-D28); it amends §6.8, §8.1, §9.2, §10 and §14. Every client-visible change becomes an edge record when it lands. ## 1. What we are building and why @@ -514,6 +518,14 @@ waits for a claim, not for the sweep, so after a sweep those terms look again a minute on. An active queue is thus swept when something can be due (about once per lease duration under load), not after every operation. +**Idle queues (revision 9, proposed).** [QUEUES-IDLE.md](QUEUES-IDLE.md) +removes the fallback of an hour out: a queue is on the roster only for a +real deadline, and deletion debt gets its own index for the collector (D23). +The roster is keyed by deadline, so a tick reads only due queues (D24). An +operation on an overdue queue runs one sweep step first (D25). A queue with +no request for 24 h is parked and swept at most once a day until nothing is +pending (D26). + **Timers are a maintenance obligation of their own.** A queue can have a scheduled message, a retry, a lease deadline or a dedup expiry with zero reclamation debt. What actually closes a shard today: fleet eviction or a @@ -582,7 +594,9 @@ from `availableAt`; `deliverAt` at most `maxAge` ahead (400 beyond). `maxAge` and `failureRetention` are fixed at creation, so availability order is expiry order; updatable retention (epochs or an expiry index) is later work. Expiry is keyless, costs no rows, and needs no consumer (§6.8). A queue -nobody consumes within `maxAge` loses what it did not take, counted. +nobody consumes within `maxAge` loses what it did not take, counted. A parked +queue's expired data is retired, and its gap counted, up to 24 h late (D26, +[QUEUES-IDLE.md](QUEUES-IDLE.md) §3.4). ### 8.2 The reclamation lifecycle @@ -936,6 +950,12 @@ subscription and at most 10 a second per project. The fan-out a receiver starts is its own choice, bounded by `maxInFlight`. A restart or move re-arms from the rows. A lost notification costs latency only. +**Dormancy and billing (revision 9, proposed).** A subscription with no +useful claim for 24 h goes dormant and is not notified until the next useful +claim or send (D27). Each attempted notification is a billed queue operation, +whatever the receiver answers (D28). See [QUEUES-IDLE.md](QUEUES-IDLE.md) +§3.5-§3.6. + **Secret custody.** The signing secret is `HMAC(cell wake key version k, project ‖ subscriptionId ‖ rotation)`, returned once at creation or rotation; the subscription records `k`. Rotating the cell key adds version `k+1` for @@ -970,7 +990,10 @@ worker** is the pilot's simplest option, with its idle cost stated. debt age, physical size, sweep lag, collection passes; cell-watch rules for failed growth, due work near `maxAge`, gaps, sweep lag and debt past its deadline, calibrated after the pilot. -- Usage: queue operations metered as today, plus read and stored bytes. +- Usage: queue operations metered as today, plus read and stored bytes, and + wake-up attempts (D28). +- Idle queues: `parkedAt`, `wakeup.dormantSince`, and the operator view's + parked and overdue counts ([QUEUES-IDLE.md](QUEUES-IDLE.md) §4). ## 11. HTTP surface @@ -1045,7 +1068,7 @@ The verification plan (TLA+, Kani, DSTs, SDK, Layer B, pilot) is in | D11 | Overload | A separate settle bucket; #184's append rules | | D12 | Batches | Atomic ≤ 1,000 messages within the body limit | | D13 | Dedup | 24 h default, 7 days maximum, never above retention | -| D14 | Billing | Cloudflare parity: $0.40 per million logical operations per message per 64 KB (send a write, each delivery a read, ack or discard a delete), one million a month included; empty claims free; bandwidth and standard retention included; live bytes count against capacity (a message's until retirement, dedup entries until their window ends) | +| D14 | Billing | Cloudflare parity: $0.40 per million logical operations per message per 64 KB (send a write, each delivery a read, ack or discard a delete, each attempted wake-up notification an operation under D28), one million a month included; empty claims free; bandwidth and standard retention included; live bytes count against capacity (a message's until retirement, dedup entries until their window ends) | | D15 | Sequencing | Q0 gates Q1; the first release gates the beta if queues are a beta requirement | | D16 | Wake-up egress | §9.2 egress policy with a versioned derived secret | | D17 | Queue storage | Queue pages in the shard database, never sealed into segment objects, under queue storage mode | @@ -1055,6 +1078,19 @@ The verification plan (TLA+, Kani, DSTs, SDK, Layer B, pilot) is in | D21 | Shard capacity | Reserve each queue's budgets at creation, no overcommit; smaller defaults with an opt-up, sized from Q0's per-shard bound; overcommit with a borrowing rule later | | D22 | Quiet-shard GC completion | Accept one residual L0 file per shard; a SlateDB patch only if Q0 shows it matters | +Revision 9 adds D23-D28, proposed in [QUEUES-IDLE.md](QUEUES-IDLE.md). The owner +approved their direction on 2026-11-03 and decided D28; the details of +D23-D27 are for review. + +| # | Decision | Recommendation | +|---|---|---| +| D23 | Sweep schedule | Only real deadlines put a queue on the roster; no fallback of an hour out; deletion debt in its own index | +| D24 | Roster order | Keyed by deadline, so a tick reads only due queues, the earliest first | +| D25 | Catch-up | An operation on an overdue queue runs one sweep step first, never more than one | +| D26 | Parking | After 24 h without a client request, swept at most once a day; expired data retired up to 24 h late | +| D27 | Wake-up dormancy | Dormant after 24 h without a useful claim; active again on the next useful claim or send | +| D28 | Wake-up billing | Each attempted notification is one billed queue operation at D14's rate, whatever the receiver answers; one the project budget refuses is not billed | + ## 15. Review responses Every review point of revisions 1-6, what each revision took, and the few diff --git a/docs/QUEUES-IDLE.md b/docs/QUEUES-IDLE.md new file mode 100644 index 00000000..91e9dd88 --- /dev/null +++ b/docs/QUEUES-IDLE.md @@ -0,0 +1,196 @@ +# Idle queues — design for review + +Status: **proposed 2026-11-03, not implemented.** This document is revision 9 +of [QUEUES-DESIGN.md](QUEUES-DESIGN.md) for queues that nobody uses. It +amends §6.8 (the maintenance sweep), §8.1 (retention), §9.2 (wake-up +notifications), §10 (telemetry) and §14 (decisions D23-D28). The changes to +those sections point here. On 2026-11-03 the owner approved the direction +(catch-up on access instead of hourly sweeps, parking after 24 h, wake-ups +stopped after 24 h) and decided D28, billing each wake-up attempt. The +details of D23-D27 are for review. Every client-visible change +becomes an edge record when it lands. + +## 1. The problem + +A queue costs nothing while it has no pending state and no deletion debt: +`nq_roster_target` (`src/shard/transaction/native_queue/mod.rs:209`) then +removes its roster row, and nothing schedules work for it. While it has +anything pending — messages, leases, retries, failures, dedup entries or +debt — it stays on its shard's roster. Its next sweep is then at its earliest +deadline, or an hour out when no deadline applies (`mod.rs:252`). Nothing +distinguishes a queue in use from one its owner has forgotten: + +| Queue | Sweeps until nothing is pending | +|---|---| +| A backlog nobody claims, default 4-day retention | about 96, one an hour, until the last message expires | +| Drained, with deletion debt below the pass floor | up to 24, one an hour, until the collector's daily pass (`MAX_INTERVAL_MS`, `src/shard/queue_maintenance.rs:29`) | +| A backlog with a wake-up subscription nobody answers | one sweep and one notification a minute: the backoff doubles from 1 s to its 60 s ceiling (`src/shard/transaction/native_queue/sweep.rs:279`) and stays there, about 1,440 a day and 5,760 over the default retention | + +Each notification is also a write, because the sweep records +`last_notified_at`, and an outbound HTTPS request with a 2 s timeout. + +What a sweep of a cold queue costs has not been measured. A cold touch has: +on 2026-11-02, with every request to the store taking 25 ms, a queue cell +served about 14 cold touches a second, each touch waiting on about three +store reads. A million queues touched once an hour needed 32 cells. A million +forgotten queues with a backlog are swept at the same rate, 278 a second. If +a sweep is as costly as a touch, they need comparable capacity while earning +nothing. + +§6.8 also lets the billing sweep open a closed shard for a due queue +deadline. Under that rule, every forgotten queue would reopen its shard once +an hour. `probe_debt` (`src/billing.rs:1667-1673`) does not yet count roster +deadlines, so today a closed shard's timers wait until something else opens +it. + +## 2. Findings in the current code + +This document fixes none of these; each is for the implementation. + +- **The sweeper reads only the first 4,096 roster rows.** `queue_roster` + (`src/shard/queue_maintenance.rs:293-318`) scans the roster from its start + every second and stops at `ROSTER_PAGE`. The roster is keyed by queue hash, + so a shard with more than 4,096 queues on its roster never sweeps the rest. + Their leases do not expire, their retention gaps are not counted, they are + not woken, and the collector (which reads the same page) never captures + their debt. Since their debt is never cleared, they never leave the roster. +- **The collector reads every roster queue's state row every 30 s.** + `capture_debt` (`queue_maintenance.rs:447`) gets the state row of each queue + on the page to sum its debt, at every `COLLECT_TICK`. The cost grows with + the roster, not with the debt. +- **Notifications are sent one at a time on the sweeper's tick.** + `sweep_due_queues` awaits each `notify` in turn. A receiver that takes its + full 2 s delays every queue after it on the page. +- **The per-project notification budget is not built.** §9.2 allows at most + 10 notifications a second per project. No code applies it. + +## 3. The rules + +### 3.1 Sweep only at a real deadline (D23) + +A queue is on the roster only for a deadline: a lease expiry, a retry cycle, +a failure expiry, a hard expiry, a dedup expiry, an untaken message's +retention end, or a wake-up. The fallback of an hour out goes. Deletion debt +alone puts no queue on the roster. + +Debt gets its own index instead: one row per queue with debt, +`DEBT_PREFIX ‖ hash → (generation, bytes, oldest_at)`, written in the batch +that changes the debt and deleted when it is cleared. The collector sums the +index, not the roster's state rows, so its tick reads the queues that have +debt and nothing else. + +### 3.2 The roster is ordered by deadline (D24) + +The roster key becomes `ROSTER_PREFIX ‖ deadline ‖ hash`, and the state row +keeps the queue's current roster key, so moving a deadline deletes the old +row in the same batch. A tick scans from the start of the prefix up to now, +at most `ROSTER_PAGE` rows, and so reads only due queues, the earliest first. +This removes the starvation of §2 and makes the tick's cost follow the work +that is due. + +### 3.3 Catch-up on access (D25) + +Every operation on a queue whose roster deadline has passed — send, claim, +settle, the message and group `GET`s, peek, the failed view, redrive, +republish — first runs one sweep step (`nq_sweep`, at most `SWEEP_STEP` rows +for each family) in its own transaction. The client then sees what a queue +swept on time would show. Claims already skip messages past their retention +(`src/shard/transaction/native_queue/claim.rs:198`). The lease step running +first gives an expired lease's message back to retry before the claim reads +the retry index. + +A step that fills its budget leaves the queue due one second on, as a sweep +does today, and the operation proceeds with what the step left. A request +never runs more than one step. A queue with a long overdue backlog catches up +over its next requests and the sweeper's ticks, never in one request. + +### 3.4 Parking after 24 h without a request (D26) + +A queue with no client request for `PARK_AFTER` (24 h) is **parked** at its +next sweep. That sweep records `parked_at` in the state row, and sets the +roster deadline to the queue's next real deadline rounded up to the next +24 h boundary after `parked_at`. A parked queue is swept at most once a day. +The sweep then retires everything due, in bounded steps a second apart while +the steps fill their budget. When nothing is pending, the queue leaves the +roster. + +The first client request after parking clears `parked_at`, runs the catch-up +of §3.3, and puts the queue back on its real deadlines. + +The rounding is the trade-off. Expired messages, expired failures and ended +dedup windows of a parked queue are retired, and stop counting as live bytes, +up to 24 h late. At D14's storage terms, that is at most one day of retained +bytes that should already have expired. The retention gap of a parked queue +is counted up to 24 h late as well. Under the default 4-day retention, a +forgotten queue with a backlog is swept about 5 times instead of about 96. +Parking needs no new timer: it is a deadline like any other. + +A parked queue on a closed shard opens it at most once a day for the billing +sweep's §6.8 service, not once an hour. + +### 3.5 Wake-ups go dormant after 24 h without a useful claim (D27) + +A sweep that would notify while `lastUsefulClaimAt` is more than 24 h old +(or, if no claim was ever useful, 24 h after the subscription was set) makes +the subscription **dormant** instead. It records `dormantSince` and sends +nothing. A dormant subscription adds no wake-up deadline to the roster. The +next useful claim, or the next send, makes it active again with its backoff +reset to 1 s. + +An unanswered subscription thus costs at most about 1,440 notifications, +over one day, instead of one a minute until its backlog expires. The receiver +contract (§9.2) gains a rule: a receiver that comes back after a day of +silence must claim once, or send, before it is woken again. + +### 3.6 Each wake-up attempt is billed (D28) + +Each wake-up notification the server attempts is one billed queue operation +at D14's rate, counted against the included million like any other. It is +counted whether the receiver acknowledges it, refuses it, times out or cannot +be reached. A notification the project's notification budget (§9.2, not yet +built) refuses is not attempted and is not billed. A subscription's cost is then visible to its +owner and bounded by §3.5: about 1,440 operations for a forgotten +subscription before it goes dormant. + +The meter is the sweeper's `notify` call, the one place a notification is +sent. It adds one to `queue_operations` for the queue's stream identity in +the read accumulator, and one to a separate `queue_wakeup_attempts` +dimension so usage can show what part of the operations were wake-ups. +[OBSERVABILITY-BILLING.md](OBSERVABILITY-BILLING.md) §5 has its row. + +## 4. Telemetry + +- The group's `GET` adds `parkedAt` and `wakeup.dormantSince`. For a parked + queue, the counters (`due`, `leased`, `retrying`, `failed`, `retentionGap`) + are as of its last sweep, and the response says so. +- `/operator/data.json` adds parked queues per shard, dormant subscriptions + per project, roster rows due and overdue per shard, and the oldest overdue + deadline. +- Usage adds `queue_wakeup_attempts` (§3.6). + +## 5. What has to be proven when it is built + +- A DST in which a queue with leases, retries, failures, dedup entries and an + untaken backlog is parked for 25 h of simulated time and then claimed. The + claim, its returned messages, the retention gap and the stored rows equal + those of the same queue swept on time. +- A shard with 10,000 queues on its roster serves every due deadline within + one rotation of the page, with none starved. +- Catch-up bounds: a claim on a queue with 10,000 expired leases runs one + step and returns, and the queue drains over the following ticks. +- Dormancy: an unanswered subscription stops after 24 h, and a send or a + useful claim makes it active again. +- Billing: every attempted notification is metered once, including timeouts + and refusals by the receiver. A notification refused by the project budget + is not metered. +- A cost run: 1,000,000 queues created, half holding one unclaimed message, + then left without a request for 26 h. It reports sweeps an hour, store + reads, memory and CPU before and after parking. + +## 6. Edge changes when it lands + +- Wake-up dormancy after 24 h without a useful claim (§3.5). +- Billing for each wake-up attempt (§3.6). +- `parkedAt`, and counters as of the last sweep, on a parked queue's group + `GET` (§4). +- Retirement of a parked queue's expired data up to 24 h late (§3.4). diff --git a/docs/README.md b/docs/README.md index ef1a87f0..f7327961 100644 --- a/docs/README.md +++ b/docs/README.md @@ -52,7 +52,8 @@ Do not read these whole. Find the heading first, then read only that range - [read-experiments/final-disposition.md](read-experiments/final-disposition.md): reads have one permanent production path, with no runtime switches. - [CONFORMANCE.md](../CONFORMANCE.md): the pinned Durable Streams suite. [sdk/README.md](../sdk/README.md): the TypeScript SDK; [sdk/CHANGELOG.md](../sdk/CHANGELOG.md): its release notes, packed with it. - [AUTHENTICATING.md](AUTHENTICATING.md): the public beta's customer reference for authentication: the credential, its exchange at the interim issuer (`POST /v1/token/streams`), token lifetime and refresh, the `Streams-Encryption-Key` header, the exchange's and the server's auth refusals with their retry rule, revocation and rotation. BETA-REFERENCE links it. -- [QUEUES-DESIGN.md](QUEUES-DESIGN.md): **revision 8 (2026-10-23); architecture and Q0-first sequence approved**: native queues over streams of kind `queue` (queue storage mode in the shard database with guarded absorption consumers, a four-stage reclamation lifecycle with deletion debt, a coverage rule over named SlateDB primitives, a trigger floor and a quiet-shard GC completion rule, accounting by state lifetime, capacity reservations, a queue-wide availability index under a durable queue clock, the `default` group with fenced leases and per-cycle deadlines, durable failed deliveries with budgeted redrive and republish, a keyless maintenance sweep with timers served through the billing sweep's rotation, a `deleting` fence, `/v1/queues` routes, the SDK runner and wake-up notifications; named and ordered groups later) and owner decisions D1-D22. +- [QUEUES-DESIGN.md](QUEUES-DESIGN.md): **revision 8 (2026-10-23); architecture and Q0-first sequence approved**: native queues over streams of kind `queue` (queue storage mode in the shard database with guarded absorption consumers, a four-stage reclamation lifecycle with deletion debt, a coverage rule over named SlateDB primitives, a trigger floor and a quiet-shard GC completion rule, accounting by state lifetime, capacity reservations, a queue-wide availability index under a durable queue clock, the `default` group with fenced leases and per-cycle deadlines, durable failed deliveries with budgeted redrive and republish, a keyless maintenance sweep with timers served through the billing sweep's rotation, a `deleting` fence, `/v1/queues` routes, the SDK runner and wake-up notifications; named and ordered groups later) and owner decisions D1-D22. Revision 9 (proposed) is QUEUES-IDLE below. +- [QUEUES-IDLE.md](QUEUES-IDLE.md): **revision 9 of the queues design, proposed 2026-11-03, not implemented**: what a forgotten queue costs today (an hourly sweep while anything is pending, a wake-up a minute while a backlog goes unclaimed), three findings in the sweeper and collector (a roster page that starves queues past 4,096, a collector that re-reads every roster state row, serial notifications), and the rules: sweeps only at real deadlines with a debt index, a roster ordered by deadline, catch-up on access, parking after 24 h, dormant wake-ups and billed wake-up attempts (D23-D28). - [QUEUES-PLAN.md](QUEUES-PLAN.md): the queues build and verification plan: packages Q0-Q6 (85-118 days for the first release), Q0's storage-feasibility experiment with its build order, cases and go/no-go gates, and the Q5 verification. - [QUEUES-DESIGN-REVIEWS.md](QUEUES-DESIGN-REVIEWS.md): the review record for the queues design: each review's points of revisions 1-6 and where the design answers or declines them. - [STREAMS-COST-LEVERS-2026-10.md](STREAMS-COST-LEVERS-2026-10.md): Streams at Cloudflare K2's price: the four-profile benchmark and modelled cost (Fable 5.1's report, kept outside the tree), the screening of the first cost levers on the launch posture (live-window scan cache, tail ring, `_ops_metrics` at 60 s landed; history fan-in deferred) and the levers that decide the goal next.