Repository navigation
Commit 94b33d2
authored
perf(outbox): resume each retention batch where the previous run's batch ended (#9446)
* perf(outbox): resume each retention batch where the previous run's batch ended
The completed-event prune deletes one oldest-first batch per type per run.
Every batch started its scan of outbox_event_type_created_idx at the oldest
row of its type, and the rows earlier batches deleted stay in that index until
vacuum removes them, so while a large backlog drains each batch first walks
everything deleted since the last vacuum. That walk grows with each run: on a
read replica during a large drain it took about 0.6s before the first live
row, and a batch can reach the 10s statement timeout.
Each batch now resumes after the (created_at, id) of the last row the previous
run's batch deleted, the same shape as the idempotency sweep's resume. Batches
are ordered by (created_at, id) so ties on created_at split exactly, the cursor
keeps PostgreSQL's microsecond text form, and the row comparison becomes a
created_at >= bound on the index, with the full comparison as a filter.
Resuming from a cursor, the same batch took about 20ms.
The prune runs once per processor run, so the cursor has to outlive the run;
it is kept in Redis as a hint, one MGET before the batches and one pipeline
after. A row the pass went by before it completed, or while an overlapping run
held its lock, is covered by the next pass: the cursor is cleared when a batch
comes up short of the cutoff, and a pass older than an hour starts over from
the oldest row. Without Redis, or with an unreadable cursor, a batch starts
from the oldest row as before. Batch size, timeout, cadence and which rows are
eligible are unchanged.
Autopsy: the retention suite asserted which rows each run deleted, which a scan
from the oldest row gets right, so nothing observed where a batch started. The
new Redis cases complete a row behind the cursor between runs: resuming leaves
it for the next pass and the old scan takes it at once, so the first case fails
on the previous code. The others split one creation time across batches with
pending rows mixed in, and restart a pass that is over an hour old or whose
cursor is unreadable.
* fix(outbox): start a pass from the oldest row when its stored cursor names no real time
A cursor whose JSON parsed but whose creation time was not a calendar time
reached the batch's ::timestamp cast and failed the run, which stopped every
later type's prune until the cursor expired. The cursor's creation time is now
checked as PostgreSQL's ISO text with a calendar round trip, so such a cursor
is dropped like an unreadable one and the pass starts from the oldest row.1 parent 8948b54 commit 94b33d2
2 files changed
Lines changed: 277 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| 11 | + | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
15 | 17 | | |
| 18 | + | |
16 | 19 | | |
17 | 20 | | |
18 | 21 | | |
| |||
24 | 27 | | |
25 | 28 | | |
26 | 29 | | |
| 30 | + | |
27 | 31 | | |
28 | 32 | | |
29 | 33 | | |
| |||
34 | 38 | | |
35 | 39 | | |
36 | 40 | | |
| 41 | + | |
37 | 42 | | |
38 | 43 | | |
39 | 44 | | |
| |||
49 | 54 | | |
50 | 55 | | |
51 | 56 | | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
52 | 61 | | |
53 | 62 | | |
54 | 63 | | |
55 | 64 | | |
56 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
57 | 71 | | |
58 | 72 | | |
59 | 73 | | |
| 74 | + | |
| 75 | + | |
60 | 76 | | |
61 | 77 | | |
62 | 78 | | |
| |||
65 | 81 | | |
66 | 82 | | |
67 | 83 | | |
| 84 | + | |
68 | 85 | | |
69 | 86 | | |
70 | 87 | | |
71 | 88 | | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
72 | 96 | | |
73 | 97 | | |
74 | 98 | | |
| |||
154 | 178 | | |
155 | 179 | | |
156 | 180 | | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
157 | 290 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
| 6 | + | |
5 | 7 | | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
| 11 | + | |
9 | 12 | | |
10 | 13 | | |
11 | 14 | | |
| |||
57 | 60 | | |
58 | 61 | | |
59 | 62 | | |
60 | | - | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
61 | 161 | | |
62 | 162 | | |
63 | | - | |
64 | | - | |
| 163 | + | |
| 164 | + | |
65 | 165 | | |
66 | 166 | | |
67 | 167 | | |
68 | 168 | | |
69 | 169 | | |
70 | | - | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
71 | 174 | | |
72 | 175 | | |
73 | | - | |
| 176 | + | |
74 | 177 | | |
75 | 178 | | |
76 | | - | |
77 | | - | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
78 | 191 | | |
79 | 192 | | |
80 | 193 | | |
| |||
83 | 196 | | |
84 | 197 | | |
85 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
86 | 205 | | |
87 | 206 | | |
88 | 207 | | |
| 208 | + | |
| 209 | + | |
89 | 210 | | |
90 | | - | |
91 | | - | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
92 | 226 | | |
93 | 227 | | |
94 | 228 | | |
| |||
0 commit comments