Repository navigation
Commit a410449
committed
fix(run-store,webapp): fix a run's snapshot store at birth, not from the live dial
A run's storage residency was re-derived from the live per-org dial on every
write and read, but a run's store is fixed when it is born. As the dial drifted,
long-lived runs, runs born in the enrollment window, and runs read while an org
sat at redis-only were routed to a store that did not hold them, stranding or
losing their execution state.
Residency is now a per-run decision recorded at birth and remembered per process
(a bounded LRU on the shared Redis store, monotonic toward redis-only, learned
for free from births, append replies, and a one-off keyspace probe, never a
database read). A redis-only-born run suppresses its Postgres snapshot rows and
refuses a Postgres fallback for life; every other run, including all runs born
before the cutover, writes and reads Postgres for life whatever the dial later
does. A halt forces Postgres writes back on so a redis-only run never writes its
transition nowhere. Reads and the fallback gate key on the run's regime, not the
global dial.
Removes the per-read run to org control-plane database read that could deadlock
dequeues for split-database runs once any org reached redis-only, along with the
live-dial Postgres suppression predicate and the birth-time dial wait. Covered
by a transition matrix over birth dial by later dial asserting where each write
lands and where each read is served.1 parent 3230a64 commit a410449
25 files changed
Lines changed: 1152 additions & 864 deletions
File tree
- apps/webapp
- app/v3
- test
- internal-packages/run-store/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | 23 | | |
25 | | - | |
26 | 24 | | |
27 | 25 | | |
28 | 26 | | |
| |||
56 | 54 | | |
57 | 55 | | |
58 | 56 | | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | 57 | | |
64 | 58 | | |
65 | 59 | | |
| |||
74 | 68 | | |
75 | 69 | | |
76 | 70 | | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
77 | 76 | | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | 77 | | |
88 | 78 | | |
89 | 79 | | |
90 | 80 | | |
91 | 81 | | |
92 | 82 | | |
93 | 83 | | |
94 | | - | |
95 | 84 | | |
96 | 85 | | |
97 | 86 | | |
| |||
107 | 96 | | |
108 | 97 | | |
109 | 98 | | |
110 | | - | |
111 | 99 | | |
112 | 100 | | |
113 | 101 | | |
114 | 102 | | |
115 | 103 | | |
116 | 104 | | |
117 | | - | |
118 | 105 | | |
119 | 106 | | |
120 | 107 | | |
| |||
128 | 115 | | |
129 | 116 | | |
130 | 117 | | |
131 | | - | |
132 | 118 | | |
133 | 119 | | |
134 | 120 | | |
| |||
227 | 213 | | |
228 | 214 | | |
229 | 215 | | |
230 | | - | |
231 | 216 | | |
232 | 217 | | |
233 | 218 | | |
| |||
246 | 231 | | |
247 | 232 | | |
248 | 233 | | |
249 | | - | |
250 | 234 | | |
251 | 235 | | |
252 | 236 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
3 | 2 | | |
4 | 3 | | |
5 | 4 | | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | 5 | | |
14 | 6 | | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | 7 | | |
30 | 8 | | |
31 | 9 | | |
| |||
41 | 19 | | |
42 | 20 | | |
43 | 21 | | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | 22 | | |
51 | 23 | | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
| 24 | + | |
56 | 25 | | |
57 | 26 | | |
58 | 27 | | |
59 | 28 | | |
60 | 29 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | 30 | | |
78 | 31 | | |
79 | 32 | | |
| |||
83 | 36 | | |
84 | 37 | | |
85 | 38 | | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | 39 | | |
110 | 40 | | |
111 | 41 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
| 97 | + | |
98 | 98 | | |
99 | | - | |
100 | | - | |
101 | 99 | | |
102 | 100 | | |
103 | 101 | | |
104 | 102 | | |
105 | 103 | | |
106 | 104 | | |
107 | 105 | | |
108 | | - | |
109 | | - | |
110 | 106 | | |
111 | 107 | | |
112 | 108 | | |
| |||
169 | 165 | | |
170 | 166 | | |
171 | 167 | | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | 168 | | |
178 | 169 | | |
179 | 170 | | |
| |||
184 | 175 | | |
185 | 176 | | |
186 | 177 | | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | 178 | | |
198 | 179 | | |
199 | 180 | | |
| |||
240 | 221 | | |
241 | 222 | | |
242 | 223 | | |
243 | | - | |
244 | 224 | | |
245 | 225 | | |
246 | 226 | | |
247 | 227 | | |
248 | | - | |
249 | 228 | | |
250 | 229 | | |
251 | 230 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | 4 | | |
34 | 5 | | |
35 | 6 | | |
36 | | - | |
37 | | - | |
38 | | - | |
| 7 | + | |
| 8 | + | |
39 | 9 | | |
40 | 10 | | |
41 | 11 | | |
42 | 12 | | |
43 | 13 | | |
44 | | - | |
| 14 | + | |
45 | 15 | | |
46 | | - | |
47 | 16 | | |
48 | 17 | | |
49 | | - | |
50 | | - | |
51 | | - | |
| 18 | + | |
| 19 | + | |
52 | 20 | | |
53 | 21 | | |
54 | 22 | | |
55 | 23 | | |
56 | 24 | | |
57 | | - | |
| 25 | + | |
58 | 26 | | |
59 | 27 | | |
60 | | - | |
61 | | - | |
62 | | - | |
| 28 | + | |
| 29 | + | |
63 | 30 | | |
64 | 31 | | |
65 | 32 | | |
66 | 33 | | |
67 | 34 | | |
68 | 35 | | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | 36 | | |
112 | 37 | | |
0 commit comments