Repository navigation
Commit bec4639
authored
feat(mothership): run scheduled jobs as durable unattended chat turns (#9331)
* feat(mothership): run unattended turns durably on the leased chat path
A scheduled job's occurrence must outlive the Sim process driving it. The
headless lifecycle the wake door uses takes no controller lease, so a pod
death leaves its run unrecoverable. An unattended turn now admits like a
typed send (user message, run and recovery intent in one commit under the
chat lock), runs detached from any HTTP response, and records a new
unattended_stream recovery kind. Takeover restores it non-interactive with
the secret authority it was admitted with, instead of the interactive
default, and the abandoned-run sweep, stream liveness and billing-admission
recording treat both durable kinds alike.
Durable start now also forwards a caller's billing-admission hook, so a
job can key its per-run budget on the run's billing request.
The integration test kills the turn's controller three times mid-run; it
fails on a headless implementation, on a sweep blind to the new kind, and
on a takeover that restores the interactive default.
* feat(db): add scheduled job occurrence columns and archive legacy jobs
Scheduled jobs return as Sim-owned schedules whose occurrences run as
durable chat turns. A job's run record (job_execution_logs) gains the
occurrence it runs (unique per schedule), its outcome and summary, and
the billing request its charges settle under; a schedule gains a per-run
spend cap. Indexes are built concurrently.
Every job row written for the retired Go runner is archived, never
deleted, so none can start once the tick claims jobs again. The archive
is a script migration committing per keyed batch, so it finishes on any
row count and resumes after a crash.
* feat(mothership): run scheduled jobs from the schedule tick
Behind MOTHERSHIP_JOBS_ENABLED (off by default, and to stay off in
production until unattended runs hold production effects for approval),
the schedule tick gains a job lane:
- Claim: each due job's occurrence is recorded (queued) and the schedule
advances past it in one transaction, skipping locked rows, so two ticks
record it once and a crash never leaves a stale claim. Missed
occurrences collapse into one; cadence is never under 15 minutes; a
one-shot job (no cron) runs once and completes; max_runs completes.
- Dispatch: the occurrence starts as an unattended durable turn in the
job's chat under its occurrence id (UUIDv5 of schedule and instant), so
every retry finds the turn it admitted. A busy chat skips the
occurrence; a payer over its plan limit skips it; an owner who lost
access turns the job off. Only secrets a person named for the job mount.
- Reconcile: a queued occurrence whose dispatch was lost is dispatched
again after two minutes; a running one whose turn ended is finalized;
one past six hours is stopped.
- Finalize: exactly once under a row lock. A reported outcome is kept; a
turn that ended well without one is not_met, summarized by its last
reply. Three failed or interrupted runs in a row turn the job off and
email its people; a met until_complete job completes.
- Budget: each billing callback records the run's cumulative settled cost
on its record (keyed by the run's billing request), and a run at its
cap pauses at its next step with a job_run_budget card. The cost is
cumulative per billing request, which recovered legs reuse, so the cap
survives worker hand-offs and Sim takeovers.
* improvement(mothership): run the job lane unflagged behind one indexed probe
* improvement(mothership): simplify the job lane and stop jobs through the shared Stop delivery
* chore(db): format the scheduled job migration metadata
* improvement(mothership): keep the durable-run predicates out of the chat protocol graph
* fix(mothership): settle job runs before claiming, keep spend-cap links and failure counts durable
* fix(mothership): stop orphaned job runs at their limit and skip any number of deleted occurrences
* fix(mothership): end a running job record whose run was removed1 parent ff98c86 commit bec4639
49 files changed
Lines changed: 34815 additions & 119 deletions
File tree
- apps/sim
- app
- api
- billing/update-cost
- schedules/execute
- workspace/[workspaceId]/home
- components
- message-content/components/special-tags
- mothership-chat
- lib
- api/contracts
- mothership
- async-runs
- chat
- application
- generated
- jobs
- request
- application
- lifecycle
- workflows/schedules
- packages
- db
- migrations
- meta
- script-migrations
- testing/src/mocks
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| 52 | + | |
52 | 53 | | |
53 | 54 | | |
54 | 55 | | |
| |||
122 | 123 | | |
123 | 124 | | |
124 | 125 | | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
125 | 148 | | |
126 | 149 | | |
127 | 150 | | |
| |||
442 | 465 | | |
443 | 466 | | |
444 | 467 | | |
445 | | - | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
446 | 473 | | |
447 | 474 | | |
448 | 475 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
111 | 116 | | |
112 | 117 | | |
113 | 118 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
1788 | 1789 | | |
1789 | 1790 | | |
1790 | 1791 | | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
1791 | 1799 | | |
1792 | 1800 | | |
1793 | 1801 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
523 | 523 | | |
524 | 524 | | |
525 | 525 | | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3443 | 3443 | | |
3444 | 3444 | | |
3445 | 3445 | | |
| 3446 | + | |
| 3447 | + | |
| 3448 | + | |
| 3449 | + | |
| 3450 | + | |
| 3451 | + | |
| 3452 | + | |
| 3453 | + | |
| 3454 | + | |
| 3455 | + | |
| 3456 | + | |
| 3457 | + | |
| 3458 | + | |
| 3459 | + | |
| 3460 | + | |
| 3461 | + | |
| 3462 | + | |
| 3463 | + | |
| 3464 | + | |
| 3465 | + | |
| 3466 | + | |
| 3467 | + | |
| 3468 | + | |
| 3469 | + | |
| 3470 | + | |
| 3471 | + | |
| 3472 | + | |
| 3473 | + | |
| 3474 | + | |
| 3475 | + | |
| 3476 | + | |
| 3477 | + | |
| 3478 | + | |
| 3479 | + | |
| 3480 | + | |
| 3481 | + | |
| 3482 | + | |
3446 | 3483 | | |
3447 | 3484 | | |
3448 | 3485 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
187 | 188 | | |
188 | 189 | | |
189 | 190 | | |
190 | | - | |
191 | | - | |
192 | | - | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
193 | 194 | | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
| 195 | + | |
| 196 | + | |
198 | 197 | | |
199 | 198 | | |
200 | | - | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
201 | 206 | | |
202 | | - | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
203 | 214 | | |
204 | 215 | | |
205 | 216 | | |
206 | | - | |
207 | | - | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
208 | 221 | | |
209 | 222 | | |
210 | 223 | | |
| |||
930 | 943 | | |
931 | 944 | | |
932 | 945 | | |
933 | | - | |
934 | | - | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
935 | 952 | | |
936 | 953 | | |
937 | 954 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
115 | 127 | | |
116 | 128 | | |
117 | 129 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
166 | | - | |
| 166 | + | |
167 | 167 | | |
168 | 168 | | |
169 | 169 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
197 | 197 | | |
198 | 198 | | |
199 | 199 | | |
200 | | - | |
201 | | - | |
| 200 | + | |
| 201 | + | |
202 | 202 | | |
203 | 203 | | |
204 | 204 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
9 | 13 | | |
10 | 14 | | |
11 | 15 | | |
| |||
315 | 319 | | |
316 | 320 | | |
317 | 321 | | |
318 | | - | |
319 | | - | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
320 | 325 | | |
321 | 326 | | |
322 | 327 | | |
| |||
0 commit comments