Repository navigation
Mark runs lost when their SSH host stays unreachable and deliver pending wake-ups together (OR-628) - #796
Conversation
…g wake-ups per session (OR-628) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
[Medium impact] The PR appears safe to merge; the host probe fixes the previous false-loss finding. SummaryThe PR adds a deadline for unreachable SSH hosts and sends ready wake-ups together for each chat session.
Reviews (3) · Last reviewed commit: "Count only polls where the SSH host fail..." · Reviewed by Greptile |
…; keep the outage alert backend-neutral (OR-628) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…al outage alert text (OR-628) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
@greptileai review — addressed both findings in 7b2c0ff/ab03c7f9: the 15-minute LOST deadline now also requires the host to fail a trivial |
…e lost deadline (OR-628) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
@greptileai review — 6506ea8: the lost deadline now only accumulates across polls where the host itself fails |
Fixes OR-628.
Summary
runningforever.watch_ssh_job(src/commands/supervise.rs) retried failedinspect_jobpolls indefinitely, so the run never reached a terminal state andorx exp wakenever fired. When a poll fails, the supervisor now probes the host withssh_run(target, "true"); only consecutive polls where that probe also fails count towardUNREACHABLE_LIMIT(15 minutes). After that the run ends aslost("<host>was unreachable for 15 minutes while the run was active"), and the existinglostpath wakes the agent ("Outcome unavailable") and lets teardown keyed on run status act. Inspection errors on a reachable host (e.g. Docker failures) keep retrying as before.first_wakeup_per_sessiondeliberately sent only the earliest ready wake-up per session per watcher tick. It is replaced bypending_wakeups_by_session, andprocess_run_wakeupsclaims all of a session's ready wake-ups and delivers them in one hidden turn (all marked delivered on start, all released otherwise or if collecting the batch fails). Sessions with a wake-up claimed elsewhere still wait.Not linked to any contributor PR. Related open PRs touch nearby SSH code but do not address this: #792 (non-numeric exit receipt), #778 (reopen SSH master), #745 (cancel unreachable runs), #791 (on-demand monitoring).
Recordings
Test plan
cargo fmt --check,cargo clippy --locked -- -D warningsready_wakeups_are_grouped_per_session_unless_one_is_claimed,unreachable_hosts_are_declared_lost_only_after_the_limitcargo test --lockedhas 3local::projectsfailures (dedicated_github_remote_is_recognized_but_remains_opt_in,imports_an_existing_repository_without_commits,ordinary_github_origin_remains_opt_in) that fail identically on unmodifiedmainin this VM (environment), and pass in CImain(Linux, real Claude Code chat in the dashboard, SSH hostgpuboxkilled via firewall + process kill)ssh-finalwent fromrunningto lost ("Outcome unavailable") ~15 min after the host died and the agent was woken; three local runs finishing during a busy turn were delivered in one wake turn (delivered within 2 ms of each other)Link to Devin session: https://app.devin.ai/sessions/c01a87ebae98423d9e6d52f3a40b940b
Open in Devin Desktop: https://app.devin.ai/desktop/session/c01a87ebae98423d9e6d52f3a40b940b?variant=devin