Skip to content

Mark runs lost when their SSH host stays unreachable and deliver pending wake-ups together (OR-628) - #796

Open
devin-ai-integration[bot] wants to merge 5 commits into
mainfrom
devin/1793055324-or628-ssh-lost-batched-wakes
Open

devin-ai-integration[bot] wants to merge 5 commits into
mainfrom
devin/1793055324-or628-ssh-lost-batched-wakes

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes OR-628.

Summary

  • SSH box dies mid-run → run stays running forever. watch_ssh_job (src/commands/supervise.rs) retried failed inspect_job polls indefinitely, so the run never reached a terminal state and orx exp wake never fired. When a poll fails, the supervisor now probes the host with ssh_run(target, "true"); only consecutive polls where that probe also fails count toward UNREACHABLE_LIMIT (15 minutes). After that the run ends as lost ("<host> was unreachable for 15 minutes while the run was active"), and the existing lost path wakes the agent ("Outcome unavailable") and lets teardown keyed on run status act. Inspection errors on a reachable host (e.g. Docker failures) keep retrying as before.
  • Wakes for runs that finished during a long turn arrived one per idle turn. first_wakeup_per_session deliberately sent only the earliest ready wake-up per session per watcher tick. It is replaced by pending_wakeups_by_session, and process_run_wakeups claims all of a session's ready wake-ups and delivers them in one hidden turn (all marked delivered on start, all released otherwise or if collecting the batch fails). Sessions with a wake-up claimed elsewhere still wait.

Not linked to any contributor PR. Related open PRs touch nearby SSH code but do not address this: #792 (non-numeric exit receipt), #778 (reopen SSH master), #745 (cancel unreachable runs), #791 (on-demand monitoring).

Recordings

Bug reproduction on main

Post-fix functionality

Test plan

  • cargo fmt --check, cargo clippy --locked -- -D warnings
  • New tests: ready_wakeups_are_grouped_per_session_unless_one_is_claimed, unreachable_hosts_are_declared_lost_only_after_the_limit
  • CI green. Locally cargo test --locked has 3 local::projects failures (dedicated_github_remote_is_recognized_but_remains_opt_in, imports_an_existing_repository_without_commits, ordinary_github_origin_remains_opt_in) that fail identically on unmodified main in this VM (environment), and pass in CI
  • Reproduced both bugs on main (Linux, real Claude Code chat in the dashboard, SSH host gpubox killed via firewall + process kill)
  • Post-fix on 6506ea8, same flow: ssh-final went from running to lost ("Outcome unavailable") ~15 min after the host died and the agent was woken; three local runs finishing during a busy turn were delivered in one wake turn (delivered within 2 ms of each other)

Link to Devin session: https://app.devin.ai/sessions/c01a87ebae98423d9e6d52f3a40b940b
Open in Devin Desktop: https://app.devin.ai/desktop/session/c01a87ebae98423d9e6d52f3a40b940b?variant=devin

…g wake-ups per session (OR-628)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@greptile-apps

greptile-apps Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the host probe fixes the previous false-loss finding.

Summary

The PR adds a deadline for unreachable SSH hosts and sends ready wake-ups together for each chat session.

  • SSH runs become lost after their host stays unreachable for 15 minutes.
  • Finished runs reach each chat session together.

Reviews (3) · Last reviewed commit: "Count only polls where the SSH host fail..." · Reviewed by Greptile

Comment thread src/commands/supervise.rs Outdated
Comment thread src/local/chat/mod.rs Outdated
…; keep the outage alert backend-neutral (OR-628)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…al outage alert text (OR-628)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai review — addressed both findings in 7b2c0ff/ab03c7f9: the 15-minute LOST deadline now also requires the host to fail a trivial ssh_run(target, "true") probe (Docker/inspection errors on a reachable host keep retrying), and the outage alert text is back to its original backend-neutral wording.

…e lost deadline (OR-628)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai review — 6506ea8: the lost deadline now only accumulates across polls where the host itself fails ssh_run(target, "true"); any successful probe (or successful inspection) resets it.

@devin-ai-integration devin-ai-integration Bot added the awaiting review Devin bugfix PR is open and needs human review before merge label Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review Devin bugfix PR is open and needs human review before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant