Skip to content

fix(serverless): bound async waits and wait on resume/activate - #143

Open
Ryank90 wants to merge 1 commit into
rc/serverlessfrom
fix/serverless-wait-timeout
Open

Ryank90 wants to merge 1 commit into
rc/serverlessfrom
fix/serverless-wait-timeout

Conversation

@Ryank90

@Ryank90 Ryank90 commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add --wait / --timeout / --poll-interval to apps resume and apps versions activate. Those commands previously returned as soon as the API accepted the request (202 / initializing) and left rollout to the operator.
  • Add --timeout to deploy and apps invoke so first-image and task waits can be bounded. Default is 0 (no limit); WaitApp / WaitTask still end on a terminal status or a cancelled context, so Ctrl-C keeps working.
  • Timeouts are applied via context.WithTimeout. stop and delete stay fire-and-forget.

Test plan

  • runware serverless deploy ./app.py --id my-app --gpu-type h100 --wait --timeout 10m polls until active/failed or times out
  • runware serverless apps invoke my-app infer --wait --timeout 2m -f payload.json times out with timed out waiting for task … after 2m if the task stays pending
  • runware serverless apps resume my-app --wait --timeout 5m waits for active/failed
  • runware serverless apps versions activate my-app 2 --wait --timeout 5m waits for rollout
  • apps stop / apps delete have no --wait / --timeout
  • --wait without --timeout still polls until a terminal status; Ctrl-C cancels

Resume and activate returned as soon as the API accepted the request, and first-image --wait could poll until Ctrl-C. --timeout (default 0) now caps those waits via context.
@coderabbitai

coderabbitai Bot commented Sep 25, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. To trigger a review, include coderabbit-review in the PR description. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: e1edc0d5-b594-4809-a9f3-ba32f9e4d6ef

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@secretfader secretfader left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 bugs, 3 risks, 2 nits. Server behaviour checked against Runware/serverless main.

  • apps_invoke.go:L73: nit: --sync POST is not under --timeout. It can block ~120s (server write timeout) before the bounded poll starts. Wrap InvokeSync in the wait ctx (task id is already known, so --task-id retries it), or say in help that --timeout covers the poll only.
  • wait.go:L20: nit: a negative --timeout silently means no limit, and --timeout without --wait is silently ignored. Reject both in flag validation.
  • test gap: nothing drives deploy --wait into an error, which is how the L387 panic got through. TestWaitForApp_Timeout covers waitForApp only. Add a deploy test with an initializing app and --timeout 50ms.

if err != nil {
spin.Stop()
return err
return waitTimeoutErr(err, "application "+app.AppId, timeout)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug: waitForAppDeploy returns a nil app on any error, so app.AppId panics here. Every failed deploy --wait (timeout, Ctrl-C, GetApp 5xx) crashes with SIGSEGV. Reproduced: --wait --timeout 50ms against an initializing app. Capture appID := app.AppId before the call and use it here and in the report closure.

spin.Stop()
return err
}
if wait && !serverlessapi.AppDeployTerminal(app.Status) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug: activate on a live app answers active (SetActiveVersion keeps active; the roll never leaves active and restores the old pin if it fails). AppDeployTerminal(active) is true, so --wait never polls and exits 0 before workers roll. A failed rollback looks like success. The API has no rollout-progress field (events are free text), so --wait cannot follow this case. Skip the wait on active and print that the roll runs in the background, and fix the help text. Or drop --wait from activate until the API exposes rollout state.

if !wait {
return nil
}
return appFailedErr(cmd.Context(), client, app)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

risk: activate on a stopped app answers stopped (pin recorded, rolls on resume). With --wait, appFailedErr exits 1 with "ended in status stopped" though the call succeeded. stopping polls until stopped, then fails the same way. Treat stopped/stopping as success with a note.

}

func waitTimeoutErr(err error, subject string, timeout time.Duration) error {
if err == nil || !errors.Is(err, context.DeadlineExceeded) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

risk: errors.Is(err, context.DeadlineExceeded) also matches the client's 30s per-request Client.Timeout (net/http timeoutError.Is). One slow GetApp mid-wait reports "timed out ... after 5m" early, and with no --timeout says "timed out waiting" when no limit was set. Rewrite only when the wait ctx itself expired (waitCtx.Err() == context.DeadlineExceeded). The timeout == 0 branch then goes away.

if canCompare {
settled, err := waitForSubmittedVersion(cmd.Context(), client, app.AppId, versionBefore, pollInterval)
settled, err := waitForSubmittedVersion(waitCtx, client, app.AppId, versionBefore, pollInterval)
if errors.Is(err, context.DeadlineExceeded) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

risk: same match here turns one 30s request timeout inside waitForSubmittedVersion into a failed deploy (exit 1, app not printed), even without --timeout. Before this PR that error only skipped the endpoint-change warning. Gate on waitCtx.Err().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants