Skip to content

fix(serverless): bound async waits and wait on resume/activate #143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ryank90 wants to merge 1 commit into
base: rc/serverless
Choose a base branch
from
Open

fix(serverless): bound async waits and wait on resume/activate #143

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/runware_serverless_apps_invoke.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ endpointPath is a bare lowercase segment as returned by apps endpoints
(e.g. infer). A leading slash is rejected.

The default is async: the command prints the accepted task id. Pass --wait
to poll until the task is completed or failed.
to poll until the task is completed or failed, and --timeout to bound that
wait.

--sync uses the sync invocation endpoint. If the platform wait window
expires, the command polls the returned task id; it never treats expiry as
Expand All @@ -35,7 +36,7 @@ runware serverless apps invoke <appId> <endpointPath> [flags]
runware serverless apps invoke my-app infer --sync -f payload.json

# async invoke and poll
runware serverless apps invoke my-app infer --wait -f payload.json
runware serverless apps invoke my-app infer --wait --timeout 2m -f payload.json

# retry a lost response without starting a second task
runware serverless apps invoke my-app infer --task-id 7c9e6679-7425-40de-944b-e07fc1f90ae7 -f payload.json
Expand All @@ -49,6 +50,7 @@ runware serverless apps invoke <appId> <endpointPath> [flags]
--poll-interval duration Polling interval when waiting for a task (default 2s)
--sync Use sync invocation and wait for a terminal task
--task-id string Client task id (UUID); generated if omitted
--timeout duration Maximum time to wait (0 = no limit)
--wait Poll until the task is completed or failed
```

Expand Down
13 changes: 10 additions & 3 deletions docs/runware_serverless_apps_resume.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ Resume a stopped serverless application
Resume a stopped serverless application.

The server accepts the resume and returns immediately with status initializing.
Worker start is asynchronous; this command does not wait until the application
is active. The application must be stopped.
Worker start is asynchronous. Pass --wait to poll until the application is
active or failed, and --timeout to bound that wait. The application must be
stopped.

```
runware serverless apps resume <appId> [flags]
Expand All @@ -19,12 +20,18 @@ runware serverless apps resume <appId> [flags]
```
# resume a stopped application
runware serverless apps resume my-app

# wait until the application is active or failed
runware serverless apps resume my-app --wait --timeout 5m
```

### Options

```
-h, --help help for resume
-h, --help help for resume
--poll-interval duration Polling interval when waiting for the application (default 2s)
--timeout duration Maximum time to wait (0 = no limit)
--wait Poll until the application is active or failed
```

### Options inherited from parent commands
Expand Down
15 changes: 11 additions & 4 deletions docs/runware_serverless_apps_versions_activate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ Activate a ready application version
Activate a ready version by number, including rollback to an older version.

The server accepts the deploy and returns immediately with the updated app.
Worker rollout is asynchronous; this command does not wait until workers are
healthy. Re-activating the currently active version is permitted and re-applies
it. On a stopped app the version is recorded and applied on resume.
Worker rollout is asynchronous. Pass --wait to poll until the application is
active or failed, and --timeout to bound that wait. Re-activating the currently
active version is permitted and re-applies it. On a stopped app the version is
recorded and applied on resume.

A missing app is 404. A missing version, a version that is not ready, or an
app that is deleting is 409.
Expand All @@ -27,12 +28,18 @@ runware serverless apps versions activate <appId> <versionNumber> [flags]

# roll back to an older ready version
runware serverless apps versions activate my-app 1

# wait until the rollout is active or failed
runware serverless apps versions activate my-app 2 --wait --timeout 5m
```

### Options

```
-h, --help help for activate
-h, --help help for activate
--poll-interval duration Polling interval when waiting for the application (default 2s)
--timeout duration Maximum time to wait (0 = no limit)
--wait Poll until the application is active or failed
```

### Options inherited from parent commands
Expand Down
6 changes: 4 additions & 2 deletions docs/runware_serverless_deploy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ builds a hosted wrapper image from that archive; the version records a buildId,
not a customer image reference. Invalid container.yaml is rejected on create
(400 if it cannot be parsed, 422 if it breaks a rule). The app stays
initializing until that first build rolls out. Pass --wait to poll until the
application is active or failed. A successful wait is not a live worker:
application is active or failed, and --timeout to bound that wait. A successful
wait is not a live worker:
minWorkers=0 stays scaled to zero until the first invoke.

--container cannot be combined with an entry file, --src-dir, --base-image, or
Expand Down Expand Up @@ -112,7 +113,7 @@ runware serverless deploy [file] [flags]
runware serverless deploy --id my-app --gpu-type h100 --container ./wrapper

# wait until the first rollout is active or failed
runware serverless deploy ./app.py --id my-app --gpu-type h100 --wait
runware serverless deploy ./app.py --id my-app --gpu-type h100 --wait --timeout 10m
```

### Options
Expand All @@ -138,6 +139,7 @@ runware serverless deploy [file] [flags]
--scaling-delay int32 Scaling delay in seconds (default 10)
--secret stringArray Organisation secret to attach at create, as NAME or NAME=ENV_VAR (repeatable)
--src-dir string Directory to package as the application source (default: the working directory; code deploys only)
--timeout duration Maximum time to wait (0 = no limit)
--volume stringArray Absolute path inside the app backed by persistent node-local storage; immutable after create (repeatable)
--wait Poll until the application is active or failed
```
Expand Down
18 changes: 12 additions & 6 deletions internal/cmd/serverless/apps_invoke.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ func newAppsInvokeCmd(logger *log.Logger) *cobra.Command {
var (
sync bool
wait bool
timeout time.Duration
bodyFile string
taskID string
pollInterval time.Duration
Expand All @@ -33,7 +34,8 @@ endpointPath is a bare lowercase segment as returned by apps endpoints
(e.g. infer). A leading slash is rejected.

The default is async: the command prints the accepted task id. Pass --wait
to poll until the task is completed or failed.
to poll until the task is completed or failed, and --timeout to bound that
wait.

--sync uses the sync invocation endpoint. If the platform wait window
expires, the command polls the returned task id; it never treats expiry as
Expand All @@ -50,7 +52,7 @@ instead of starting a second run.`,
runware serverless apps invoke my-app infer --sync -f payload.json

# async invoke and poll
runware serverless apps invoke my-app infer --wait -f payload.json
runware serverless apps invoke my-app infer --wait --timeout 2m -f payload.json

# retry a lost response without starting a second task
runware serverless apps invoke my-app infer --task-id 7c9e6679-7425-40de-944b-e07fc1f90ae7 -f payload.json`,
Expand Down Expand Up @@ -79,12 +81,15 @@ instead of starting a second run.`,

shouldWait := sync || wait
if shouldWait && task.Status == serverlessapi.TaskStatusPending {
_, _ = fmt.Fprintf(cmd.ErrOrStderr(), "Task %s accepted; waiting...\n", task.Id)
spin.SetMessage(fmt.Sprintf("Waiting for task %s...", task.Id))
task, err = client.WaitTask(cmd.Context(), appID, task.Id, pollInterval)
waitedID := task.Id
_, _ = fmt.Fprintf(cmd.ErrOrStderr(), "Task %s accepted; waiting...\n", waitedID)
spin.SetMessage(fmt.Sprintf("Waiting for task %s...", waitedID))
waitCtx, cancel := waitContext(cmd.Context(), timeout)
task, err = client.WaitTask(waitCtx, appID, waitedID, pollInterval)
cancel()
if err != nil {
spin.Stop()
return err
return waitTimeoutErr(err, "task "+waitedID, timeout)
}
}
spin.Stop()
Expand All @@ -98,6 +103,7 @@ instead of starting a second run.`,

cmd.Flags().BoolVar(&sync, "sync", false, "Use sync invocation and wait for a terminal task")
cmd.Flags().BoolVar(&wait, "wait", false, "Poll until the task is completed or failed")
cmd.Flags().DurationVar(&timeout, "timeout", 0, "Maximum time to wait (0 = no limit)")
cmd.Flags().StringVarP(&bodyFile, "body", "f", "", "JSON payload file, or - for stdin (default {})")
cmd.Flags().StringVar(&taskID, "task-id", "", "Client task id (UUID); generated if omitted")
cmd.Flags().DurationVar(&pollInterval, "poll-interval", 2*time.Second, "Polling interval when waiting for a task")
Expand Down
56 changes: 47 additions & 9 deletions internal/cmd/serverless/apps_lifecycle.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"log/slog"
"os"
"strings"
"time"

"github.com/charmbracelet/log"
serverlessapi "github.com/runware/runware-cli/internal/api/serverless"
Expand Down Expand Up @@ -38,27 +39,43 @@ is stopped. The application must be active.`,
runware serverless apps stop my-app`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
return runLifecycle(cmd, logger, args[0], "Stopping", (*serverlessapi.Client).StopApp)
return runLifecycle(cmd, logger, args[0], "Stopping", (*serverlessapi.Client).StopApp, waitOptions{})
},
}
}

func newAppsResumeCmd(logger *log.Logger) *cobra.Command {
return &cobra.Command{
var (
wait bool
timeout time.Duration
pollInterval time.Duration
)

cmd := &cobra.Command{
Use: "resume <appId>",
Short: "Resume a stopped serverless application",
Long: `Resume a stopped serverless application.

The server accepts the resume and returns immediately with status initializing.
Worker start is asynchronous; this command does not wait until the application
is active. The application must be stopped.`,
Worker start is asynchronous. Pass --wait to poll until the application is
active or failed, and --timeout to bound that wait. The application must be
stopped.`,
Example: ` # resume a stopped application
runware serverless apps resume my-app`,
runware serverless apps resume my-app

# wait until the application is active or failed
runware serverless apps resume my-app --wait --timeout 5m`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
return runLifecycle(cmd, logger, args[0], "Resuming", (*serverlessapi.Client).ResumeApp)
return runLifecycle(cmd, logger, args[0], "Resuming", (*serverlessapi.Client).ResumeApp, waitOptions{
Wait: wait,
Timeout: timeout,
Interval: pollInterval,
})
},
}
addAppWaitFlags(cmd, &wait, &timeout, &pollInterval)
return cmd
}

func newAppsDeleteCmd(logger *log.Logger) *cobra.Command {
Expand Down Expand Up @@ -88,7 +105,7 @@ Confirmation is required unless --yes or --force is passed.`,
if err := confirmDelete("application "+id, yes || force, cmd.InOrStdin(), cmd.ErrOrStderr(), stdinIsTerminal(cmd.InOrStdin()), config.GetAPIKey()); err != nil {
return err
}
return runLifecycle(cmd, logger, id, "Deleting", (*serverlessapi.Client).DeleteApp)
return runLifecycle(cmd, logger, id, "Deleting", (*serverlessapi.Client).DeleteApp, waitOptions{})
},
}

Expand All @@ -98,7 +115,13 @@ Confirmation is required unless --yes or --force is passed.`,

type lifecycleAction func(*serverlessapi.Client, context.Context, string) (*serverlessapi.App, error)

func runLifecycle(cmd *cobra.Command, logger *log.Logger, id, verb string, action lifecycleAction) error {
type waitOptions struct {
Wait bool
Timeout time.Duration
Interval time.Duration
}

func runLifecycle(cmd *cobra.Command, logger *log.Logger, id, verb string, action lifecycleAction, wait waitOptions) error {
spin := cmdutil.NewSpinner(fmt.Sprintf("%s application %s...", verb, id))
spin.Start()

Expand All @@ -108,9 +131,24 @@ func runLifecycle(cmd *cobra.Command, logger *log.Logger, id, verb string, actio
spin.Stop()
return err
}
if wait.Wait && !serverlessapi.AppDeployTerminal(app.Status) {
_, _ = fmt.Fprintf(cmd.ErrOrStderr(), "Application %s is %s; waiting...\n", app.AppId, app.Status)
spin.SetMessage(fmt.Sprintf("Waiting for application %s...", app.AppId))
app, err = waitForApp(cmd.Context(), client, app.AppId, wait.Interval, wait.Timeout)
if err != nil {
spin.Stop()
return err
}
}
spin.Stop()

return output.Print(cmdutil.FormatFor(cmd), appResult(*app))
if err := output.Print(cmdutil.FormatFor(cmd), appResult(*app)); err != nil {
return err
}
if !wait.Wait {
return nil
}
return appFailedErr(cmd.Context(), client, app)
}

func addDeleteConfirmFlags(cmd *cobra.Command, yes, force *bool) {
Expand Down
40 changes: 34 additions & 6 deletions internal/cmd/serverless/apps_versions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"fmt"
"log/slog"
"strconv"
"time"

"github.com/charmbracelet/log"
serverlessapi "github.com/runware/runware-cli/internal/api/serverless"
Expand Down Expand Up @@ -107,15 +108,22 @@ func newAppsVersionsShowCmd(logger *log.Logger) *cobra.Command {
}

func newAppsVersionsActivateCmd(logger *log.Logger) *cobra.Command {
return &cobra.Command{
var (
wait bool
timeout time.Duration
pollInterval time.Duration
)

cmd := &cobra.Command{
Use: "activate <appId> <versionNumber>",
Short: "Activate a ready application version",
Long: `Activate a ready version by number, including rollback to an older version.

The server accepts the deploy and returns immediately with the updated app.
Worker rollout is asynchronous; this command does not wait until workers are
healthy. Re-activating the currently active version is permitted and re-applies
it. On a stopped app the version is recorded and applied on resume.
Worker rollout is asynchronous. Pass --wait to poll until the application is
active or failed, and --timeout to bound that wait. Re-activating the currently
active version is permitted and re-applies it. On a stopped app the version is
recorded and applied on resume.

A missing app is 404. A missing version, a version that is not ready, or an
app that is deleting is 409.`,
Expand All @@ -124,7 +132,10 @@ app that is deleting is 409.`,
runware serverless apps versions activate my-app 2

# roll back to an older ready version
runware serverless apps versions activate my-app 1`,
runware serverless apps versions activate my-app 1

# wait until the rollout is active or failed
runware serverless apps versions activate my-app 2 --wait --timeout 5m`,
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
appID := args[0]
Expand All @@ -142,11 +153,28 @@ app that is deleting is 409.`,
spin.Stop()
return err
}
if wait && !serverlessapi.AppDeployTerminal(app.Status) {

@secretfader secretfader Sep 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug: activate on a live app answers active (SetActiveVersion keeps active; the roll never leaves active and restores the old pin if it fails). AppDeployTerminal(active) is true, so --wait never polls and exits 0 before workers roll. A failed rollback looks like success. The API has no rollout-progress field (events are free text), so --wait cannot follow this case. Skip the wait on active and print that the roll runs in the background, and fix the help text. Or drop --wait from activate until the API exposes rollout state.

_, _ = fmt.Fprintf(cmd.ErrOrStderr(), "Application %s is %s; waiting...\n", app.AppId, app.Status)
spin.SetMessage(fmt.Sprintf("Waiting for application %s...", app.AppId))
app, err = waitForApp(cmd.Context(), client, app.AppId, pollInterval, timeout)
if err != nil {
spin.Stop()
return err
}
}
spin.Stop()

return output.Print(cmdutil.FormatFor(cmd), appResult(*app))
if err := output.Print(cmdutil.FormatFor(cmd), appResult(*app)); err != nil {
return err
}
if !wait {
return nil
}
return appFailedErr(cmd.Context(), client, app)

@secretfader secretfader Sep 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

risk: activate on a stopped app answers stopped (pin recorded, rolls on resume). With --wait, appFailedErr exits 1 with "ended in status stopped" though the call succeeded. stopping polls until stopped, then fails the same way. Treat stopped/stopping as success with a note.

},
}
addAppWaitFlags(cmd, &wait, &timeout, &pollInterval)
return cmd
}

func newAppsVersionsDeleteCmd(logger *log.Logger) *cobra.Command {
Expand Down
Loading
Loading