Skip to content

fix(serverless): set secrets and scale knobs on deploy create #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ryank90 merged 2 commits into from
Sep 23, 2026
Merged

fix(serverless): set secrets and scale knobs on deploy create #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d309973
fix(serverless): set secrets and scale knobs on deploy create
Ryank90 Sep 22, 2026
031748d
fix(serverless): prove create carries secrets and reject a bad idle-w…
Ryank90 Sep 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 43 additions & 26 deletions docs/runware_serverless_deploy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@ Create or update a serverless application from Python code or a container source
A first deploy with a new --id creates the application. A later deploy with the
same --id uploads a new source, records version N+1, and rolls it when the
build is ready. Create-only flags (--gpu-type, worker settings, --volume,
--env, --env-file, --name) apply only to create; passing them when the
application already exists is an error. Change workers with 'apps scale' and
environment with 'apps env'. A source update on a stopped application is 409.
--secret, --env, --env-file, --name) apply only to create; passing them when
the application already exists is an error. Change workers with 'apps scale',
attach a secret later with 'secrets attach', and change environment with
'apps env'. A source update on a stopped application is 409.

A code deploy takes a Python entry file. The whole source directory is zipped
and submitted as the application source, so the entry file can import its own
Expand Down Expand Up @@ -49,12 +50,19 @@ returns 409 and does not store the value. Prefer --env-file for anything
secret: a value passed as --env is visible in the process list and recorded
in shell history.

Anything the app downloads at runtime belongs on a --volume. The app runs in a
sandbox whose filesystem is part of the checkpointed state, so an unmounted
download is copied into every checkpoint and fetched again on every cold start.
A volume keeps it out of both.
Anything the app downloads at runtime belongs on a --volume. Volumes are set
at create and cannot be changed afterwards: a later deploy that passes
--volume is rejected. The app runs in a sandbox whose filesystem is part of
the checkpointed state, so an unmounted download is copied into every
checkpoint and fetched again on every cold start. A volume keeps it out of
both.

Worker settings are supplied via flags on create. Endpoints are derived
--secret NAME, or NAME=ENV_VAR, attaches an existing organisation secret at
create so the first rollout carries it. Repeat the flag for more than one.
Attach or detach later with 'secrets attach' and 'secrets detach'.

Worker settings are supplied via flags on create, including concurrency, a
fallback GPU type, and the idle-worker buffer. Endpoints are derived
server-side from the SDK (code) or from container.yaml (container).

A code app's endpoint path is its handler's method name with underscores turned
Expand Down Expand Up @@ -90,6 +98,10 @@ runware serverless deploy [file] [flags]
runware serverless deploy model.py --id my-app --gpu-type l40s \
--volume /root/.cache/huggingface

# attach an existing secret and keep one idle worker warm
runware serverless deploy ./app.py --id my-app --gpu-type h100 \
--secret HF_TOKEN=HUGGING_FACE_HUB_TOKEN --min-available-workers 1

# override worker settings and base image
runware serverless deploy ./app.py --id my-app --name "My App" \
--max-workers 2 --idle-ttl 120 --gpu-type h100 \
Expand All @@ -105,24 +117,29 @@ runware serverless deploy [file] [flags]
### Options

```
--base-image string Builder base image (code deploys only) (default "python:3.11-slim")
--container string Directory whose root contains Dockerfile and container.yaml
--env stringArray Environment variable as KEY=VALUE (repeatable)
--env-file stringArray File of KEY=VALUE lines to read environment variables from (repeatable)
--gpu-type string GPU type ID (see 'serverless gpus'; required when creating)
--gpus-per-worker int32 GPUs allocated per worker (1, 2, 4, or 8) (default 1)
-h, --help help for deploy
--id string Application ID (immutable, lowercase slug)
--idle-ttl int32 Idle TTL in seconds before scaling down (default 60)
--max-workers int32 Maximum number of workers (default 1)
--min-workers int32 Minimum number of workers
--name string Display name (defaults to --id)
--poll-interval duration Polling interval when waiting for the application (default 2s)
--requirement stringArray Additional pip package to install (repeatable; code deploys only)
--scaling-delay int32 Scaling delay in seconds (default 10)
--src-dir string Directory to package as the application source (default: the working directory; code deploys only)
--volume stringArray Absolute path inside the app backed by persistent node-local storage (repeatable)
--wait Poll until the application is active or failed
--available-workers-pct int32 Idle-worker buffer as a percentage of load (0-100)
--base-image string Builder base image (code deploys only) (default "python:3.11-slim")
--concurrency int32 Max tasks a single worker handles simultaneously (default 1)
--container string Directory whose root contains Dockerfile and container.yaml
--env stringArray Environment variable as KEY=VALUE (repeatable)
--env-file stringArray File of KEY=VALUE lines to read environment variables from (repeatable)
--fallback-gpu-type string Secondary GPU type if the preferred type is unavailable
--gpu-type string GPU type ID (see 'serverless gpus'; required when creating)
--gpus-per-worker int32 GPUs allocated per worker (1, 2, 4, or 8) (default 1)
-h, --help help for deploy
--id string Application ID (immutable, lowercase slug)
--idle-ttl int32 Idle TTL in seconds before scaling down (default 60)
--max-workers int32 Maximum number of workers (default 1)
--min-available-workers int32 Minimum idle workers kept as a buffer
--min-workers int32 Minimum number of workers
--name string Display name (defaults to --id)
--poll-interval duration Polling interval when waiting for the application (default 2s)
--requirement stringArray Additional pip package to install (repeatable; code deploys only)
--scaling-delay int32 Scaling delay in seconds (default 10)
--secret stringArray Organisation secret to attach at create, as NAME or NAME=ENV_VAR (repeatable)
--src-dir string Directory to package as the application source (default: the working directory; code deploys only)
--volume stringArray Absolute path inside the app backed by persistent node-local storage; immutable after create (repeatable)
--wait Poll until the application is active or failed
```

### Options inherited from parent commands
Expand Down
2 changes: 1 addition & 1 deletion internal/cmd/serverless/apps_scale_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ func TestWorkerConfigPatchFromFlags_EachFlag(t *testing.T) {
{[]string{"--idle-ttl", "120"}, "idleTtlSecs", float64(120)},
{[]string{"--scaling-delay", "15"}, "scalingDelaySecs", float64(15)},
{[]string{"--concurrency", "4"}, "concurrency", float64(4)},
{[]string{"--gpu-type", testGPUType}, "gpuType", testGPUType},
{[]string{testGPUTypeFlag, testGPUType}, "gpuType", testGPUType},
{[]string{"--gpus-per-worker", "2"}, "gpusPerWorker", float64(2)},
{[]string{"--fallback-gpu-type", testGPUType}, "fallbackGpuType", testGPUType},
{[]string{"--min-available-workers", "1"}, "minAvailableWorkers", float64(1)},
Expand Down
152 changes: 118 additions & 34 deletions internal/cmd/serverless/deploy.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"context"
"fmt"
"log/slog"
"strings"
"time"

"github.com/charmbracelet/log"
Expand Down Expand Up @@ -33,7 +34,12 @@ var createOnlyDeployFlags = []string{
"scaling-delay",
"min-workers",
"gpus-per-worker",
"concurrency",
"fallback-gpu-type",
"min-available-workers",
"available-workers-pct",
"volume",
"secret",
"env",
"env-file",
}
Expand Down Expand Up @@ -112,23 +118,28 @@ func buildDeployArchive(srcDir, containerDir, baseImage string, requirements []s

func newDeployCmd(logger *log.Logger) *cobra.Command {
var (
id string
name string
maxWorkers int32
idleTTL int32
scalingDelay int32
baseImage string
gpuType string
requirements []string
minWorkers int32
gpusPerWorker int32
srcDir string
containerDir string
volumes []string
envVars []string
envFiles []string
wait bool
pollInterval time.Duration
id string
name string
maxWorkers int32
idleTTL int32
scalingDelay int32
baseImage string
gpuType string
requirements []string
minWorkers int32
gpusPerWorker int32
concurrency int32
fallbackGPUType string
minAvailableWorkers int32
availableWorkersPct int32
srcDir string
containerDir string
volumes []string
secrets []string
envVars []string
envFiles []string
wait bool
pollInterval time.Duration
)

cmd := &cobra.Command{
Expand All @@ -139,9 +150,10 @@ func newDeployCmd(logger *log.Logger) *cobra.Command {
A first deploy with a new --id creates the application. A later deploy with the
same --id uploads a new source, records version N+1, and rolls it when the
build is ready. Create-only flags (--gpu-type, worker settings, --volume,
--env, --env-file, --name) apply only to create; passing them when the
application already exists is an error. Change workers with 'apps scale' and
environment with 'apps env'. A source update on a stopped application is 409.
--secret, --env, --env-file, --name) apply only to create; passing them when
the application already exists is an error. Change workers with 'apps scale',
attach a secret later with 'secrets attach', and change environment with
'apps env'. A source update on a stopped application is 409.

A code deploy takes a Python entry file. The whole source directory is zipped
and submitted as the application source, so the entry file can import its own
Expand Down Expand Up @@ -179,12 +191,19 @@ returns 409 and does not store the value. Prefer --env-file for anything
secret: a value passed as --env is visible in the process list and recorded
in shell history.

Anything the app downloads at runtime belongs on a --volume. The app runs in a
sandbox whose filesystem is part of the checkpointed state, so an unmounted
download is copied into every checkpoint and fetched again on every cold start.
A volume keeps it out of both.
Anything the app downloads at runtime belongs on a --volume. Volumes are set
at create and cannot be changed afterwards: a later deploy that passes
--volume is rejected. The app runs in a sandbox whose filesystem is part of
the checkpointed state, so an unmounted download is copied into every
checkpoint and fetched again on every cold start. A volume keeps it out of
both.

Worker settings are supplied via flags on create. Endpoints are derived
--secret NAME, or NAME=ENV_VAR, attaches an existing organisation secret at
create so the first rollout carries it. Repeat the flag for more than one.
Attach or detach later with 'secrets attach' and 'secrets detach'.

Worker settings are supplied via flags on create, including concurrency, a
fallback GPU type, and the idle-worker buffer. Endpoints are derived
server-side from the SDK (code) or from container.yaml (container).

A code app's endpoint path is its handler's method name with underscores turned
Expand Down Expand Up @@ -212,6 +231,10 @@ endpoint on purpose is allowed.`,
runware serverless deploy model.py --id my-app --gpu-type l40s \
--volume /root/.cache/huggingface

# attach an existing secret and keep one idle worker warm
runware serverless deploy ./app.py --id my-app --gpu-type h100 \
--secret HF_TOKEN=HUGGING_FACE_HUB_TOKEN --min-available-workers 1

# override worker settings and base image
runware serverless deploy ./app.py --id my-app --name "My App" \
--max-workers 2 --idle-ttl 120 --gpu-type h100 \
Expand All @@ -232,6 +255,11 @@ endpoint on purpose is allowed.`,
return err
}
}
if cmd.Flags().Changed("available-workers-pct") {
if err := validateAvailableWorkersPct(availableWorkersPct); err != nil {
return err
}
}
if name == "" {
name = id
}
Expand Down Expand Up @@ -271,6 +299,7 @@ endpoint on purpose is allowed.`,
var (
appVolumes *[]serverlessapi.AppVolume
appEnv *map[string]string
appSecrets *[]serverlessapi.SecretAttach
)
if !update {
appVolumes, err = buildVolumes(volumes)
Expand All @@ -281,6 +310,10 @@ endpoint on purpose is allowed.`,
if err != nil {
return err
}
appSecrets, err = parseSecretAttaches(secrets)
if err != nil {
return err
}
}

spin := cmdutil.NewSpinner(fmt.Sprintf("Uploading source for %s...", id))
Expand Down Expand Up @@ -312,13 +345,18 @@ endpoint on purpose is allowed.`,
AppSource: appSource,
Volumes: appVolumes,
EnvironmentVariables: appEnv,
Secrets: appSecrets,
Comment thread
wilsonsilva marked this conversation as resolved.
Configuration: serverlessapi.WorkerConfigCreate{
MaxWorkers: maxWorkers,
IdleTtlSecs: idleTTL,
ScalingDelaySecs: scalingDelay,
GpuType: gpuType,
MinWorkers: optionalInt32Ptr(cmd, "min-workers", minWorkers),
GpusPerWorker: optionalInt32Ptr(cmd, "gpus-per-worker", gpusPerWorker),
MaxWorkers: maxWorkers,
IdleTtlSecs: idleTTL,
ScalingDelaySecs: scalingDelay,
GpuType: gpuType,
MinWorkers: optionalInt32Ptr(cmd, "min-workers", minWorkers),
GpusPerWorker: optionalInt32Ptr(cmd, "gpus-per-worker", gpusPerWorker),
Concurrency: optionalInt32Ptr(cmd, "concurrency", concurrency),
FallbackGpuType: optionalFlagStringPtr(cmd, "fallback-gpu-type", fallbackGPUType),
MinAvailableWorkers: optionalInt32Ptr(cmd, "min-available-workers", minAvailableWorkers),
AvailableWorkersPct: optionalInt32Ptr(cmd, "available-workers-pct", availableWorkersPct),
},
})
if isHTTPConflict(err) {
Expand Down Expand Up @@ -368,7 +406,8 @@ endpoint on purpose is allowed.`,

cmd.Flags().StringVar(&srcDir, "src-dir", "", "Directory to package as the application source (default: the working directory; code deploys only)")
cmd.Flags().StringVar(&containerDir, "container", "", "Directory whose root contains Dockerfile and container.yaml")
cmd.Flags().StringArrayVar(&volumes, "volume", nil, "Absolute path inside the app backed by persistent node-local storage (repeatable)")
cmd.Flags().StringArrayVar(&volumes, "volume", nil, "Absolute path inside the app backed by persistent node-local storage; immutable after create (repeatable)")
cmd.Flags().StringArrayVar(&secrets, "secret", nil, "Organisation secret to attach at create, as NAME or NAME=ENV_VAR (repeatable)")
cmd.Flags().StringArrayVar(&envVars, "env", nil, "Environment variable as KEY=VALUE (repeatable)")
cmd.Flags().StringArrayVar(&envFiles, "env-file", nil, "File of KEY=VALUE lines to read environment variables from (repeatable)")
cmd.Flags().StringVar(&id, "id", "", "Application ID (immutable, lowercase slug)")
Expand All @@ -381,6 +420,10 @@ endpoint on purpose is allowed.`,
cmd.Flags().StringArrayVar(&requirements, "requirement", nil, "Additional pip package to install (repeatable; code deploys only)")
cmd.Flags().Int32Var(&minWorkers, "min-workers", 0, "Minimum number of workers")
cmd.Flags().Int32Var(&gpusPerWorker, "gpus-per-worker", 1, "GPUs allocated per worker (1, 2, 4, or 8)")
cmd.Flags().Int32Var(&concurrency, "concurrency", 1, "Max tasks a single worker handles simultaneously")
Comment thread
Ryank90 marked this conversation as resolved.
cmd.Flags().StringVar(&fallbackGPUType, "fallback-gpu-type", "", "Secondary GPU type if the preferred type is unavailable")
cmd.Flags().Int32Var(&minAvailableWorkers, "min-available-workers", 0, "Minimum idle workers kept as a buffer")
cmd.Flags().Int32Var(&availableWorkersPct, "available-workers-pct", 0, "Idle-worker buffer as a percentage of load (0-100)")
Comment thread
wilsonsilva marked this conversation as resolved.
cmd.Flags().BoolVar(&wait, "wait", false, "Poll until the application is active or failed")
cmd.Flags().DurationVar(&pollInterval, "poll-interval", 2*time.Second, "Polling interval when waiting for the application")

Expand Down Expand Up @@ -425,6 +468,13 @@ func validateGPUsPerWorker(n int32) error {
return fmt.Errorf("--gpus-per-worker must be 1, 2, 4, or 8")
}

func validateAvailableWorkersPct(n int32) error {
if n < 0 || n > 100 {
return fmt.Errorf("--available-workers-pct must be between 0 and 100")
}
return nil
}

func validateUpdateDeployFlags(cmd *cobra.Command) error {
for _, name := range createOnlyDeployFlags {
if cmd.Flags().Changed(name) {
Expand All @@ -436,17 +486,51 @@ func validateUpdateDeployFlags(cmd *cobra.Command) error {

func createOnlyDeployHint(name string) string {
switch name {
case "gpu-type", "max-workers", "idle-ttl", "scaling-delay", "min-workers", "gpus-per-worker":
case "gpu-type", "max-workers", "idle-ttl", "scaling-delay", "min-workers", "gpus-per-worker", "concurrency", "fallback-gpu-type", "min-available-workers", "available-workers-pct":
return "use 'runware serverless apps scale' to change worker configuration"
case "env", "env-file":
return "use 'runware serverless apps env' to change environment variables"
case "secret":
return "use 'runware serverless secrets attach' to attach a secret"
case "volume":
return "volumes are set at create time and cannot be changed here"
return "volumes are immutable after create"
default:
return "omit it when updating an existing application"
}
}

func parseSecretAttaches(values []string) (*[]serverlessapi.SecretAttach, error) {
if len(values) == 0 {
return nil, nil
}
out := make([]serverlessapi.SecretAttach, 0, len(values))
for _, raw := range values {
name, envVar, err := splitSecretAttach(raw)
if err != nil {
return nil, err
}
attach := serverlessapi.SecretAttach{
SecretName: name,
}
if envVar != "" {
attach.EnvVarName = &envVar
}
out = append(out, attach)
}
return &out, nil
}

func splitSecretAttach(raw string) (string, string, error) {
name, envVar, hasEnv := strings.Cut(raw, "=")
if name == "" || !envNamePattern.MatchString(name) {
return "", "", fmt.Errorf("invalid --secret %q (want NAME or NAME=ENV_VAR)", raw)
}
if hasEnv && (envVar == "" || !envNamePattern.MatchString(envVar)) {
return "", "", fmt.Errorf("invalid --secret %q (want NAME or NAME=ENV_VAR)", raw)
}
return name, envVar, nil
}

func optionalStringSlice(vals []string) *[]string {
if len(vals) == 0 {
return nil
Expand Down
Loading
Loading