Squall

Scale-to-zero LLM serving on GPUs that are not in your cluster.

Squall gives a Kubernetes cluster access to GPU capacity it does not own. You declare a Model custom resource in Git; squall rents a GPU from an external provider when the first request arrives, serves OpenAI-compatible traffic from it, and gives the machine back when the traffic stops. Between requests, there is nothing to pay for.

If you know KubeAI: squall is that idea for capacity that is not in your cluster. KubeAI also scales models to zero, and paired with a cluster autoscaler it can grow GPU nodes on demand — but the capacity always comes from the same cluster, so it is bounded by that cluster's region and quota. Squall scales machines to zero and rents them through dstack from Vast.ai, AWS or DigitalOcean. The GPU need not be where the cluster is: on AWS it can run in a region your cluster has no presence in, which is often the only region with the card you want. Kubernetes holds the intent and does the coordinating; no Kubernetes runs on the serving hardware.

In shape, squall is a Kubernetes operator for dstack: the Model CR carries the intent and the lifecycle, dstack owns every provider integration. Squall keeps no list of supported clouds, so any backend dstack speaks is one squall can place on.

Callers see a plain OpenAI-compatible endpoint, so anything that already speaks that API reaches a squall-backed model unchanged.

Highlights

What you get

🔌 Zero idle cost — no request, no machine, no bill ⏱️ Wake on first request — the request is held, not rejected, while the GPU comes up 🌍 Capacity anywhere — any backend dstack supports, in regions your cluster does not run in 📄 Declared in Git — a Model CR, reconciled like everything else on the platform 🔗 OpenAI-compatible — /v1/chat/completions, /v1/completions, /v1/models 🛡️ Safe by construction — a wrong wake costs money, a wrong sleep kills a generation, and every ambiguity is resolved in that direction

What it is not

Squall does not schedule Pods onto GPU nodes. If you already have idle GPUs in your cluster, KubeAI is the better tool and squall buys you nothing. Squall exists for the case where the hardware does not exist until someone asks for it.

Why Squall?

The GPU is not where your cluster is

Plenty of platforms run in a region with no GPU quota, or with GPUs priced so that a permanently-provisioned instance spends most of its life idle and billed. Squall makes the GPU a consequence of traffic rather than a standing cost: the Model sits at zero replicas until a request arrives, and returns to zero once the idle window passes with nothing in flight.

One model, many places to run it

A Model names what to serve and the shape of machine it needs — a VRAM floor, a price ceiling, the regions it may run in. It does not name a provider or a particular machine. Squall asks dstack for the cheapest offer satisfying those constraints, wherever that happens to be today, and the same CR moves between Vast.ai, AWS and DigitalOcean unchanged.

Requests reach the replica through squall-proxy, which forwards over its own SSH tunnel where the topology allows it and falls back to dstack's service proxy where it does not. The choice is automatic and per-request; nothing in the Model selects it. See docs/operating.md for how it authenticates and how to tell which path a request took.

Separate failure domains

Two binaries, deliberately not one:

• squall-controller — reconciliation only: the 0↔1 replica flip, drain-first teardown via finalizer, orphan reconciliation, and status as the shared truth.
• squall-proxy — the per-request data path: forward when the model is Ready, hold while it wakes, and answer the wait contract honestly when the deadline expires.

A crash-looping proxy cannot strand a GPU, and a controller under load cannot drop a request.

Architecture

flowchart LR
    client["OpenAI client"] --> proxy["squall-proxy"]
    proxy -- "1 · records demand" --> model[("Model CR")]
    model --> ctrl["squall-controller"]
    ctrl -- "2 · replicas 0 → 1" --> dstack["dstack server"]
    dstack -- "3 · rents a machine" --> gpu["GPU · Vast.ai / AWS / DO"]
    proxy -. "4 · forwards over its own SSH tunnel" .-> gpu

Loading

1. A request for a sleeping model reaches squall-proxy. It records demand on the Model and holds the request rather than failing it.
2. squall-controller sees the demand and flips the run to one replica.
3. dstack provisions a machine from whichever backend the Model allows, and starts the serving engine on it.
4. The held request is retried against the replica as it comes up — the first real request is the readiness probe. When it succeeds, the caller gets tokens.

When the hold deadline expires before the GPU is ready, the caller gets 503 with Retry-After and a JSON body naming the state (asleep, waking, recreating) — never a silent hang and never a lie.

Idle works in reverse: once nothing is in flight and the newest request is older than idleTimeout, the run goes to zero replicas — and squall always tells dstack idle_duration: 0, so the machine goes with it, on every backend. There is no warm pool anywhere: idle releases both layers at once, and the next wake is always a full cold start. See Send it a request for why that decides your holdTimeout.

Quickstart

Prerequisites

• A Kubernetes cluster — developed and tested against 1.31 — and Helm 3.8+ (for OCI charts)
• An API key for at least one dstack backend — Vast.ai is the cheapest way to try this
• Squall's images and chart are published to GHCR and are public — both pull anonymously, no pull secret needed.

Install

The chart deploys squall and a dstack server — you do not need to run dstack yourself.

helm install squall oci://ghcr.io/ackstorm/charts/squall --version 0.1.6 \
  --namespace squall-system --create-namespace \
  --set-string dstack.adminToken.value="$(openssl rand -hex 16)" \
  --set controller.env.dstackURL="http://dstack.squall-system.svc.cluster.local:3000"

Upgrading from an earlier version: apply the CRD by hand. Helm installs crds/ on install and never upgrades them, so helm upgrade alone leaves the Model CRD at whatever version you first installed — while reporting success. Every field a release adds is missing until you do this, and kubectl apply of a Model using one fails with unknown field:

kubectl apply --server-side -f \
  https://github.com/ackstorm/squall/releases/download/v0.1.6/squall.ackstorm.ai_models.yaml

With no dstack.backends, dstack provisions onto your own cluster — no GPU, no bill, useful for trying the lifecycle end to end. The chart discovers the in-cluster SSH jump address; you do not need to supply a node IP. To rent real hardware, point it at a provider:

# values.yaml
dstack:
  credentialsSecretName: vast-credentials   # must contain VAST_API_KEY
  backends:
    - type: vastai
      creds:
        type: api_key
        api_key: ${VAST_API_KEY}
      # Operator policy: the widest infrastructure scope Models may use.
      regions: [es-spain, pt-portugal, fr-france, it-italy, de-germany,
                nl-netherlands, be-belgium, at-austria, ch-switzerland,
                pl-poland, cz-czechia, se-sweden, fi-finland, no-norway]
      # Optional: false excludes individually operated Community Cloud hosts.
      # community_cloud: false

dstack.backends[].regions is the cluster operator's allow-list. A Model author can narrow it with spec.placement.regions, but cannot widen it: dstack first restricts the backend catalogue and then applies the Model profile, so the effective set is their intersection. Vast.ai regions use dstack/gpuhunt's canonical names such as es-spain, not ISO codes such as ES.

community_cloud also belongs to the Vast.ai backend, not the Model. dstack defaults it to true; setting it to false excludes individual operators and may improve the trust profile at the cost of fewer offers. Both pools are third-party marketplace capacity, so this switch is not a substitute for a workload security or compliance policy.

Create vast-credentials in the chart namespace with a VAST_API_KEY key. The chart imports that Secret as environment variables and replaces ${VAST_API_KEY} in the backend config before dstack starts; the credential does not belong in values.yaml.

Which backends

There is no allow-list to be on. dstack.backends reaches dstack's own server/config.yml verbatim and spec.placement.backends is an unconstrained list of names, so any backend dstack supports is usable from a Model. Bringing up a new provider is a credentials and measurement exercise, not a change to squall.

dstack 0.21.3 ships aws, azure, gcp, oci, nebius, lambda, vastai, runpod, digitalocean, amddevcloud, vultr, cloudrift, crusoe, datacrunch, hotaisle, jarvislabs, seeweb, verda, slurm, kubernetes and dstack Sky.

No property of a backend changes squall's behaviour here: squall always tells dstack idle_duration: 0, so there is no warm pool on any backend, dockerized or not, and every wake is a full cold provision. holdTimeout must cover one regardless of which backend a Model lands on.

Only vastai and kubernetes have been run end to end. The rest are untested rather than unsupported — worth knowing which of the two you are dealing with when something fails.

Helm never upgrades CRDs. Files under a chart's crds/ directory are applied on install and ignored on every upgrade. After a version bump, apply the CRD yourself — it ships as a release asset:

kubectl apply --server-side --force-conflicts -f squall.ackstorm.ai_models.yaml

Declare a model

apiVersion: squall.ackstorm.ai/v1alpha1
kind: Model
metadata:
  name: qwen-tiny
  namespace: squall-system       # any namespace works; this one already exists
spec:
  engine: ollama
  image: ollama/ollama:latest    # a tag is fine to try it; pin a digest for real use
  model: qwen2.5:0.5b
  features: [TextGeneration]

  minReplicas: 0                 # 0 = sleeps when idle; 1 = always on
  holdTimeout: 10m               # how long a request waits for a cold start
  idleTimeout: 10m               # idle window before the run — and the machine — release.
                                 # Also the demand signal's TTL: see Choosing idleTimeout.
  provisioningTimeout: 20m       # a wake that never becomes Ready is destroyed
  drainTimeout: 30s
  maxLifetime: 4h

  resources:                     # dstack's own selector syntax, passed through verbatim
    cpu:
      count: "2.."               # note: an OBJECT here — unlike the chart's fleet block,
    memory: 8GB..                # which is dstack's flat fleet schema (cpu: "2..")
    gpu:
      memory: 8GB..              # a floor, not a pin: cheapest card that clears it

  placement:
    backends: [vastai]           # only backends named here AND in the chart are eligible
    regions: [es-spain, fr-france, nl-netherlands] # narrower than operator policy
    maxPricePerHour: "0.80"      # quoted — an unquoted float fails CRD validation

  health:
    unhealthyAfter: 2m           # takes traffic, delivers no 2xx → torn down
    failureThreshold: 3

kubectl apply -f qwen-tiny.yaml
kubectl -n squall-system get models -w

Worked examples with measured resource and placement choices are below, and as runnable files in config/samples/.

Send it a request

kubectl -n squall-system port-forward svc/squall-proxy 8080:8080 &

curl -s localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen-tiny","messages":[{"role":"user","content":"hello"}]}'

The first call blocks while the GPU is provisioned — minutes, on a real backend — and then answers. Watch it happen:

kubectl -n squall-system get model qwen-tiny -w   # Asleep → Waking → Ready
kubectl -n squall-system logs -f deploy/squall-proxy

Leave it alone and the machine goes away.

How long that takes does not depend on the backend. Squall always tells dstack idle_duration: 0, so the machine is released the moment the run does, on every backend — there is no knob left that changes this. Measured on Vast.ai: instance gone 45 seconds after sleep.

The consequence is not the teardown, it is the wake. There is no warm pool anywhere, so every wake is a full cold provision and holdTimeout has to cover one — a holdTimeout short enough to habitually time out before a cold start finishes is a misconfiguration in all but intent, and squall warns when it looks that short relative to provisioningTimeout.

Choosing idleTimeout

idleTimeout is the only idle knob, and it does three jobs at once. The first two are the obvious ones; the third is the one that surprises people.

1. How long you keep paying with the model hot. The clock starts from the newest request, and only runs once in-flight requests reach zero. A request arriving inside the window is answered by a model already resident in VRAM — sub-second. One arriving outside it pays a full cold start.
2. When the machine is released. Squall always sends dstack idle_duration: 0, so the run and the instance go together. There is no second, longer window keeping a warm machine around — releasing the job releases the hardware, on every backend.
3. How long a demand signal stays valid. When squall-proxy sees a request for a sleeping model it stamps an annotation on the Model, and idleTimeout is that annotation's TTL. This is what makes a too small value dangerous rather than merely expensive.

The cost/latency trade

Every second of idleTimeout is billed at the full GPU rate, whether or not a request arrives. So the knob is really asking: how much idle GPU time is one avoided cold start worth?

Cold starts are not cheap. Measured on Vast.ai with a small fixture: a wake takes ~2m12s, and teardown completes 45 seconds after sleep. Against a $1.89/h card, five minutes of idle costs about $0.16 — usually less than making the next caller wait two minutes.

Rules of thumb:

Traffic	Reasonable idleTimeout
Bursty and interactive (a person, an IDE, a chat UI)	5m–15m — bridge the gaps within a session
Batch, on a schedule	short, or minReplicas: 1 for the duration of the run
Steady all day	minReplicas: 1 — you are paying anyway; stop re-provisioning
Rare and latency-tolerant	1m–5m, and raise holdTimeout so callers wait rather than get a 503

There is a floor, and it is one minute

Because of job 3, an idleTimeout too short to survive until the controller next evaluates the Model makes it permanently unwakeable: the proxy stamps demand, the annotation expires before anyone reads it, and the wake never happens. The request 503s, and every retry does the same — with no error, no event and no condition to explain it.

Squall refuses anything below one minute. A Model under the floor reports Schedulable=False with the reason in status.conditions and is not provisioned, so the failure is visible to kubectl describe model instead of silent.

One minute is a floor under a cliff, not a recommendation. It clears three separate contributors at once:

• up to a second lost to the demand annotation's RFC3339 second granularity;
• the controller's idle requeue interval — 15s by default (SQUALL_IDLE_REQUEUE_INTERVAL);
• cold starts, which are measured in minutes on every backend squall supports.

Pick a real value from the table above rather than the floor; the default is 5m. Measured directly (ledger D171): with a 2s requeue, idleTimeout: 2s never woke at all, while 8s and 30s reached Ready about two seconds after the request.

This only bites when the demand stamp is not refreshed. While squall-proxy is actively holding a request it re-stamps at idleTimeout/10 (floored at 500ms), so a caller waiting out a cold start keeps its own wake alive regardless of how long provisioning takes.

Point LiteLLM at it

Squall speaks the OpenAI API, so LiteLLM needs no plugin:

model_list:
  - model_name: qwen-tiny
    litellm_params:
      model: openai/qwen-tiny
      api_base: http://squall-proxy.squall-system.svc.cluster.local:8080/v1
      api_key: none

Worked examples

Every model here was actually provisioned and served through squall. The numbers are measured on the run, not estimated from a spec sheet.

Model	Card	Price/h	First wake	Verified
Qwen3.8-27B-FP8	1x RTX PRO 6000 WS, 96GB	$2.003	10-15 min	2026-08-27
Qwen3-8B	1x RTX 5090, 32GB	$0.4463	9m 53s	2026-09-04
GLM-5.3-Flash	4x RTX PRO 6000	$9.00 cap	not yet run	—

First wake is dominated by pulling the image and then the weights; a warm request afterwards is ordinary inference latency. On the Qwen3-8B run: 9m 53s cold, 0.8s warm.

Qwen3.8-27B-FP8 — the flagship, on one card

config/samples/squall_v1alpha1_model.yaml is a runnable CR, heavily commented with why each number is what it is. It provisioned an RTX PRO 6000 WS (96GB) on Vast.ai in cz-czechia at $2.003/h, served chat, streaming and vision, then slept and released the instance on its own.

Qwen publish an official FP8 checkpoint, so squall serves that rather than asking vLLM to quantize at load: 28.8 GiB of weights, which is why the GPU floor is 90GB.. rather than something tighter — FP8 KV cache at 128k context needs the headroom.

spec:
  engine: vllm
  image: vllm/vllm-openai@sha256:2286e8533ca8b6bc777594bae30524f1426ba46ca21797524e06df6a94b06635
  model: Qwen/Qwen3.8-27B-FP8
  args: [--max-model-len, "131072", --kv-cache-dtype, fp8, --reasoning-parser, qwen3]
  resources:
    gpu:
      name: [RTXPRO6000WS, RTXPRO6000S]
      memory: 90GB..
  placement:
    backends: [vastai]
    maxPricePerHour: "2.20"
  holdTimeout: 20m               # must cover a full cold start — no warm pool anywhere
  idleTimeout: 5m

Qwen3-8B — the cheap one, for checking the lifecycle costs nothing

config/samples/squall_v1alpha1_model_qwen3_8b.yaml, the same shape at a fifth of the price. Useful when what you want to verify is squall's behaviour rather than a model's quality: the whole exercise that produced these numbers — provision, serve, sleep, release — cost $0.197.

apiVersion: squall.ackstorm.ai/v1alpha1
kind: Model
metadata:
  name: qwen3-8b
  namespace: squall
spec:
  engine: vllm
  image: vllm/vllm-openai@sha256:61fc8a896b0a4fbbbdc063bc4b0dbc25ce98e02b5050c24aeb7830ac02039b14
  model: Qwen/Qwen3-8B
  features: [TextGeneration]
  args: [--max-model-len, "8192", --gpu-memory-utilization, "0.90"]

  resources:
    cpu:
      count: "4.."
    memory: 16GB..
    shmSize: 8GB
    gpu:
      memory: 24GB..             # a floor: an RTX 5090 (32GB) cleared it at $0.4463/h

  placement:
    backends: [vastai]
    maxPricePerHour: "0.80"

  probe:
    path: /health
    interval: 10s
    readyAfter: 2

  minReplicas: 0
  holdTimeout: 20m               # cold start measured at 9m53s — leave room
  idleTimeout: 5m
  provisioningTimeout: 30m

Pin the multi-arch index digest, not a per-architecture one. A @sha256: digest can address either an image index or a single-platform leaf, and squall's schema cannot tell them apart. Pinning an arm64 leaf makes dstack require arm64, Vast.ai matches zero offers, and the failure is reported as failed_to_start_due_to_no_capacity — a message that sends you tuning CPU, memory and price, none of which can fix it. This cost a whole live run. The digest above is the index; get one with docker buildx imagetools inspect <repo>:<tag> and use its top-level Digest:.

GLM-5.3-Flash — four cards, tensor-parallel

config/samples/squall_v1alpha1_model_glm53_flash.yaml requests gpu.count: "4" and 64GB of system RAM, transcribed from the rtx6kpro field wiki. It is written and reviewed but has not been run end to end — treat its numbers as sourced, not measured.

Status and limitations

Pre-alpha. Squall has served production-shaped traffic on rented GPUs, and the worked examples above are real runs. It is not yet something to put in front of untrusted users, for these reasons:

• squall-proxy performs no authentication. Anyone who can reach it can wake capacity and spend money. Deploy it on a trusted network, or put an authenticating gateway in front. This is the constraint every other limitation assumes.
• No admission control once a model is Ready. The proxy forwards unlimited concurrency into whatever capacity exists. Nothing queues, sheds or prioritises, so under load each caller's share of the replica shrinks rather than any request being refused.
• Marketplace hosts are not data processors. Internal, non-regulated workloads only.
• The unhealthy teardown can flap. A model broken by configuration rather than by machine re-wakes on the next request.
• spec.hardStop does not fire on the Kubernetes backend. It is wired correctly — the value reaches dstack's max_duration and the runner's submit body — but it is not enforced: a job with max_duration: 2m was still running at 621s, and one live Model ran 104 minutes against a 60-minute limit. Root-caused to dstack wrapping commands in sh -i -c, so the timeout signals the wrapper shell while the workload is reparented and survives; reported upstream as dstackai/dstack#4259. Do not treat hardStop as the bound that survives a dead controller. Whether VM backends enforce it is untested. Ledger D161.

Full list in the CHANGELOG; the running engineering ledger is docs/references/deviations-and-findings.md.

Documentation

docs/operating.md	Running it: the direct SSH path, serving limits, deployment traps
docs/metrics.md	Prometheus endpoints, metric catalogue and optional ServiceMonitors
docs/specs/	The design of record — state machine, wait contract, provider boundary
docs/references/	Toolchain traps, testing discipline, decisions, the findings ledger
CHANGELOG.md	Release notes and known limitations
rtx6kpro	Community field wiki for large LLMs on the 96GB Blackwell cards squall rents — model runbooks (GLM-5, Kimi, Qwen), quantization and speculative-decoding measurements behind our samples

The spec is authoritative. Read it before changing behaviour; every implementation task references its section numbers.

Development

The only thing your host needs is Docker. No Go toolchain, no kubebuilder, no controller-gen, no kubectl. Squall targets Go 1.26.6, and pinning that on every developer machine is not a thing anyone should have to do — so the whole toolchain lives in a container built from Dockerfile.devtools.

make help                          # every target, self-documenting
make doctor                        # preflight: docker, image, caches, tools. No network.
./scripts/dev.sh go version        # -> go1.26.6
./scripts/dev.sh bash              # interactive shell in the devtools container

make targets that need the toolchain re-enter the container themselves, so make test-unit and ./scripts/dev.sh make test-unit are equivalent. Host-only targets (make doctor, make clean) deliberately do not.

Tests

Target	What it runs
make test-unit	Pure logic. No envtest, no cluster.
make test-envtest	Controller against a real API server, race-enabled. The CI gate.
make test-envtest-fast	Same without -race, for the inner loop. Not a CI gate.
make e2e-full	kind cluster + the ginkgo suite against a fake dstack.

The envtest Kubernetes assets are pre-baked into the devtools image, so envtest never hits the network.

Linux only, by design

scripts/dev.sh requires the default Docker socket at /var/run/docker.sock and runs the container with --network=host. DOCKER_HOST=tcp://…, rootless Docker and Docker Desktop sockets are unsupported, and the wrapper fails fast with a clear message rather than letting a downstream kind call die cryptically several minutes later.

Releasing

Push a v* tag. .github/workflows/release.yml runs the gates, then publishes both images, the Helm chart as an OCI artifact, and a GitHub Release with the chart tarball, the CRD and checksums.

Contributing

See CONTRIBUTING.md and the Code of Conduct. Security reports: SECURITY.md.

License

MIT — see LICENSE. NOTICE records what Squall depends on and under which terms.