Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions apis/inferenceclasses/definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -611,6 +611,18 @@ spec:
default: 1
minimum: 1
maximum: 64
allowMultipleAllocations:
type: boolean
description: >-
Whether several claims can allocate one of these
devices at once, each consuming part of its capacity,
as DRA's consumable capacity (KEP-5075) allows. Set it
only when the DRA driver publishes the device with
allowMultipleAllocations. A ModelDeployment request
that sets capacity.requests matches only a device that
allows multiple allocations, and is scheduled by the
capacity its device has free. Only a claim: DRA device
can set it. Omitted, the scheduler counts the device whole.
attributes:
type: object
description: >-
Expand Down Expand Up @@ -655,6 +667,8 @@ spec:
x-kubernetes-validations:
- rule: "self.claim != 'DRA' || has(self.deviceClassName)"
message: "deviceClassName is required when claim is DRA"
- rule: "self.claim == 'DRA' || !has(self.allowMultipleAllocations) || !self.allowMultipleAllocations"
message: "allowMultipleAllocations requires claim: DRA"
status:
type: object
properties:
Expand Down
29 changes: 29 additions & 0 deletions apis/inferenceclusters/definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -870,6 +870,33 @@ spec:
additionalProperties:
type: integer
minimum: 0
sharedDevices:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What writes sharedDevices? compose-inference-cluster rebuilds each bucket from the agent's NodesByFreeDevice (fn.py:2618), and the vendored PoolObservation schema has only freeDevices and nodes. Nothing here or in cluster-agent writes it. So once a share's replica settles, its GPU shows as taken and a second share gets InsufficientCapacity, the failure the PR sets out to fix.

docs/content/architecture/scheduling.md describes sharedDevices as working today. Could this wait on the agent change, or could the design and docs say what the agent must report and that the change belongs in cluster-agent?

type: object
description: >-
The free capacity of each of the nodes' devices
that allows multiple allocations and that a
claim has allocated, keyed by DeviceClass. Such a
device doesn't count in freeDevices. One no claim
has allocated does, with all of its capacity
free.
maxProperties: 32
additionalProperties:
type: array
maxItems: 64
x-kubernetes-list-type: atomic
items:
type: object
properties:
freeCapacity:
type: object
description: >-
The device's capacity no claim has
consumed, keyed by the names its driver
publishes capacity under.
maxProperties: 32
additionalProperties:
type: string
maxLength: 32
nodes:
type: integer
minimum: 1
Expand Down Expand Up @@ -900,6 +927,8 @@ spec:
maxLength: 253
count:
type: integer
allowMultipleAllocations:
type: boolean
attributes:
type: object
maxProperties: 32
Expand Down
37 changes: 37 additions & 0 deletions apis/modeldeployments/definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -542,6 +542,43 @@ spec:
device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("141Gi")) >= 0.
minLength: 1
maxLength: 10240
capacity:
type: object
description: >-
The share of each device's capacity the
request consumes, mirroring a DRA
DeviceRequest's capacity and carried
through to the ResourceClaim. Setting it
makes the request match only a device
whose InferenceClass sets
allowMultipleAllocations and declares at
least the requested amount of each
capacity. The scheduler then places the
request on devices with that much
capacity free, so several requests can
share one device. Omitted, the request
takes whole devices.
required: [requests]
properties:
requests:
type: object
description: >-
The capacity to consume on each
device, keyed by the names the
device declares capacity under (e.g.
memory); values are Kubernetes
Quantities. When exactly one of a
member's requests asks for memory,
its engine container gets
MODELPLANE_DEVICE_MEMORY_FRACTION:
the requested memory over the memory
the device's class declares.
minProperties: 1
maxProperties: 8
additionalProperties:
type: string
pattern: '^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this pattern allow a sign? It accepts -24Gi, which _shareable matches and _consume adds to the device's free capacity. Until the replica settles, the ledger then shows other deployments more free memory than the GPU has. Could we drop the leading (\+|-)?, or add a CEL rule that the quantity is positive?

maxLength: 32
# This object has no schema default on purpose: the
# apiserver applies defaults before CEL validation, so
# defaulting it would inject it onto Standalone and
Expand Down
15 changes: 15 additions & 0 deletions apis/modelreplicas/definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,21 @@ spec:
type: string
minLength: 1
maxLength: 10240
capacity:
type: object
description: >-
The capacity to consume on each device,
copied verbatim from the nodeSelector
request into the DeviceRequest.
required: [requests]
properties:
requests:
type: object
minProperties: 1
maxProperties: 8
additionalProperties:
type: string
maxLength: 32
worker:
type: object
# No schema default, mirroring the ModelDeployment
Expand Down
135 changes: 135 additions & 0 deletions design/gpu-sharing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Sharing a GPU between replicas

**Status:** Draft
**Date:** November 2026
**Author:** Tobias Kässer (@kaessert)

This document proposes that the fleet scheduler place replicas by the capacity
a GPU has free, where the GPU's DRA driver lets several claims share it, so two
small models can run on one GPU
([#644](https://github.com/modelplaneai/modelplane/issues/644)). It builds on
[free-gpu-scheduling.md](./free-gpu-scheduling.md).

## Background

DRA's consumable capacity
([KEP-5075](https://github.com/kubernetes/enhancements/issues/5075)) lets a
driver publish a device with `allowMultipleAllocations`, and a claim request a
share of its capacity with `capacity.requests`. The scheduler records what each
claim consumed in `status.allocation.devices.results[].consumedCapacity`, and
allocates the device to further claims until their shares add up to its
capacity. It's alpha behind the `DRAConsumableCapacity` feature gate in
Kubernetes 1.34 and 1.35, and on by default from 1.36. The NVIDIA DRA driver
publishes its GPUs this way with its `ConsumableShares` feature gate and a
`consumableShares` mode set. In its
`unlimited` mode a claim that requests no memory consumes none, and in its
`memory` mode it consumes the whole GPU.

On a k3s 1.35 cluster with the gate on, and NVIDIA's DRA driver v0.5.0 in
`unlimited` mode, a ModelDeployment's claim and a plain pod's claim were both
allocated the same GPU, each with a share of its own. A second ModelDeployment didn't schedule, reporting
`InsufficientCapacity`. The cluster agent counts a device as taken once any
claim has allocated it, so the pool's `nodesByFreeDevices` showed no GPU free.
A ModelDeployment also has no way to ask for a share, and an engine no way to
learn how much of the GPU's memory it may use.

## Proposal

A class marks the devices that several claims may share, and a deployment's
request states how much of each device it consumes, both in DRA's terms:

```yaml
# The InferenceClass.
spec:
devices:
- name: gpu
driver: gpu.nvidia.com
deviceClassName: gpu.nvidia.com
count: 1
allowMultipleAllocations: true # New.
capacity:
memory: { value: "81559Mi" }
---
# A ModelDeployment member.
nodeSelector:
devices:
- name: gpu
count: 1
selectors:
- cel: device.driver == "gpu.nvidia.com"
capacity: # New.
requests:
memory: 24Gi
```

A request that sets `capacity.requests` matches only a device that allows
multiple allocations and declares at least the requested amount of each
capacity. A deployment doesn't name a class, so CEL on its XRD can't check
this, and the check is part of matching.

The cluster agent reports, in each bucket of a pool's histogram, the capacity
each shared device has left. A shared device is one that allows multiple
allocations and that a claim has allocated. Its free capacity is its capacity
less the claims' `consumedCapacity`, and none of it if a claim has no recorded
consumption. It still doesn't count in `freeDevices`, so `freeDevices` keeps
meaning devices no claim holds:

```yaml
nodesByFreeDevices:
- freeDevices:
gpu.nvidia.com: 0
sharedDevices: # New.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will free capacity blow up the histogram's bucket count? free-gpu-scheduling.md relies on nodes with the same free devices sharing a bucket, and falls back to whole-node accounting past 256 combinations. Free memory per shared GPU varies with every mix of share sizes, so on a large pool I'd expect few nodes to match exactly, and sharing to switch off with no signal. Could the agent round free capacity down to a coarser unit, or could the design argue the pools it targets stay under the limit?

gpu.nvidia.com:
- freeCapacity:
memory: 56Gi
nodes: 1
```

The scheduler fits a request that sets `capacity.requests` on `count` distinct
devices, each a shared device with that much free or a device in
`freeDevices`, which has the capacity its class declares. It takes shared
devices first, the fullest first, guessing that DRA packs a device before
starting another. A request that sets no `capacity.requests` takes whole

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does #644's own case work under this? There, GLM runs TP8 on whole B300s, and a small model shares those GPUs. A request without capacity.requests takes whole devices from freeDevices, so nothing can join GPUs GLM holds. GLM would have to request most of each GPU's memory as a share itself, and its engine would then need the fraction too. Could the design walk through the big-plus-small case and say that every tenant of a shared GPU must request a share?

devices from `freeDevices`, as it does on any pool. That's stricter than DRA in
NVIDIA's `unlimited` mode, where such a claim consumes nothing, and the same as
DRA without a request policy or in `memory` mode. The scheduler can't see which
mode a driver runs in, and the stricter guess can't overcommit a device. A pool
the agent doesn't report on is still counted in whole nodes.

compose-model-deployment copies `capacity.requests` onto the ModelReplica's
device requests, and compose-model-replica into the `ResourceClaimTemplate`. So
each pod gets a claim of its own, and DRA places and accounts for each share.
When exactly one of a member's requests asks for memory, its engine container
gets `MODELPLANE_DEVICE_MEMORY_FRACTION`, the requested memory over the memory
the class declares, rounded down to three places. DRA accounts for a share but
doesn't confine an engine to it, so Modelplane tells the engine. Modelplane
doesn't write engine flags, so the variable is for the deployment's args to
reference, as in `--gpu-memory-utilization=$(MODELPLANE_DEVICE_MEMORY_FRACTION)`.

Turning sharing on stays with the platform team: the feature gate, and the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where can a platform team turn this on? The serving stack composes the NVIDIA DRA driver's Helm release on every cloud, Existing included (functions/compose-serving-stack/function/stacks/clouds/existing.py:105), with fixed values. A team that sets the driver's ConsumableShares feature gate or its consumableShares mode on that release has it undone at the next reconcile. Deferring driver configuration to Future improvements then leaves sharing usable only where the platform team installs the driver itself. Should configuring the driver's sharing mode from the InferenceCluster be in scope, or should the design state that limit?

driver's sharing mode.

## Future improvements

The serving stack could configure the NVIDIA DRA driver's sharing mode from the
InferenceCluster on clusters Modelplane provisions.

Per-replica GPU metrics would need the share each replica's pods hold, which
the ReplicaObservation could report from the same `consumedCapacity`.

## Alternatives considered

### Share one ResourceClaim per pool

Every replica sharing a GPU could reference one ResourceClaim, which works
without consumable capacity. DRA would then account for nothing: each replica's
share would be Modelplane's bookkeeping alone, and every pod on the claim would
see the same devices and have to run on their node. The
`capacity.requests` field doesn't preclude adding it later for clusters before
1.36.

### Write the share into known engines' args

compose-model-replica could add `--gpu-memory-utilization` for vLLM and its
equivalent for other engines. Modelplane passes engine args through as written
and doesn't know any engine by name, and a variable keeps it that way.
21 changes: 21 additions & 0 deletions docs/content/architecture/scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,27 @@ NUMA node, with that node's cores as its capacity, and the agent counts a
device as taken once any pod claims some of its cores. A pool's nodes fit at
most one CPU replica per NUMA node, whatever number of cores each claims.

A device whose class sets `allowMultipleAllocations` can be shared. Once a pod
claims part of one, the agent stops counting it in `freeDevices` and reports
the capacity it has left instead, under `sharedDevices`:

```yaml
nodesByFreeDevices:
- freeDevices:
gpu.nvidia.com: 0
sharedDevices:
gpu.nvidia.com:
- freeCapacity:
memory: 56Gi
nodes: 1
```

A pod whose request sets `capacity.requests` fits a device with that much free,
either a shared one or one in `freeDevices`, which has all the capacity its
class declares. One that sets none takes whole devices from `freeDevices`, as on
any other pool. That's stricter than DRA when the driver consumes nothing for a
claim that requests nothing, and the same when it consumes the whole device.

The scheduler counts a pool in whole nodes when the agent hasn't reported on it
or can't observe it. Such a pool publishes no `observedNodes`, and every node
it can have starts empty. A pod that claims any device takes a whole one.
Expand Down
28 changes: 28 additions & 0 deletions docs/content/models/model-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,34 @@ nodeSelector:
- cel: device.attributes["nic.nvidia.com"].linkType == "infiniband"
```

### Sharing a GPU

When the platform team's class allows a GPU to be shared, several pods can each
take part of its memory instead of a whole GPU. Request the share with
`capacity.requests`:

```yaml {nocopy=true}
nodeSelector:
devices:
- name: gpu
count: 1
selectors:
- cel: device.driver == "gpu.nvidia.com"
capacity:
requests:
memory: 24Gi
```

The request matches only a device whose class sets `allowMultipleAllocations`
and declares at least that much memory, and the scheduler places it on a GPU
with that much memory free. Modelplane copies `capacity.requests` into the pod's
`ResourceClaim`, and sets `MODELPLANE_DEVICE_MEMORY_FRACTION` in the engine
container to the requested memory over the device's, `0.3` for 24Gi of an 80Gi
GPU. DRA accounts for the share but doesn't confine the engine to it, so pass it
to the engine, such as with vLLM's
`--gpu-memory-utilization=$(MODELPLANE_DEVICE_MEMORY_FRACTION)`. A request without
`capacity` takes whole GPUs, even on a class that allows sharing.

### What you can match on

Each selector is evaluated against one device and must return a boolean. The
Expand Down
27 changes: 27 additions & 0 deletions docs/content/platform/inference-class.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,33 @@ Each device sets a `claim` discriminator:
hardware that matters for placement but has no DRA driver yet, like an
InfiniBand fabric.

## Shared devices

Set `allowMultipleAllocations: true` on a `DRA` device whose driver publishes it
that way. Several pods can then share the device, each consuming part of its
capacity. This is Kubernetes' consumable capacity
([KEP-5075](https://github.com/kubernetes/enhancements/issues/5075)). On
Kubernetes 1.34 and 1.35 it needs the `DRAConsumableCapacity` feature gate on the
API server, controller manager, scheduler, and Kubelet. From 1.36 it's on by
default. The NVIDIA DRA driver publishes GPUs this way with its
`ConsumableShares` feature gate and a `consumableShares` mode set. A claim that
doesn't request memory consumes none of a GPU in the driver's `unlimited` mode,
and the whole GPU in its `memory` mode.

```yaml {nocopy=true}
devices:
- name: gpu
driver: gpu.nvidia.com
deviceClassName: gpu.nvidia.com
count: 1
allowMultipleAllocations: true
capacity:
memory: { value: "81559Mi" }
```

A `ModelDeployment` takes a share of such a device with `capacity.requests`, and
the scheduler places it by the capacity each device has free.

## The device contract

The `driver`, attribute keys, and capacity keys a class declares are a contract
Expand Down
Loading
Loading