Skip to content

Share a GPU between replicas on DRA consumable capacity - #680

Draft
kaessert wants to merge 4 commits into
modelplaneai:mainfrom
kaessert:gpu-sharing
Draft

kaessert wants to merge 4 commits into
modelplaneai:mainfrom
kaessert:gpu-sharing

Conversation

@kaessert

Copy link
Copy Markdown

Towards #644.

The fleet scheduler won't place two replicas on one GPU, even where the GPU's DRA driver lets several claims share it. On a k3s 1.35 cluster with DRAConsumableCapacity on and the NVIDIA DRA driver's consumable shares in unlimited mode, DRA allocated one GPU to a ModelDeployment's claim and a plain pod's claim at once, but a second ModelDeployment still reported InsufficientCapacity: the cluster agent counts a GPU as taken once any claim holds it, a deployment has no way to ask for a share, and an engine no way to learn how much of the GPU's memory it may use.

This adds fractional sharing on DRA's consumable-capacity path, in DRA's own terms. An InferenceClass device can set allowMultipleAllocations, and a member's nodeSelector request can set capacity.requests. A request that asks for capacity matches only a device that allows multiple allocations and declares at least that much of each capacity, checked while matching since a deployment doesn't name its class. The InferenceCluster publishes the capacity each shared device has left in every bucket of a pool's histogram, and the scheduler fits a share across count distinct devices, taking shared devices fullest-first and falling back to whole free ones. The request is copied onto the ModelReplica and into its ResourceClaimTemplate, so each pod gets a claim of its own and DRA accounts for every share. When exactly one of a member's requests asks for memory, its engine container gets MODELPLANE_DEVICE_MEMORY_FRACTION for an arg like --gpu-memory-utilization=$(MODELPLANE_DEVICE_MEMORY_FRACTION) to reference, since DRA accounts for a share but doesn't confine the engine to it.

A preparatory commit names a ResourceClaimTemplate for a hash of its spec. The spec is immutable, so a changed device request otherwise failed to apply with "field is immutable" and the pods kept claiming the old devices.

design/gpu-sharing.md has the full proposal: the before/after API and histogram shapes, and the alternatives considered (one shared ResourceClaim per pool; writing the share into known engines' args). Turning sharing on stays with the platform team, through the feature gate and the driver's sharing mode.

Not run here: the e2e suite (add the test-e2e label). nix flake check's unit, type, style, and docs checks pass.

The fleet scheduler won't place two replicas on one GPU, even where the
GPU's DRA driver lets several claims share it through consumable
capacity. On a k3s 1.35 cluster with DRAConsumableCapacity on and the
NVIDIA DRA driver's consumable shares in unlimited mode, DRA allocated
one GPU to a ModelDeployment's claim and a plain pod's claim, while a
second ModelDeployment reported InsufficientCapacity: the cluster agent
counts a GPU as taken once any claim holds it.

This proposes that a class mark a device as shareable with DRA's
allowMultipleAllocations, a request ask for a share with DRA's
capacity.requests, the agent report the capacity each shared device has
left, and the scheduler fit shares against it. Each pod keeps a
ResourceClaim of its own, so DRA allocates and accounts for every
share. The engine learns its share of the device's memory from
MODELPLANE_DEVICE_MEMORY_FRACTION.

Towards modelplaneai#644.

Signed-off-by: Tobias Kässer <tobias.kasser@upbound.io>
A ModelDeployment couldn't ask for part of a GPU, and the scheduler
counted every GPU a claim held as taken, so two replicas never shared
one even where the GPU's DRA driver allows multiple allocations.

An InferenceClass device can now set allowMultipleAllocations, and a
nodeSelector request can set capacity.requests, both as DRA names them.
A request that sets capacity.requests matches only a device that allows
multiple allocations and declares at least that much of each capacity.
A deployment doesn't name its class, so this is checked when matching
rather than by CEL on the XRD. The request is copied onto the
ModelReplica and into the member's ResourceClaimTemplate, so each pod
gets a claim of its own and DRA accounts for every share.

The InferenceCluster can now publish, in each bucket of a pool's
histogram, the capacity each of its shared devices has left. The
scheduler fits a share on count distinct devices, each either a shared
device with enough left or a free one, which has the capacity its class
declares. It takes shared devices first, fullest first. A request
without capacity.requests still takes whole free devices. That's
stricter than the NVIDIA driver's unlimited mode, where such a claim
consumes nothing, and matches DRA without a request policy and the
driver's memory mode. The scheduler can't tell which mode a driver runs
in, and this guess can't overcommit a device.

compose-inference-cluster copies each bucket from the pool's
PoolObservation as it is, so the shared capacity reaches the scheduler
once the cluster agent reports it. Until then a shared device looks
taken, as it did before.

Towards modelplaneai#644.

Signed-off-by: Tobias Kässer <tobias.kasser@upbound.io>
DRA accounts for a request's share of a device but doesn't confine the
engine to it, and the NVIDIA DRA driver doesn't read a claim's consumed
capacity when it prepares a GPU. vLLM, for one, takes a fraction of the
whole GPU's memory, 0.9 by default, so an engine on a shared GPU can
take memory promised to the device's other claims.

When exactly one of a member's device requests asks for memory, its
engine container now gets MODELPLANE_DEVICE_MEMORY_FRACTION: the
requested memory over the memory the device's class declares, rounded
down to three places. It goes ahead of the user's env entries, next to
MODELPLANE_SERVED_MODEL_NAME, so an arg like
--gpu-memory-utilization=$(MODELPLANE_DEVICE_MEMORY_FRACTION) resolves.
Modelplane passes engine args through as written, so it sets a variable
rather than a flag of a particular engine. One variable can't describe
two devices, so a member requesting memory on two requests gets none.

Towards modelplaneai#644.

Signed-off-by: Tobias Kässer <tobias.kasser@upbound.io>
A ResourceClaimTemplate's spec is immutable, so a change to a member's
device requests, such as its capacity.requests, count, selectors or
DeviceClass, or the cpu request a dra.cpu claim is sized from, couldn't
be applied to the template the member's pods claim through.
provider-kubernetes failed to update it with "field is immutable", and
the pods kept claiming the old devices until someone deleted the
template by hand.

The template's name and its composed resource key now carry a short
hash of its spec. A changed spec composes a new template under a new
name, the member's pods, which reference the template by that name,
roll onto it, and Crossplane deletes the Object composing the old one.

Towards modelplaneai#644.

Signed-off-by: Tobias Kässer <tobias.kasser@upbound.io>
@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-680.vercel.app (ready once the site's Content workflow finishes)

@negz

negz commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

/review

Thanks for looking into this. I'm considering a larger redesign of how we approach scheduling - TBD how it'll interact with this.

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs a step back before the details matter. Mirroring DRA's allowMultipleAllocations and capacity.requests, and handing the engine its share through a MODELPLANE_ variable, fit the committed designs. But the scheduler half can't take effect until cluster-agent reports a shared device's free capacity.

#644 asks the maintainers whether to place on shared claims now or wait for consumable capacity, and whether to inject the memory fraction or write engine args. It also promises numbers on the cost to GLM before a PR. None of that has an answer on the issue yet, and the PR settles both questions itself. In a comment here, a maintainer said a scheduling redesign may change how this fits. Could we settle those on the issue first, then decide whether the API and claim plumbing should land apart from the scheduling.py changes?

The PR description says the InferenceCluster publishes each shared device's free capacity, which nothing does yet.

Reviewed by modelplane-review 0f30f2d. Reply, or react 👍 or 👎, on each comment to help tune it.

additionalProperties:
type: integer
minimum: 0
sharedDevices:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What writes sharedDevices? compose-inference-cluster rebuilds each bucket from the agent's NodesByFreeDevice (fn.py:2618), and the vendored PoolObservation schema has only freeDevices and nodes. Nothing here or in cluster-agent writes it. So once a share's replica settles, its GPU shows as taken and a second share gets InsufficientCapacity, the failure the PR sets out to fix.

docs/content/architecture/scheduling.md describes sharedDevices as working today. Could this wait on the agent change, or could the design and docs say what the agent must report and that the change belongs in cluster-agent?

Comment thread design/gpu-sharing.md
nodesByFreeDevices:
- freeDevices:
gpu.nvidia.com: 0
sharedDevices: # New.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will free capacity blow up the histogram's bucket count? free-gpu-scheduling.md relies on nodes with the same free devices sharing a bucket, and falls back to whole-node accounting past 256 combinations. Free memory per shared GPU varies with every mix of share sizes, so on a large pool I'd expect few nodes to match exactly, and sharing to switch off with no signal. Could the agent round free capacity down to a coarser unit, or could the design argue the pools it targets stay under the limit?

Comment thread design/gpu-sharing.md
devices, each a shared device with that much free or a device in
`freeDevices`, which has the capacity its class declares. It takes shared
devices first, the fullest first, guessing that DRA packs a device before
starting another. A request that sets no `capacity.requests` takes whole

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does #644's own case work under this? There, GLM runs TP8 on whole B300s, and a small model shares those GPUs. A request without capacity.requests takes whole devices from freeDevices, so nothing can join GPUs GLM holds. GLM would have to request most of each GPU's memory as a share itself, and its engine would then need the fraction too. Could the design walk through the big-plus-small case and say that every tenant of a shared GPU must request a share?

selectors = [{"cel": {"expression": s.cel}} for s in (r.selectors or []) if s.cel]
if selectors:
exactly["selectors"] = selectors
if r.capacity:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a whole-device replica on a shareable class lose exclusive use of its GPU? Its claim doesn't request capacity, and DRA consumes the request policy's default for a capacity a claim doesn't name (fillEmptyRequest). That's zero in NVIDIA's unlimited mode, the mode the design tested. DRA can then put a 24Gi share on a GPU a vLLM engine at 0.9 utilisation already uses, while the scheduler counts that GPU as taken. Could a request without capacity on an allowMultipleAllocations device ask for the device's full declared capacity?

Comment thread design/gpu-sharing.md
doesn't write engine flags, so the variable is for the deployment's args to
reference, as in `--gpu-memory-utilization=$(MODELPLANE_DEVICE_MEMORY_FRACTION)`.

Turning sharing on stays with the platform team: the feature gate, and the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where can a platform team turn this on? The serving stack composes the NVIDIA DRA driver's Helm release on every cloud, Existing included (functions/compose-serving-stack/function/stacks/clouds/existing.py:105), with fixed values. A team that sets the driver's ConsumableShares feature gate or its consumableShares mode on that release has it undone at the next reconcile. Deferring driver configuration to Future improvements then leaves sharing usable only where the platform team installs the driver itself. Should configuring the driver's sharing mode from the InferenceCluster be in scope, or should the design state that limit?

if r.capacity:
# A share of a device that allows multiple allocations, as the
# deployment's nodeSelector requested it.
exactly["capacity"] = {"requests": dict(r.capacity.requests)}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should dra.cpu devices go through the same consumable-capacity path? On a CPU class that sets allowMultipleAllocations, the scheduler would charge the user's capacity.requests, while the dra.cpu branch below overwrites it with the container's CPU request. dra-driver-cpu's NUMA devices are already consumable, yet scheduling.md says a pool fits "at most one CPU replica per NUMA node, whatever number of cores each claims", the over-counting this PR fixes for GPUs. Could CPU devices use the new accounting, or could the design say why the two paths differ?

return all(name in have and have[name] >= amount for name, amount in wants)


def _consume(capacity: _Capacity, wants: _Capacity) -> _Capacity:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this undercount a capacity the request doesn't name? The rule from the comment on base.py:1003 applies here too: DRA consumes every capacity a device declares (CmpRequestOverCapacity), but _consume subtracts only the requested names. With a hypothetical class declaring memory: 80Gi and a shares: "2" capacity whose default is 1, eight 10Gi requests fit one GPU where DRA allows two. Could the scheduler treat an unnamed capacity as consumed in full, the stricter guess the design already takes for requests without capacity?

maxProperties: 8
additionalProperties:
type: string
pattern: '^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this pattern allow a sign? It accepts -24Gi, which _shareable matches and _consume adds to the device's free capacity. Until the replica settles, the ledger then shows other deployments more free memory than the GPU has. Could we drop the leading (\+|-)?, or add a CEL rule that the quantity is positive?

total = dict(_device_capacity(device)).get(_CAPACITY_MEMORY)
if requested is None or not total:
return ""
fraction = (requested / total).quantize(decimal.Decimal("0.001"), rounding=decimal.ROUND_DOWN)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't a share under 0.1% of the device inject MODELPLANE_DEVICE_MEMORY_FRACTION=0? 1Mi of 80Gi rounds down to 0.000, which normalises to "0". That's a non-empty string, so _inject_device_memory_fraction sets it, and --gpu-memory-utilization=0 stops vLLM starting. Could a zero fraction count as no fraction, or could matching reject a request too small to express?

roll onto it. child_name truncates the name to a DNS label.
"""
return resource.child_name(_name(replica.metadata), engine.name, member_role(member), _POD_CLAIM_NAME)
return resource.child_name(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should naming the template for its spec be its own PR? It changes every deployment that claims devices, not only shared ones: any edit to a device request now composes a new template under a new key and rolls the member's pods. The "field is immutable" failure it fixes exists without sharing, and design/gpu-sharing.md doesn't mention it. Could we move it to its own PR, so it can land whatever happens to the scheduling side?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants