Skip to content

Tolerate the AMD GPU and CPU pool taints when counting schedulable nodes - #2

Merged
negz merged 3 commits into
mainfrom
zero-tolerance
Nov 5, 2026
Merged

negz merged 3 commits into
mainfrom
zero-tolerance

Conversation

@negz

@negz negz commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Part of modelplaneai/modelplane#601

The agent counted a node as schedulable only if every NoSchedule or NoExecute taint on it was nvidia.com/gpu. Modelplane's engine pods also tolerate amd.com/gpu and modelplane.ai/cpu, so it wouldn't count any node in an AMD GPU or CPU pool as schedulable.

The agent now tolerates the taints named by a --tolerated-taints flag (the chart's toleratedTaints value), written key[=value]:effect. A taint without a value matches any value, like the Exists tolerations on Modelplane's pods. Both default to:

toleratedTaints:
- nvidia.com/gpu:NoSchedule
- amd.com/gpu:NoSchedule
- modelplane.ai/cpu:NoSchedule

The first commit makes each controller's Setup take ReconcilerOptions rather than a positional logger, so main passes the configured pool.Observer with WithPoolObserver. pool.NewObserver takes a WithTolerations option, and tolerates nothing without it.

Once we release this, Modelplane needs to bump the agent and drop the NVIDIA-only filter in compose-inference-cluster.

negz added 3 commits November 4, 2026 19:28
Each controller's Setup took a logger positionally and built its
Reconciler's options itself, so configuring a Reconciler any other way
meant adding another positional argument to Setup. Setup now takes the
same ReconcilerOptions NewReconciler does, and passes them through.

Signed-off-by: Nic Cope <nicc@rk0n.org>
The agent counted a node as schedulable only if every NoSchedule or
NoExecute taint on it was nvidia.com/gpu with NoSchedule. Modelplane's
engine pods also tolerate amd.com/gpu and modelplane.ai/cpu, so the agent
would report AMD GPU and CPU pools as having no schedulable nodes. To
avoid that Modelplane only asks the agent to observe NVIDIA pools, and
falls back to whole-node scheduling on the others (modelplaneai/modelplane#601).

The agent now tolerates the taints its --tolerated-taints flag names,
which default to those three. A Modelplane that installs it can observe
every pool.

Signed-off-by: Nic Cope <nicc@rk0n.org>
Signed-off-by: Nic Cope <nicc@rk0n.org>
@negz
negz merged commit a833d3f into main Nov 5, 2026
3 checks passed
@negz
negz deleted the zero-tolerance branch November 5, 2026 03:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant