Skip to content

fix(api): do not count DRA admin access GPU claims as GPU usage - #26

Open
brokenlander wants to merge 1 commit into
kai-scheduler:mainfrom
brokenlander:fix/dra-admin-access-gpu-count
Open

brokenlander wants to merge 1 commit into
kai-scheduler:mainfrom
brokenlander:fix/dra-admin-access-gpu-count

Conversation

@brokenlander

Copy link
Copy Markdown

Description

DRA claims with adminAccess: true do not consume a device: the API lets them share a device with a regular allocation, and kube-scheduler's allocated-device tracking skips them. The GPU helpers here counted them anyway. The NVIDIA GPU Operator in DRA mode keeps admin access claims on every GPU node (DRA validator, DCGM exporter), so in KAI a one-GPU node looked fully used and every GPU pod was rejected with didn't have enough resources: GPUs.

countGPUDevicesFromClaim and getGPUDeviceClassNameFromClaim now skip requests with AdminAccess set, through one isGPUConsumingRequest check. That carries through ExtractDRAGPUResourcesFromClaims and DRAGPUResourceListFromClaims, which KAI uses for the pod's GPU request and the podgroup's requested and allocated resources.

This is the dra.go half of kai-scheduler/KAI-Scheduler#2237, moved here as asked in that PR. The other half (dra_shared_device_info.go) stays in KAI and goes in with the module bump.

Related Issues

Part of kai-scheduler/KAI-Scheduler#2236

Checklist

  • Self-reviewed
  • Added/updated tests (if needed)
  • Updated documentation (if needed)
  • Added a changelog fragment via make changelog (or applied the skip-changelog label). Do not edit CHANGELOG.md directly; pending fragments are folded into it at release time.

Breaking Changes

None. Claims without admin access are counted as before.

Additional Notes

Tests in dra_gpu_test.go: ExactCount and All admin requests count 0, a mixed claim counts only its regular request, and admin claims are left out of ExtractDRAGPUResourcesFromClaims and DRAGPUResourceListFromClaims. They fail on main and pass with the fix.

Checked against KAI main (049d1ba6, on v0.2.0) with a replace pointing at this branch: the tests from kai-scheduler/KAI-Scheduler#2237 in node_info and pod_info pass. With the replace pointing at unpatched main, the two admin access tests in node_info fail. make validate and make test pass, and go mod tidy leaves go.mod unchanged.

Signed-off-by: brokenlander <94259413+brokenlander@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants