You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I hit this using gVisor pods with Kubernetes in-place pod resize, the same way as #14141 but
for memory. When I raise a pod's memory limit at runtime, the host cgroup is updated
correctly, but the guest keeps reporting the memory it had at boot, so anything that sizes
itself from what the guest reports stays capped at the old value and never uses the memory I
just added, unless I recreate the pod.
Two separate guest-visible signals are stale here, and they matter to different consumers:
/proc/meminfoMemTotal (and sysinfo(2)), read by free-style tooling and by anything
sizing itself from total system memory.
The sandbox's own cgroup limit, /sys/fs/cgroup/.../memory.limit_in_bytes, which is seeded
once when the sandboxed cgroupfs is mounted. Container-aware runtimes prefer this over MemTotal — the JVM's UseContainerSupport and .NET's GC heap limit read the cgroup, and
I believe Node's container heuristics do too, though I am less sure of that one.
I originally wrote this issue as though MemTotal were the signal those runtimes read. That
was wrong, and it matters for scoping: updating MemTotal alone would leave the JVM case
unfixed, and would also make the two signals disagree where today they are at least
consistently stale.
As far as I can tell this is the memory counterpart of #14141: runsc update already accepts --memory and the host cgroup does change, but nothing tells the running sentry, so usage.MaximumTotalMemoryBytes is still whatever was computed at sandbox boot. Note #14141's
fix has the same shape on the CPU side — it updates the reported CPU count and leaves the
guest's cpu.cfs_quota_us stale — so I would value a maintainer's steer on whether the
cgroup view is meant to be in scope for either.
Environment:
runsc version release-20260817.0 (spec 1.2.1), systrap platform
kind v0.31.0, Kubernetes v1.35.0 (in-place resize is GA), containerd 2.2.0
node image kindest/node:v1.35.0
node: Linux 6.12.76-linuxkit aarch64
Steps to reproduce. Boot a gVisor pod with a 128Mi memory limit, then resize it to 2Gi in
place (no restart):
Expected MemTotal to follow the new limit (~2 GiB), the way nproc is expected to follow a
CPU resize in #14141.
The memory is genuinely usable — only the reported total is stale. Writing 1.5 GiB into tmpfs
inside the same sandbox succeeds while it still reports 128 MiB total:
So this is a guest-visibility problem rather than an enforcement one, which makes it the
same shape as #14141: the limit is applied, the guest just cannot see it.
Pointer to the relevant code, in case it helps: runsc/boot/loader.go sets both bounds once
at boot,
and usage.TotalMemory() clamps to MaximumTotalMemoryBytes, which is what /proc/meminfo renders. Nothing updates that ceiling after boot, so it looks analogous to ApplicationCores before #14277.
Happy to work on a patch along the same lines as #14277 (an updater wired through the
existing runsc update --memory path) if that direction seems right. Two things I would want
guidance on:
Whether this should be grow-only, as runsc: update sentry CPU count on live cgroup CPU changes #14277 is. Lowering the reported total below what is
already allocated would make MemFree meaningless — TotalMemory() already has a special
case for memSize < used.
What to do about MinimumTotalMemoryBytes, since boot sets both bounds to the same value.
I hit this using gVisor pods with Kubernetes in-place pod resize, the same way as #14141 but
for memory. When I raise a pod's memory limit at runtime, the host cgroup is updated
correctly, but the guest keeps reporting the memory it had at boot, so anything that sizes
itself from what the guest reports stays capped at the old value and never uses the memory I
just added, unless I recreate the pod.
Two separate guest-visible signals are stale here, and they matter to different consumers:
/proc/meminfoMemTotal(andsysinfo(2)), read byfree-style tooling and by anythingsizing itself from total system memory.
/sys/fs/cgroup/.../memory.limit_in_bytes, which is seededonce when the sandboxed cgroupfs is mounted. Container-aware runtimes prefer this over
MemTotal— the JVM'sUseContainerSupportand .NET's GC heap limit read the cgroup, andI believe Node's container heuristics do too, though I am less sure of that one.
I originally wrote this issue as though
MemTotalwere the signal those runtimes read. Thatwas wrong, and it matters for scoping: updating
MemTotalalone would leave the JVM caseunfixed, and would also make the two signals disagree where today they are at least
consistently stale.
As far as I can tell this is the memory counterpart of #14141:
runsc updatealready accepts--memoryand the host cgroup does change, but nothing tells the running sentry, sousage.MaximumTotalMemoryBytesis still whatever was computed at sandbox boot. Note #14141'sfix has the same shape on the CPU side — it updates the reported CPU count and leaves the
guest's
cpu.cfs_quota_usstale — so I would value a maintainer's steer on whether thecgroup view is meant to be in scope for either.
Environment:
Steps to reproduce. Boot a gVisor pod with a 128Mi memory limit, then resize it to 2Gi in
place (no restart):
Booted at 128Mi:
Resize in place:
Kubernetes applied it, and the container was not restarted:
The host cgroup was updated correctly:
But the guest still reports the boot-time value, and so does the guest's own cgroup view:
Expected
MemTotalto follow the new limit (~2 GiB), the waynprocis expected to follow aCPU resize in #14141.
The memory is genuinely usable — only the reported total is stale. Writing 1.5 GiB into tmpfs
inside the same sandbox succeeds while it still reports 128 MiB total:
So this is a guest-visibility problem rather than an enforcement one, which makes it the
same shape as #14141: the limit is applied, the guest just cannot see it.
Pointer to the relevant code, in case it helps:
runsc/boot/loader.gosets both bounds onceat boot,
and
usage.TotalMemory()clamps toMaximumTotalMemoryBytes, which is what/proc/meminforenders. Nothing updates that ceiling after boot, so it looks analogous toApplicationCoresbefore #14277.Happy to work on a patch along the same lines as #14277 (an updater wired through the
existing
runsc update --memorypath) if that direction seems right. Two things I would wantguidance on:
already allocated would make
MemFreemeaningless —TotalMemory()already has a specialcase for
memSize < used.MinimumTotalMemoryBytes, since boot sets both bounds to the same value.