Skip to content

In-place memory resize doesn't update the guest memory total (MemTotal stays at the boot value) #14409

Description

@mayur-tolexo

I hit this using gVisor pods with Kubernetes in-place pod resize, the same way as #14141 but
for memory. When I raise a pod's memory limit at runtime, the host cgroup is updated
correctly, but the guest keeps reporting the memory it had at boot, so anything that sizes
itself from what the guest reports stays capped at the old value and never uses the memory I
just added, unless I recreate the pod.

Two separate guest-visible signals are stale here, and they matter to different consumers:

  • /proc/meminfo MemTotal (and sysinfo(2)), read by free-style tooling and by anything
    sizing itself from total system memory.
  • The sandbox's own cgroup limit, /sys/fs/cgroup/.../memory.limit_in_bytes, which is seeded
    once when the sandboxed cgroupfs is mounted. Container-aware runtimes prefer this over
    MemTotal — the JVM's UseContainerSupport and .NET's GC heap limit read the cgroup, and
    I believe Node's container heuristics do too, though I am less sure of that one.

I originally wrote this issue as though MemTotal were the signal those runtimes read. That
was wrong, and it matters for scoping: updating MemTotal alone would leave the JVM case
unfixed, and would also make the two signals disagree where today they are at least
consistently stale.

As far as I can tell this is the memory counterpart of #14141: runsc update already accepts
--memory and the host cgroup does change, but nothing tells the running sentry, so
usage.MaximumTotalMemoryBytes is still whatever was computed at sandbox boot. Note #14141's
fix has the same shape on the CPU side — it updates the reported CPU count and leaves the
guest's cpu.cfs_quota_us stale — so I would value a maintainer's steer on whether the
cgroup view is meant to be in scope for either.

Environment:

runsc version release-20260817.0 (spec 1.2.1), systrap platform
kind v0.31.0, Kubernetes v1.35.0 (in-place resize is GA), containerd 2.2.0
node image kindest/node:v1.35.0
node: Linux 6.12.76-linuxkit aarch64

Steps to reproduce. Boot a gVisor pod with a 128Mi memory limit, then resize it to 2Gi in
place (no restart):

apiVersion: v1
kind: Pod
metadata:
  name: gv-mem
spec:
  runtimeClassName: gvisor
  containers:
  - name: c
    image: busybox
    command: ["sleep","3600"]
    resizePolicy:
    - resourceName: memory
      restartPolicy: NotRequired
    resources:
      requests: {cpu: "500m", memory: "128Mi"}
      limits:   {cpu: "500m", memory: "128Mi"}

Booted at 128Mi:

$ kubectl exec gv-mem -- grep MemTotal /proc/meminfo
MemTotal:         131072 kB
$ kubectl exec gv-mem -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes
134217728

Resize in place:

$ kubectl patch pod gv-mem --subresource resize --patch \
    '{"spec":{"containers":[{"name":"c","resources":{"requests":{"memory":"2Gi"},"limits":{"memory":"2Gi"}}}]}}'
pod/gv-mem patched

Kubernetes applied it, and the container was not restarted:

spec limit:      2Gi
status alloc:    2Gi
restartCount:    0

The host cgroup was updated correctly:

$ cat /sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pode71e1fcd_....slice/memory.max
2147483648

But the guest still reports the boot-time value, and so does the guest's own cgroup view:

$ kubectl exec gv-mem -- grep MemTotal /proc/meminfo
MemTotal:         131072 kB
$ kubectl exec gv-mem -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes
134217728

Expected MemTotal to follow the new limit (~2 GiB), the way nproc is expected to follow a
CPU resize in #14141.

The memory is genuinely usable — only the reported total is stale. Writing 1.5 GiB into tmpfs
inside the same sandbox succeeds while it still reports 128 MiB total:

$ kubectl exec gv-mem -- sh -c 'mount -o remount,size=2G /dev/shm; dd if=/dev/zero of=/dev/shm/f bs=1M count=1500'
1572864000 bytes (1.5GB) copied, 2.305989 seconds, 650.5MB/s

$ kubectl exec gv-mem -- df -h /dev/shm
none                      3.9G      1.5G      2.4G  38% /dev/shm

$ kubectl exec gv-mem -- grep MemTotal /proc/meminfo
MemTotal:         131072 kB

So this is a guest-visibility problem rather than an enforcement one, which makes it the
same shape as #14141: the limit is applied, the guest just cannot see it.

Pointer to the relevant code, in case it helps: runsc/boot/loader.go sets both bounds once
at boot,

usage.MinimumTotalMemoryBytes = args.TotalMem
usage.MaximumTotalMemoryBytes = args.TotalMem

and usage.TotalMemory() clamps to MaximumTotalMemoryBytes, which is what
/proc/meminfo renders. Nothing updates that ceiling after boot, so it looks analogous to
ApplicationCores before #14277.

Happy to work on a patch along the same lines as #14277 (an updater wired through the
existing runsc update --memory path) if that direction seems right. Two things I would want
guidance on:

  • Whether this should be grow-only, as runsc: update sentry CPU count on live cgroup CPU changes #14277 is. Lowering the reported total below what is
    already allocated would make MemFree meaningless — TotalMemory() already has a special
    case for memSize < used.
  • What to do about MinimumTotalMemoryBytes, since boot sets both bounds to the same value.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions