runsc: no way to keep /tmp on the rootfs when the image's /tmp is empty (EXDEV vs Docker)

[xyuzh]

Description

When the OCI spec has no mount at /tmp and the rootfs /tmp is empty, runsc mounts an internal tmpfs over it. That puts /tmp on a different device from the rootfs, so the common tempfile-then-rename(2) (or link(2)) pattern fails with EXDEV — where the same workload under runc/Docker succeeds, because Docker leaves /tmp as plain rootfs.

There appears to be no way to opt out: the tmpfs is skipped only when the rootfs /tmp is non-empty, and a spec-level bind mount of the image's own tmp directory doesn't help (it becomes a separate gofer mount — still a different device than an overlay-backed rootfs).

Steps to reproduce

Any rootfs with an empty /tmp (repro verified with an extracted ubuntu:24.04):

{
  "ociVersion": "1.0.0",
  "process": {
    "user": {"uid": 0, "gid": 0},
    "args": ["/bin/sh", "-c", "grep \" /tmp \" /proc/mounts || echo no-tmp-mount; echo x > /tmp/f; ln /tmp/f /f2 && echo LINK-OK || echo LINK-FAILED"],
    "cwd": "/",
    "env": ["PATH=/usr/bin:/bin"]
  },
  "root": {"path": "<rootfs>", "readonly": false},
  "mounts": [{"destination": "/proc", "type": "proc", "source": "proc"}],
  "linux": {"namespaces": [{"type": "pid"}, {"type": "mount"}]}
}

# rootfs /tmp empty:
$ runsc --rootless --ignore-cgroups --network=none --overlay2=root:dir=/tmp/ov run --bundle . demo
none /tmp tmpfs rw,mode=01777 0 0
ln: failed to create hard link '/f2' => '/tmp/f': Invalid cross-device link
LINK-FAILED

# identical spec, after: touch <rootfs>/tmp/.keep
$ runsc --rootless --ignore-cgroups --network=none --overlay2=root:dir=/tmp/ov2 run --bundle . demo2
no-tmp-mount
LINK-OK

Impact

Found running Terminal-Bench under Ray Sandbox (ray-project/ray#65744): tasks using Python's tempfile + os.rename into the working tree fail with OSError: [Errno 18] Invalid cross-device link on gVisor while passing under Docker. Our current workaround is seeding extracted image caches with a placeholder file (/tmp/.ray-sandbox-keep) so the emptiness check keeps /tmp on the rootfs — which pollutes shared image caches and leaks an artifact into sandboxes unless cleaned up post-boot.

Feature request

A way to keep /tmp on the rootfs regardless of whether it's empty — e.g. a flag like --tmp-mount=auto|tmpfs|rootfs (defaulting to today's auto), or honoring an annotation in the spec. That would give integrators Docker-parity /tmp semantics without marker-file workarounds.

runsc version

spec: 1.2.1

[EtiennePerot]

This will make performance much slower in practice for the vast majority of workloads that don't care about this distinction. Not convinced --tmp-mount=rootfs is what you'd actually want to use, even if such a flag did exist :)

[pcmoritz]

Hey @EtiennePerot thanks a lot for your answer! There are certain workloads where docker compatibility is extremely important, so even if there is a performance slowdown it would still be worth it. Given that docker puts /tmp on the rootfs (and mounts a tmpfs on top of that), do they take the same performance hit or is there anything about gvisor that makes the performance worse? If that's the case, is there any way to improve the gvisor performance for this scenario?

Right now the workaround we have to do is to create an empty file in /tmp so it gets mounted, and then delete it before the actual workload is run.

[andrewsykim]

Opened #14553 which adds a flag to opt into /tmp being in rootfs