eval-under-loop: options for image allocation, direct I/O, mkfs and mount

GOTCHAS.md

@@ -19,14 +19,17 @@ rediscovered the hard way.
 
 ### Loop (`bin/eval-under-loop`)
 
-A sparse backing image is `dd`'d, `mkfs.<fs>`'d, and loop-mounted.
+A backing image is created under `$TMPDIR`, `mkfs.<fs>`'d, and
+loop-mounted.
 
 | Knob | Value | Why |
 | --- | --- | --- |
-| `mkfs` options | none -- distro defaults | Whatever a user gets from `mkfs.ext4 /dev/sdX`, deliberately. |
-| Image size | per target, `loop-size-mb` in `evals/matrix.yaml` | `git annex test` needs room for many small objects; the other three do not. |
+| `mkfs` options | none -- distro defaults | Whatever a user gets from `mkfs.ext4 /dev/sdX`, deliberately. `--mkfs-opts` adds some; CI passes none. |
+| Image size | per target, `loop-size-mb` in `evals/matrix.yaml` | `git annex test` needs room for many small objects; the other three do not. btrfs refuses anything under 109 MiB. |
+| Image allocation | `--alloc zero`: `dd if=/dev/zero`, every block written | Works on any host fs. `sparse` (`truncate`) and `fallocate` are instant, for large images; `fallocate` fails outright where the host lacks it (NFS < 4.2, ramfs) rather than silently writing every block -- though vfat implements it by zeroing in the kernel. |
+| Loop I/O | buffered (image cached in the host page cache) | `--direct-io` attaches with `losetup --direct-io=on`; on a host fs without O_DIRECT (ramfs; tmpfs before Linux 6.6) current kernels refuse to attach and the backend stops with an error; older ones attach buffered and the backend warns. |
 | Mount (vfat, msdos, exfat, ntfs) | `-o uid=<invoker>,gid=<invoker>` | These filesystems store no ownership. Without `uid=`, everything belongs to root and an unprivileged wrapped command cannot write. |
-| Mount (everything else) | plain `mount`, then `chown <invoker>` on the mountpoint | ext4/xfs/btrfs carry real ownership; setting it once on the root is enough. |
+| Mount (everything else) | `mount -o defaults`, then `chown <invoker>` on the mountpoint | ext4/xfs/btrfs carry real ownership; setting it once on the root is enough. |
 
 **The vfat consequence worth knowing.** `fmask`, `dmask` and `umask` are
 left at kernel defaults, so every file on the mount reads as mode `0755`

README.md

@@ -167,6 +167,11 @@ sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test'
 sudo bin/eval-under nfs --set-home --mount-opts actimeo=0 -- \
   bash -c 'cd "$HOME" && git annex test'
 
+# A large btrfs volume without writing it out first: sparse image,
+# page cache bypassed for the image file
+sudo bin/eval-under loop --fs btrfs --size 2000000 --alloc sparse --direct-io \
+  --mount-opts discard=async -- bash -c 'df -h "$TMPDIR"'
+
 # Skip teardown to poke around after a failure
 sudo bin/eval-under beegfs --set-home --keep -- some-failing-command
 
@@ -210,7 +215,7 @@ into `bin/eval-under`, bumped with each release tag.
 | `bin/eval-under`                         | Dispatcher: routes to `bin/eval-under-<backend>`                                   |
 | `bin/eval-under-beegfs`                  | BeeGFS backend (containerised cluster + kernel client mount)                       |
 | `bin/eval-under-nfs`                     | NFS backend (localhost loopback export)                                            |
-| `bin/eval-under-loop`                    | Loop-device backend (dd + losetup + mkfs.<fs> + mount)                             |
+| `bin/eval-under-loop`                    | Loop-device backend (image + losetup + mkfs.<fs> + mount)                          |
 | `fixtures/beegfs/docker-compose-v7.yml`  | BeeGFS v7 test cluster (mgmtd + meta + storage), `network_mode: host`              |
 | `fixtures/beegfs/docker-compose-v8.yml`  | Same, for BeeGFS v8.x (different mgmtd command style / gRPC control plane)         |
 | `fixtures/beegfs/beegfs-*.conf.template` | Minimal client + helperd confs for the throwaway cluster                           |

bin/eval-under-loop

@@ -5,8 +5,8 @@
 # Generated with Claude Code 2.1.233 / Claude Opus 4.7
 #
 # eval-under-loop: run a command with TMPDIR / DATALAD_TESTS_TEMP_DIR (and
-# optionally HOME) pointing at a loop-mounted local filesystem (dd a backing
-# image, losetup, mkfs.<fs>, mount).
+# optionally HOME) pointing at a loop-mounted local filesystem (create a
+# backing image, losetup, mkfs.<fs>, mount).
 #
 # Backend of the eval-under framework. Absorbed from datalad's
 # tools/eval_under_testloopfs, with framework env-var names, a teardown
@@ -30,13 +30,19 @@ SET_HOME="${EVAL_UNDER_HOME_ON_MOUNT:-0}"
 FS="${EVAL_UNDER_LOOP_FS:-${DATALAD_TESTS_TEMP_FS:-vfat}}"
 SIZE_MB="${EVAL_UNDER_LOOP_SIZE_MB:-${DATALAD_TESTS_TEMP_FSSIZE:-100}}"
 
+ALLOC="${EVAL_UNDER_LOOP_ALLOC:-zero}"
+DIRECT_IO="${EVAL_UNDER_LOOP_DIRECT_IO:-0}"
+MKFS_OPTS="${EVAL_UNDER_LOOP_MKFS_OPTS:-}"
+MOUNT_OPTS="${EVAL_UNDER_LOOP_MOUNT_OPTS:-}"
+
 usage() {
   cat <<'EOF'
 Usage: eval-under-loop [OPTIONS] -- CMD [ARGS...]
 
 Run CMD with TMPDIR / DATALAD_TESTS_TEMP_DIR (and optionally HOME)
 pointing at a loop-mounted local filesystem. A fresh backing image is
-dd'd, mkfs.<fs>'d, and loop-mounted for the duration of the command.
+created (see --alloc), mkfs.<fs>'d, and loop-mounted for the duration
+of the command.
 
 Requires `mkfs.<fs>` for the chosen filesystem. For the default
 (`vfat`) that means `dosfstools`. Also common: `xfsprogs` (xfs),
@@ -69,6 +75,45 @@ Options (flag / env var / default / purpose):
   --size MB            EVAL_UNDER_LOOP_SIZE_MB        100
       Backing image size in megabytes. Bumped from datalad's original
       10 -- git-annex's testsuite fills that in a few seconds.
+      btrfs refuses anything under 109 MiB.
+
+  --alloc MODE         EVAL_UNDER_LOOP_ALLOC          zero
+      How the backing image gets its size:
+
+        zero       dd from /dev/zero: every block written and allocated
+                   on the host up front. Slow and host-space-hungry at
+                   large sizes (a TB image is a TB of writes), but
+                   works on any host filesystem.
+        sparse     truncate: instant, allocates nothing; the host fills
+                   in blocks as the filesystem inside writes them. If
+                   the host runs out of space, writes inside the image
+                   fail with I/O errors rather than ENOSPC.
+        fallocate  fallocate(2): reserves every block without writing
+                   it -- instant on most hosts (vfat zeroes them in the
+                   kernel, so not there). Fails, with no
+                   write-everything fallback of its own, where the host
+                   does not support it (NFS older than 4.2, ramfs, ext4
+                   without extents).
+
+  --direct-io          EVAL_UNDER_LOOP_DIRECT_IO      (unset)
+      Attach with `losetup --direct-io=on`, so image I/O bypasses the
+      host page cache instead of being cached twice (once for the
+      filesystem inside, once for the image file). The env var turns
+      it on when set to 1. The host filesystem must support O_DIRECT
+      (ramfs does not; tmpfs does since Linux 6.6): current kernels
+      then refuse to attach, and this backend stops with an error.
+      Older kernels may attach buffered instead; that is warned about.
+
+  --mkfs-opts OPTS     EVAL_UNDER_LOOP_MKFS_OPTS      (unset)
+      Extra arguments for mkfs.<fs>, split on whitespace (newlines
+      included) with no quoting -- so no argument can contain a space.
+      E.g. "-m single" or "-K" (skip the initial discard) for btrfs.
+
+  --mount-opts OPTS    EVAL_UNDER_LOOP_MOUNT_OPTS     (unset)
+      Comma-separated options for the loop mount's -o (appended to
+      uid=/gid= for vfat and friends), e.g.
+      "discard=async,compress=zstd" for btrfs. With "ro" the mountpoint
+      cannot be chown'd to the invoking user; that is warned about.
 
   -h, --help
       Print this help.
@@ -97,6 +142,10 @@ while [ $# -gt 0 ]; do
     --set-home) SET_HOME=1; shift ;;
     --fs) FS="$2"; shift 2 ;;
     --size) SIZE_MB="$2"; shift 2 ;;
+    --alloc) ALLOC="$2"; shift 2 ;;
+    --direct-io) DIRECT_IO=1; shift ;;
+    --mkfs-opts) MKFS_OPTS="$2"; shift 2 ;;
+    --mount-opts) MOUNT_OPTS="$2"; shift 2 ;;
     -h|--help) usage; exit 0 ;;
     --) shift; break ;;
     *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;;
@@ -105,6 +154,21 @@ done
 
 [ $# -gt 0 ] || { echo "no command given" >&2; usage >&2; exit 2; }
 
+case "$SIZE_MB" in
+  ''|*[!0-9]*|0) echo "--size must be a positive integer (MB), got: $SIZE_MB" >&2
+                 exit 2 ;;
+esac
+
+case "$ALLOC" in
+  zero|sparse|fallocate) ;;
+  *) echo "unknown --alloc mode: $ALLOC (expected zero|sparse|fallocate)" >&2
+     exit 2 ;;
+esac
+
+# Whitespace-split once here, so an option string cannot glob. read
+# stops at the first newline, so fold those into spaces first.
+read -ra MKFS_ARGS <<<"${MKFS_OPTS//$'\n'/ }"
+
 if [ "$(id -u)" -ne 0 ]; then
   command -v sudo >/dev/null || {
     echo "must run as root (or have sudo available)" >&2; exit 2; }
@@ -149,21 +213,45 @@ LOOP=""
 MNT_CREATED=0
 
 start_loop() {
-  log "creating ${SIZE_MB}MB backing image $IMAGE"
-  # Use bs=1M count=SIZE_MB (portable, unambiguous). datalad's original
-  # used bs=1032192 which is a slightly-off historical vfat-cluster
-  # rounding; bs=1M is simpler and works for every fs we care about.
-  dd if=/dev/zero "of=$IMAGE" bs=1M "count=$SIZE_MB" status=none
-
-  log "losetup --find --show $IMAGE"
-  LOOP=$("${SUDO[@]}" losetup --find --show "$IMAGE")
+  local losetup_opts=(--find --show)
+  log "creating ${SIZE_MB}MB backing image $IMAGE (--alloc $ALLOC)"
+  case "$ALLOC" in
+    # Use bs=1M count=SIZE_MB (portable, unambiguous). datalad's original
+    # used bs=1032192 which is a slightly-off historical vfat-cluster
+    # rounding; bs=1M is simpler and works for every fs we care about.
+    zero)      dd if=/dev/zero "of=$IMAGE" bs=1M "count=$SIZE_MB" status=none ;;
+    sparse)    truncate -s "${SIZE_MB}M" "$IMAGE" ;;
+    # util-linux fallocate calls fallocate(2) directly; unlike glibc's
+    # posix_fallocate() it never falls back to writing every block.
+    fallocate) fallocate -l "${SIZE_MB}M" "$IMAGE" ;;
+  esac
+  echo "I: image on host: $(du -h --apparent-size "$IMAGE" | cut -f1) apparent," \
+       "$(du -h "$IMAGE" | cut -f1) allocated"
+
+  if [ "$DIRECT_IO" = 1 ]; then losetup_opts+=(--direct-io=on); fi
+  log "losetup ${losetup_opts[*]} $IMAGE"
+  if ! LOOP=$("${SUDO[@]}" losetup "${losetup_opts[@]}" "$IMAGE"); then
+    if [ "$DIRECT_IO" = 1 ]; then
+      echo "E: losetup --direct-io=on failed; the host filesystem under" \
+           "$IMAGE likely lacks O_DIRECT. Retry without --direct-io." >&2
+    fi
+    exit 1
+  fi
   echo "I: loop device: $LOOP"
+  # Older kernels attach buffered rather than refuse when the host fs
+  # cannot do O_DIRECT; say so rather than let a run be read as direct.
+  if [ "$DIRECT_IO" = 1 ] \
+     && [ "$("${SUDO[@]}" losetup --list --noheadings --output DIO "$LOOP" \
+            | tr -d ' ')" != 1 ]; then
+    echo "W: --direct-io requested, but $LOOP is using buffered I/O" >&2
+  fi
 
-  log "mkfs.$FS $LOOP"
-  "${SUDO[@]}" "mkfs.$FS" "$LOOP" >/dev/null
+  log "mkfs.$FS${MKFS_ARGS[*]:+ ${MKFS_ARGS[*]}} $LOOP"
+  "${SUDO[@]}" "mkfs.$FS" "${MKFS_ARGS[@]}" "$LOOP" >/dev/null
 }
 
 mount_loop() {
+  local opts
   log "mounting $LOOP ($FS) at $MNT"
   [ -d "$MNT" ] || MNT_CREATED=1
   "${SUDO[@]}" mkdir -p "$MNT"
@@ -172,11 +260,14 @@ mount_loop() {
   # option to grant the invoking user write access.
   case "$FS" in
     vfat|msdos|exfat|ntfs)
-      "${SUDO[@]}" mount -o "uid=$INVOKER_UID,gid=$INVOKER_GID" "$LOOP" "$MNT"
+      opts="uid=$INVOKER_UID,gid=$INVOKER_GID${MOUNT_OPTS:+,$MOUNT_OPTS}"
+      "${SUDO[@]}" mount -o "$opts" "$LOOP" "$MNT"
       ;;
     *)
-      "${SUDO[@]}" mount "$LOOP" "$MNT"
-      "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$MNT"
+      opts="${MOUNT_OPTS:-defaults}"
+      "${SUDO[@]}" mount -o "$opts" "$LOOP" "$MNT"
+      "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$MNT" \
+        || echo "W: could not chown $MNT (mounted with -o $opts)" >&2
       ;;
   esac
   "${SUDO[@]}" mount | grep -F "$MNT" | sed 's/^/I: /'

tests/eval-under.bats

@@ -6,7 +6,8 @@
 # discovery, dispatch, and version reporting.
 #
 # Everything here is unprivileged and mounts nothing -- it covers the
-# entry point, not the backends' actual mount/teardown logic (that needs
+# entry point and the backends' option parsing, not their mount/teardown
+# logic (that needs
 # root, a kernel module, and a live cluster; see the CI matrix). The
 # dispatcher is exercised against throwaway trees of stub backends, so
 # the suite does not change shape every time a real backend is added.
@@ -349,3 +350,43 @@ FAKE
     done
   done
 }
+
+# ------------------------------------------------------- loop backend
+
+@test "loop: image/mkfs/mount options are documented and parsed" {
+  run "$REPO_ROOT/bin/eval-under-loop" --help
+  [ "$status" -eq 0 ]
+  for opt in --alloc --direct-io --mkfs-opts --mount-opts \
+             EVAL_UNDER_LOOP_ALLOC EVAL_UNDER_LOOP_DIRECT_IO \
+             EVAL_UNDER_LOOP_MKFS_OPTS EVAL_UNDER_LOOP_MOUNT_OPTS; do
+    [[ "$output" == *"$opt"* ]] || { echo "--help lacks $opt" >&2; return 1; }
+  done
+  # Each value-taking option must consume exactly its value, or --help
+  # would be swallowed / rejected as an unknown arg.
+  run "$REPO_ROOT/bin/eval-under-loop" --alloc sparse --direct-io \
+    --mkfs-opts "-m 0" --mount-opts noatime --help
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"Usage: eval-under-loop"* ]]
+}
+
+@test "loop: --mkfs-opts without a value is an error, not a silent skip" {
+  run "$REPO_ROOT/bin/eval-under-loop" --mkfs-opts
+  [ "$status" -ne 0 ]
+}
+
+@test "loop: an unknown --alloc mode is rejected with exit 2" {
+  run bash -c "'$REPO_ROOT/bin/eval-under-loop' --alloc bogus -- true 2>&1"
+  [ "$status" -eq 2 ]
+  [[ "$output" == *"unknown --alloc mode: bogus"* ]]
+}
+
+@test "loop: a non-integer or zero --size is rejected with exit 2" {
+  local size
+  for size in 1.5 -5 0 abc ""; do
+    run bash -c "'$REPO_ROOT/bin/eval-under-loop' --size '$size' -- true 2>&1"
+    if [ "$status" -ne 2 ] || [[ "$output" != *"--size must be a positive integer"* ]]; then
+      echo "--size '$size': expected exit 2, got $status: $output" >&2
+      return 1
+    fi
+  done
+}