Integrate scoped Linux Test Project syscall conformance #171

Goal

Integrate a scoped Linux Test Project (LTP) syscall conformance suite into elfuse.

Use a pinned shallow LTP checkout, apply local patches if needed, and run a curated syscall subset through the existing make check flow.

Rules:

• Do not use git submodules.
• Do not copy LTP files into elfuse.
• Retrieve LTP with git clone --depth=1.
• Keep LTP under build/ltp-src.
• Treat LTP as a vendored test dependency and syscall semantics oracle.
• Keep elfuse-specific changes as local patches.
• Keep the default make check slice fast and deterministic.

Sub-tasks

1. Fetch pinned LTP with shallow clone

Add a reproducible fetch step for LTP.

Implementation:

• Add tests/fetch-ltp.sh.
• Fetch LTP into:

  build/ltp-src
  

• Use shallow clone:

  git clone --depth=1 --branch <tag-or-branch> \
      https://github.com/linux-test-project/ltp \
      build/ltp-src

• Resolve and record the exact commit:

  git -C build/ltp-src rev-parse HEAD > build/ltp-src/.ltp-ref

• If build/ltp-src/.ltp-ref already matches the requested SHA, reuse the checkout.
• If it does not match, delete build/ltp-src and fetch again.
• Expose override variables:

  LTP_REF=<tag-or-branch-or-sha>
  LTP_URL=https://github.com/linux-test-project/ltp
  LTP_DIR=build/ltp-src

Considerations:

• No git submodule.
• No generated LTP build output should be committed.
• A shallow clone by SHA may need a fetch fallback because git clone --depth=1 --branch <sha> is not portable for raw SHAs. Accept tag/branch by default, then verify resolved SHA.
• Network access should happen only in explicit fetch/build targets, not every make check.
• The script should be idempotent.

Validation:

rm -rf build/ltp-src
LTP_REF=<tag-or-branch> tests/fetch-ltp.sh
test -s build/ltp-src/.ltp-ref
git -C build/ltp-src rev-parse --is-shallow-repository

Expected:

• .ltp-ref exists.
• checkout is shallow.
• rerunning the script reuses the checkout when the ref matches.

2. Apply elfuse-specific LTP patches

Support a small patch stack on top of the pinned LTP checkout.

Implementation:

• Add patch directory:

  tests/fixtures/ltp/patches/
  

• Apply patches after clone:

  for p in tests/fixtures/ltp/patches/*.patch; do
      git -C build/ltp-src apply "$p"
  done

• Make patch application deterministic:
  • apply in lexical order,
  • fail on reject,
  • do not continue with a partially patched tree.
• Add a marker file after successful patching, for example:

  build/ltp-src/.elfuse-patches-applied
  

Considerations:

• Patches may adapt build flags, skip unavailable fixtures, or make paths deterministic.
• Patches must not weaken expected Linux syscall semantics just to make elfuse pass.
• Prefer upstreamable patches where possible.
• If a case needs unsupported kernel features, classify it in the manifest rather than changing the test expectation.
• Reapplying patches should not silently double-apply; fetch script should start from a clean checkout when the patch set changes.

Validation:

rm -rf build/ltp-src
tests/fetch-ltp.sh
test -f build/ltp-src/.elfuse-patches-applied
git -C build/ltp-src status --short

Expected:

• patches apply cleanly,
• patched tree state is reproducible.

3. Define the syscall conformance scope

Keep the initial scope tight: only syscall behavior elfuse actively emulates and can run without privileged host setup.

Implementation:

• Add tests/ltp-syscalls.tsv.
• Populate it only with scoped syscall cases from LTP runtest/syscalls.
• Group rows by family:
  • fd
  • path
  • process
  • signal
  • identity
  • memory
  • resource
  • poll
  • timer
  • event
  • socket
  • ipc

Initial in-scope families:

Path and FD:

• openat
• close
• close_range
• dup
• dup3
• fcntl
• lseek
• read
• write
• pread64
• pwrite64
• readv
• writev
• statx
• newfstatat
• faccessat2
• renameat2
• unlinkat
• linkat
• symlinkat
• readlinkat
• mkdirat
• getdents64

Process, signal, identity:

• clone
• clone3
• fork
• execve
• execveat
• wait4
• waitid
• exit_group
• kill
• tkill
• tgkill
• rt_sigaction
• rt_sigprocmask
• rt_sigtimedwait
• rt_sigsuspend
• sigaltstack
• getpid
• gettid
• getppid
• getuid
• geteuid
• getgid
• getegid
• getresuid
• getresgid
• set_tid_address

Memory and resource:

• brk
• mmap
• mprotect
• munmap
• mremap
• msync
• mincore
• madvise
• getrlimit
• prlimit64
• getrusage
• sysinfo

Poll, timer, event:

• poll
• ppoll
• pselect6
• epoll_create1
• epoll_ctl
• epoll_pwait
• eventfd2
• timerfd_create
• timerfd_settime
• timerfd_gettime
• signalfd4
• pipe2

Fixture-light socket and IPC:

• socketpair
• sendmsg
• recvmsg
• getsockopt
• setsockopt
• shutdown
• semget
• semop
• semtimedop
• semctl
• shmget
• shmat
• shmdt
• shmctl
• msgget
• msgsnd
• msgrcv
• msgctl

Out of scope for this issue:

• namespaces
• cgroups
• BPF
• perf events
• keyrings
• module loading
• fanotify
• io_uring
• Landlock
• raw packet sockets / AF_PACKET
• raw block-device tests
• hugepage-only tests
• root-only host mutation
• broad mount/umount coverage outside existing elfuse FUSE tests
• tests requiring a real Linux /dev, block device, network namespace, or kernel security module setup

Considerations:

• The manifest should classify cases, not silently drop them.
• non-goal is only for features already documented as elfuse Non-Goals.
• Implemented syscalls must not be hidden as non-goal.
• Fixture-heavy but relevant tests should be skip-fixture, not non-goal.
• Scope can expand later after the fast path is stable.

Validation:

make check-ltp-plan

Expected:

• all in-scope families have manifest entries,
• out-of-scope rows include a reason,
• unclassified in-scope rows fail the check.

4. Add the LTP syscall manifest

Define exactly what runs, what skips, and why.

Implementation:

• Add:

  tests/ltp-syscalls.tsv
  

• Columns:

  case	family	ltp_command	status	reason	fixture	make_check
  

• Status values:
  • check-enabled: run in make check and make test-ltp
  • enabled: run in make test-ltp-full
  • skip-fixture: relevant, but fixture unavailable by default
  • skip-privileged: relevant, but needs root or unsafe host mutation
  • known-fail: relevant, expected to fail until elfuse implementation changes
  • non-goal: outside elfuse scope and documented under Non-Goals
  • unclassified: discovered but not triaged yet

Considerations:

• make_check=true only for fast deterministic cases.
• known-fail rows must point at a TODO or issue and likely source area.
• Manifest rows should use LTP case names, not local aliases.
• The manifest should be reviewable by humans; avoid generating unreadable blobs.
• Updating the pinned LTP ref should surface renamed/deleted cases.

Validation:

make check-ltp-plan

Checks:

• every check-enabled / enabled case exists in runtest/syscalls,
• every enabled command has a built binary/script,
• every in-scope family has at least one classified row,
• no implemented syscall maps to non-goal,
• no stale LTP case names remain after updating the pin.

5. Build only the selected LTP syscall subset

Add build support for the manifest-selected cases.

Implementation:

• Add:

  make ltp-syscalls

• Depend on tests/fetch-ltp.sh.
• Configure LTP in build/ltp-src.
• Build only the cases referenced by tests/ltp-syscalls.tsv.
• Avoid installing full LTP into /opt/ltp.
• Stage runnable artifacts under:

  build/ltp/
  

• Export required LTP runtime variables when running:

  LTPROOT=$(pwd)/build/ltp
  PATH="$LTPROOT/testcases/bin:$PATH"

Considerations:

• Full LTP is too broad for the default path.
• Some LTP tests require helper binaries; the subset build must include those helpers.
• Some shell tests need LTPROOT and PATH.
• Build failures for skipped rows should not fail the selected subset.
• Keep cross-build flags compatible with elfuse guest binaries.

Validation:

make ltp-syscalls
test -d build/ltp/testcases/bin

6. Run conformance through the existing test driver

Consolidate execution under the current test system.

Implementation:

• Add:

  make test-ltp
  make test-ltp-full

• Add test-ltp to make check.
• Reuse:

  tests/lib/test-runner.sh
  

• Run:
  • check-enabled rows for make check and make test-ltp,
  • enabled plus check-enabled rows for make test-ltp-full.
• Emit:

  build/ltp-syscall-results.json
  

• Store per-case logs under:

  build/ltp/logs/
  

Considerations:

• make check must remain fast enough for normal CI/CD.
• make test-ltp is the quick conformance gate.
• make test-ltp-full is for broader compatibility verification.
• Use timeout protection for every LTP case.
• Use skip code 77 for unavailable optional fixtures.
• Preserve LTP's own pass/fail/skip status when possible.
• Do not create a second unrelated test reporting format; fit into existing output conventions.

Validation:

make check
make test-ltp
make test-ltp-full

7. Compare against Linux reference when available

Use QEMU AArch64 as the Linux syscall semantics reference.

Implementation:

• Support:

  make test-ltp LTP_REFERENCE=qemu
  make test-ltp-full LTP_REFERENCE=qemu

• Run the same selected LTP case under:
  • elfuse
  • QEMU AArch64 Linux reference
• Compare:
  • LTP result status
  • exit status
  • signal termination
  • timeout
  • child wait status where visible
  • relevant stdout/stderr summary
• Store both logs.

Considerations:

• QEMU absence should skip reference comparison, not fail local make check.
• Prefer comparing LTP result status before raw output.
• Raw output can differ in harmless paths, timings, PIDs, or hostnames.
• For errno-visible failures, logs must retain enough detail to diagnose mismatch.

Validation:

make test-ltp LTP_REFERENCE=qemu

8. Promote failures into actionable implementation work

Make conformance failures easy to turn into elfuse fixes.

Implementation:

• Add:

  scripts/ltp-results-to-todo.py
  

• Read:

  build/ltp-syscall-results.json
  

• Emit one TODO-ready block per unique failure.

Each block should include:

• LTP case
• syscall family
• LTP command
• elfuse result
• QEMU/Linux reference result when available
• log path
• likely elfuse owner file under src/syscall/*.c
• reproducer command

Considerations:

• Do not generate tasks for skip-* or non-goal.
• Group duplicate failures by syscall family and likely owner file.
• Keep generated tasks small and independently fixable.
• Prefer fixing shared syscall handlers over patching individual tests.

Validation:

python3 scripts/ltp-results-to-todo.py build/ltp-syscall-results.json

Completion Criteria

• LTP is retrieved with git clone --depth=1.
• No git submodule is used.
• The resolved LTP commit is recorded.
• Local LTP patches, if any, apply cleanly.
• tests/ltp-syscalls.tsv defines a scoped syscall set.
• make check-ltp-plan validates the manifest.
• make ltp-syscalls builds selected cases.
• make check runs make test-ltp.
• make test-ltp runs the quick CI/CD conformance slice.
• make test-ltp-full runs the broader compatibility suite.
• QEMU comparison works when configured.
• Results are written to build/ltp-syscall-results.json.
• Failures can be promoted into actionable implementation TODOs.