Test the remaining JIT backends under QEMU - #939
Conversation
| - { name: mips, triple: mips-linux-gnu, qemu: mips, libc: mips, container: "ubuntu:24.04" } | ||
| - { name: mipsel, triple: mipsel-linux-gnu, qemu: mipsel, libc: mipsel, container: "ubuntu:24.04" } | ||
| - { name: mips64, triple: mips64-linux-gnuabi64, qemu: mips64, libc: mips64, container: "ubuntu:24.04" } | ||
| - { name: mips64el, triple: mips64el-linux-gnuabi64, qemu: mips64el, libc: mips64el, container: "ubuntu:24.04" } |
There was a problem hiding this comment.
Does this mean ubuntu dropped mips support?
There was a problem hiding this comment.
Ubuntu never had MIPS support, as far as I know. This PR is using Ubuntu because it's convenient and provides access to Debian's cross toolchain packages (which is what these are).
What I've gathered is that there is currently no maintainer for the mips cross toolchain packages in Debian, so they haven't been updated in whatever version of Debian feeds into Ubuntu 26.04.
NWilson
left a comment
NWilson
left a comment
There was a problem hiding this comment.
Cool! I'll trying running. I've added a few comments and queries; let me know what you think.
| - { name: powerpc, triple: powerpc-linux-gnu, qemu: ppc, libc: powerpc, container: "ubuntu:26.04" } | ||
| - { name: powerpc64, triple: powerpc64-linux-gnu, qemu: ppc64, libc: ppc64, container: "ubuntu:26.04" } | ||
| # The MIPS cross compilers are no longer built for Ubuntu 26.04, | ||
| # hence the older container. |
There was a problem hiding this comment.
If MIPS has been dropped by Ubuntu, could we just leave it out ourselves? The usage with PCRE2 must be really, really low.
Ubuntu 24 has two years of support left. If we're going to remove MIPS at that point, I wonder whether it's worth adding it at all.
There was a problem hiding this comment.
The story is a bit more complex than "Ubuntu dropped MIPS" since Ubuntu itself never supported MIPS. See my other comment here: #939 (comment)
I'd suggest landing the MIPS CI jobs even if you're planning to remove the code (and it seems you're both antsy to do so). I'd like to at least fix the N32 ABI support, which is currently broken (O32 and N64 ABIs work fine) before its sent off into the sunset.
We have hand-written code for every architecture SLJIT supports, but CI only exercises a few of them. The existing "ptarmigan" job covers s390x, ppc64le, armv7, aarch64 and riscv64, which leaves the 32-bit ARM, PowerPC, MIPS and LoongArch backends untested. The ARM one is easy to miss: Ubuntu's armhf compiler defaults to Thumb-2, so armv7 exercises sljitNativeARM_T2_32.c and never sljitNativeARM_32.c. Add a job which cross-compiles for each of them and runs the test suite under qemu-user. This is a different mechanism to ptarmigan, which boots a container of the target architecture: run-on-arch-action supports only armv6, armv7, aarch64, riscv64, s390x and ppc64le, so none of these targets can be added to it. Cross-compiling is also faster, since only the test programs are emulated rather than the compiler as well. RunTest already accepts a "-sim" prefix for exactly this purpose, so the test scripts need no changes. Shared libraries are disabled so that the test programs are real executables rather than libtool wrapper scripts, which qemu-user cannot run. Both MIPS release 6 and the earlier releases are covered, since SLJIT_MIPS_REV selects different code for each.
| name: Zig | ||
| coelacanth: | ||
| # Tests with: cross-compilation and qemu-user, for the JIT backends that no | ||
| # other job covers. A coelacanth is a fish that was thought to be extinct. |
There was a problem hiding this comment.
Also - I absolutely love your choice of name for the build stage.
3 participants
PCRE2 has hand-written code for every architecture SLJIT supports, but CI only exercises some of it. The existing
ptarmiganjob covers s390x, ppc64le, armv7, aarch64 and riscv64. This adds a job covering the backends that nothing currently touches:-marm)sljitNativeARM_32.c-mthumb)sljitNativeARM_T2_32.csljitNativePPC_32.csljitNativePPC_64.csljitNativeMIPS_32.csljitNativeMIPS_64.cSLJIT_MIPS_REV >= 6pathssljitNativeLOONGARCH_64.cTwo of these are easy to overlook. Ubuntu's armhf compiler defaults to Thumb-2, so
armv7inptarmigannever exercises the ARM-mode backend. And ppc64le is ELFv2 while ppc64 big-endian is ELFv1, with function descriptors and a different call sequence, so the two are not interchangeable.Why a new job rather than more rows in ptarmigan
ptarmiganusesrun-on-arch-action, which boots a container of the target architecture. That action supports only armv6, armv7, aarch64, riscv64, s390x and ppc64le, so none of the targets above can be added to it. (It also emulates with QEMU, so this is not a change in kind, just in mechanism.)Cross-compiling and emulating only the test programs is also faster than emulating a whole toolchain.
RunTestalready accepts a-simprefix for exactly this, so no test scripts change. Shared libraries are disabled so the test programs are real executables rather than libtool wrapper scripts, whichqemu-usercannot run.Cost
Eleven jobs. Measured locally on a fast desktop, each is about 2.5 minutes of compute: the cross-build runs at native speed,
RunTestcosts ~10x native under emulation (15-21s) andpcre2_jit_test~20-29x (82-113s). Allowing for a slower runner plus container pull, apt and autogen, expect 6-10 minutes per job. The timeout is set to 20 minutes.Gated to
pushandworkflow_dispatchonly, matchingptarmigan, so pull requests are unaffected.Testing
Cross-built and run under qemu-user locally for arm, loongarch64, powerpc, powerpc64 and mips64.
RunTestandpcre2_jit_testpass on all five, including both big-endian PowerPC targets.Not verified locally: the 32-bit MIPS targets, the little-endian MIPS variants and the release 6 rows, for want of toolchains, and the Thumb-2 row. Package availability for all rows was checked against the Ubuntu archive indices rather than by building, so the first CI run may need fixups.
Notes
Two implementation details that I should note:
libc6-dev-<arch>-crossis installed explicitly. That needs the Debian architecture name, which is not always derivable from the triple (mipsisa32r6el-linux-gnuneedslibc6-dev-mipsr6el-cross).qemu-userbinary, soQEMU_CPUnames the emulated CPU instead. The defaults predate release 6.The MIPS cross compilers are no longer built for Ubuntu 26.04, so those rows use a 24.04 container.