Claude/upgrade ape c library mm z gd

.gitignore

@@ -94,3 +94,7 @@ sys/src/ape/cmd/clisp/build-*/
 # munmaptrim-xcheck's generated extract and its host binary
 sys/lib/tests/munmaptrim-extract.h
 sys/lib/tests/munmaptrim-xcheck
+
+# makectx-xcheck's host binary and its scratch include directory
+sys/lib/tests/makectx-xcheck
+sys/lib/tests/.makectx-xcheck.d/

CLAUDE.md

@@ -271,6 +271,14 @@ reach its start routine and `pthread_cond_timedwait` must actually
 wait; the two are different bugs and the sections are split so), `poll-test` (the count is
 of pollfd ENTRIES, a negative fd is ignored, a closed one is POLLNVAL —
 the two rules whose absence makes a `while(poll())` caller SPIN),
+`ucontext-test` (**the one test here with a REAL REFERENCE
+IMPLEMENTATION** — glibc has ucontext, so all six sections were checked
+against a working one before the VM saw them, 0 failures; *a failure on
+Plan 9 that also fails on gcc is the test's bug*. Section 1 asks whether
+the function ran **on the stack it was given**, which a switch that
+quietly used the caller's would otherwise pass; section 3 passes the
+maximum arguments because the ucontext cell sits just above them and the
+failure mode of an off-by-one is `exit(0)`, so it prints a marker first),
 `rdev-test` (two different files must
 be distinguishable — it transcribes diffutils' `same_file()` verbatim,
 because the question is what THAT function answers), `stdio-test`.
@@ -282,8 +290,8 @@ so a test built from a fresh pull can run days-old library code and say
 nothing. `_sock_listenmark`, `_execmark`, `_ttymark`, `_getcwdmark`,
 `_printfmark`, `_dupmark`, `_dotdirmark`, `_tempmark`, `_fdinfomark`,
 `_umaskmark`, `_bufmark`, `_rdevmark`, `_selregmark`, `_pthreadmark`,
-`_syncmark`, `_mmapmark` each make the test **fail to LINK** against a libap
-predating its fix. The idiom has paid twelve times; bump one when a file changes in a way a test must see — and
+`_syncmark`, `_mmapmark`, `_ucmark` each make the test **fail to LINK** against a libap
+predating its fix. The idiom has paid thirteen times; bump one when a file changes in a way a test must see — and
 it matters most where the bug's whole symptom is that nothing is
 printed.
 
@@ -313,6 +321,18 @@ field was. The harness wraps `malloc` now, as it already wrapped
 `free`. *The assertion had been phrased against the thing under test
 instead of against something independent of it*, which is the shape
 to look for when a break does not fire.
+**`makectx-xcheck.{c,sh}`** checks `makecontext`'s FRAME on the host,
+and **it compiles the real `ucontext.c` rather than a copy** — no
+extraction and no markers to rot, which is possible only because
+everything structural in that file is C and the assembly knows just
+"eight slots". It exists because the one number spanning the two
+languages (where the ucontext pointer sits in the new frame) fails
+SILENTLY on the VM. No external oracle, so its control is a deliberate
+break: the cell off by one gives **3** failures, `SP` not one slot below
+**1**. ***Its first run reported its own STUB***: the real
+`_setmcontext` never returns, the fake recorded and returned, and
+section 3 duly failed — *the assertion was right and the fake was
+wrong*, which is the shape to recognise rather than argue with.
 **`hostkencc/clisp-mirror.sh` stages clisp's generated tree on the
 host** so the host kencc can compile and inspect it -- `--flags` prints
 the compile line, `--compile` builds all 31 TUs and GATES. *Its
@@ -485,8 +505,16 @@ and 3 say both trims survive. **But three of the eight `mmap` calls
 FAILED**, and 10MB is not a memory limit, so **section 0b** asks how
 many mappings a process may HOLD, at three sizes, *with the prediction
 written down first*: slots give the same count at every size, address
-space gives one falling as 1/size. It is **5 at 2MB and 5 at 16MB**,
-and 64 at a size below `SEG_THRESHOLD` — the control. **Two of its
+space gives one falling as 1/size. It was **5 at 2MB and 5 at 16MB**,
+and 64 at a size below `SEG_THRESHOLD` — the control. ***AND THE FIX
+DISARMED IT***: with `SEG_THRESHOLD` at 64MB all three of 0b's sizes
+take the malloc arm, so it answers 64/64/64 — right, worth keeping as
+the regression check that `mmap` spends no slot, and *unable to say
+how many slots exist*. **Section 0c** asks the platform instead,
+`_SEGATTACH` in a loop at 64KB: **5**, the same number over a 256-fold
+range of sizes for a total of 320KB, which excludes memory outright.
+*A section whose cases have collapsed onto one path is a check that
+can only pass.* **Two of its
 sections can kill the process, so it takes an ARGUMENT**: no argument
 runs the tail trim, `3` runs the head trim, never sharing a run,
 because a section that can kill the process hides every section after
@@ -1121,11 +1149,12 @@ two are the opposite of what it says — `HAVE_TM_ZONE` and
 `HAVE_ST_BLOCKS` are both YES, `struct tm` and `struct stat` having
 those members. *A sweep over function names, asked about a struct
 field, answers `absent` confidently.*
-**What can still stop it, in order**: (1) **fibers have no context
-switch** — `zend_fibers.c` is unconditional and wants either
-boost.context's GAS assembly or `<ucontext.h>`, and APExp has neither;
-a libap `ucontext` for amd64 is the reusable half, and it is a decision
-to make BEFORE the mkfile, since it changes what the mkfile builds.
+**What can still stop it, in order**: (1) fibers — **CLOSED on the
+host, unmeasured on the VM**: `zend_fibers.c` is unconditional and
+wants either boost.context's GAS assembly or `<ucontext.h>`, and APExp
+now has the second (`lib/ap/arch/amd64/mcontext.s` + `ucontext.c`,
+`docs/notes/libap.md`). amd64 only; the other ten follow the same
+shape when wanted.
 (2) **`ext/opcache` is mandatory in 8.5** — there is no
 `--disable-opcache`, only `--disable-opcache-jit`, which is what drops
 the 13-file `jit/ir` code generator; opcache only has to COMPILE, since
@@ -1690,12 +1719,25 @@ the index, so that nothing here is a surprise.
   reason it scrubs `FD_ISTTY` and `FD_ISREG`. `FD_BUFFEREDX` is poison:
   `read()` and `select()` both answer EIO for it without touching the
   fd.
-- **A process gets FIVE spare segment slots, measured**, so `mmap`
-  cannot be backed by `segattach` in general: `mmaptrim-probe` section
-  0b maps until refused at three sizes and answers 5 at 2MB and 5 at
-  16MB -- *the same count for eight times the memory*. `SEG_THRESHOLD`
-  is 64MB for that reason and anything below it is `malloc`-backed,
-  page-aligned by hand since `malloc` is not.
+- **kencc amd64 has NO callee-saved registers, measured.** A host
+  `6c -S` on a function that computes three values, calls through a
+  function pointer and then uses them spills all three to the stack and
+  touches BX and R12-R15 not at all. *So nothing may be kept in a
+  register across a call into arbitrary code* — which is why libap's
+  `ucontext` puts its bookkeeping pointer on the new stack. `main9.s:7`
+  calls R12-R15 "callee-saved REGEXT registers" and both can be true: a
+  register the allocator never hands out is trivially preserved.
+- **A process gets FIVE spare segment slots, measured twice and the
+  second time at 64KB**, so `mmap` cannot be backed by `segattach` in
+  general. `mmaptrim-probe` section 0b got 5 at 2MB and 5 at 16MB
+  through `mmap`; **section 0c calls `_SEGATTACH` directly and gets 5
+  at 64KB**. *Five, over a 256-fold range of sizes and a total of
+  320KB* -- so memory and address space are excluded outright, and the
+  number is the kernel's rather than the library's. `SEG_THRESHOLD` is
+  64MB for that reason and anything below it is `malloc`-backed,
+  page-aligned by hand since `malloc` is not. **0c is the one that
+  still asks**: every size 0b uses is now under the threshold, so it
+  answers 64/64/64 and can only confirm that `mmap` spends no slot.
 - `/dev/snarf` is the clipboard and has no concept of ownership.
 - Plan 9 has no loopback unless `ip/ipconfig loopback /dev/null 127.1`
   has been run -- and that belongs in the machine's startup, not in

amd64/include/ape/ucontext_arch.h

@@ -0,0 +1,32 @@
+#ifndef __UCONTEXT_ARCH_H
+#define __UCONTEXT_ARCH_H
+
+/*
+ * amd64 machine context for <ucontext.h>.
+ *
+ * ***THE EIGHT SLOTS ARE setjmp's EIGHT, AND THAT IS DELIBERATE.***
+ * `arch/amd64/setjmp.s' saves SP, PC, BP, BX, R12, R13, R14, R15 and its
+ * header says why the last six are there: APExp's 6c has
+ * REGEXT = D_R15, so R15 and R14 are reserved program-wide
+ * (`6c/reg.c:55-57') and R13/R12 can follow them as external register
+ * caches for global variables.  A context switch that restored only
+ * SP and PC would leave the resumed code reading another context's
+ * cached globals.  *ucontext has exactly setjmp's problem, so it gets
+ * exactly setjmp's answer* -- and `arch/amd64/ucontext.s' is that file's
+ * save/restore sequence with the return-value convention changed.
+ *
+ * The order is fixed because the assembly indexes it by number.  It is
+ * the ONLY thing the assembly knows about any of these structures:
+ * everything else -- `uc_link', `uc_stack', `uc_sigmask' -- is handled
+ * in C, so no struct offset is ever written down twice.
+ */
+typedef struct mcontext_t {
+	unsigned long long mc_regs[8];
+} mcontext_t;
+
+/* names for the three slots `makecontext' has to fill */
+#define _MC_SP	0
+#define _MC_PC	1
+#define _MC_ARG	2	/* BP, which is 6c's RARG: the first argument */
+
+#endif /* __UCONTEXT_ARCH_H */

docs/notes/libap.md

@@ -3900,6 +3900,23 @@ anything a program is at all likely to allocate. *The limit was known
 and the constant was chosen as though it were not.* At 1MB, php's Zend
 allocator -- 2MB chunks -- would have stopped at about 10MB of heap.
 
+***AND THE NUMBER IS CONFIRMED INDEPENDENTLY, AT 64KB.*** Section 0c
+calls `_SEGATTACH` in a loop rather than going through `mmap`, and
+gets **5** -- the same five, now over a 256-fold range of sizes and a
+total of 320KB, which leaves no reading in which memory or address
+space is the limit. It also says the ceiling did not move when
+`SEG_THRESHOLD` did, which is the point of asking the kernel rather
+than the library.
+
+*That section exists because the fix disarmed the one that found the
+problem.* With the threshold at 64MB all three of 0b's sizes take the
+malloc arm, so it answers 64/64/64 -- correct, and worth keeping as
+the regression check that `mmap` spends no slot, but **its smallest
+size has stopped being a control and the run says nothing about how
+many slots exist**. Read on its own it is indistinguishable from the
+Linux control. A section whose cases have collapsed onto one path is
+a check that can only pass.
+
 64MB now: a program needs five simultaneous 64MB mappings to reach the
 ceiling, and the detachable path is kept for the only case worth a
 slot, a mapping big enough that leaving it in the data segment for the
@@ -3928,3 +3945,122 @@ already wrapped `free`, and the same break gives 3 failures. *A check
 that cannot fail is not a check*, and the way it got written is worth
 naming: the assertion was phrased against the thing under test instead
 of against something independent of it.
+
+## ucontext for amd64
+
+php's `Zend/zend_fibers.c` is unconditional -- `Fiber` is a language
+class -- and wants either boost.context's GAS assembly or
+`<ucontext.h>`. APExp had neither. This is the second, chosen because
+it is libap's own code and reusable beyond php; the other ten
+architectures can follow the same shape.
+
+**Three files.** `arch/amd64/mcontext.s` holds `_getmcontext`,
+`_setmcontext`, `_swapmcontext` and the `_ctxreturn` stub;
+`arch/amd64/ucontext.c` holds the four POSIX entry points and
+`_ctxfinish`; `sys/include/ape/ucontext.h` and
+`amd64/include/ape/ucontext_arch.h` are the interface.
+
+**It is `setjmp.s` with the return-value convention changed**, and
+deliberately so: the two problems are the same one. Both of that file's
+findings carry over unchanged --
+
+* **eight registers, not two.** 6c has `REGEXT = D_R15`, so R15 and R14
+  are reserved program-wide (`6c/reg.c:55-57`) and R13/R12 can follow
+  them as caches for global variables. A switch restoring only SP and
+  PC leaves the resumed code reading another context's globals.
+* **the restore must not WRITE to the stack.** `ADDQ $8,SP` then `JMP`
+  instead of storing the PC at `0(SP)` and `RET`ing. A context resumed
+  at the top of a fresh fiber stack is exactly the case that faults.
+
+**And AX is zeroed before the jump.** Without it a `getcontext` resumed
+by `setcontext` returns whatever was in AX and its caller believes the
+number -- the same reason `longjmp` sets it, found by writing the
+section that checks it rather than by reasoning.
+
+### No register survives a call here, and that is measured
+
+The first design kept the ucontext pointer in a callee-saved register
+across the context's own function. **There are no callee-saved
+registers.** `hostkencc/out/bin/6c -S` on a function that computes three
+values, calls through a function pointer and then uses them spills all
+three to the stack and touches BX and R12-R15 not at all. So the
+pointer goes on the new stack instead.
+
+*`main9.s:7` calls R12-R15 "callee-saved REGEXT registers", and both
+statements can be true* -- a register the allocator never hands out is
+trivially preserved -- but nothing in this implementation now depends on
+either reading, which is the point.
+
+### The one number that spans two languages
+
+`makecontext` writes the ucontext pointer at `base[UC_ARGMAX+1]`;
+`_ctxfinish` reads it at `sp[UC_ARGMAX]`, where `sp` is what
+`_ctxreturn` hands over after the RET has popped one slot. **Off by one
+and the pointer read back is an argument**, so `uc_link` is garbage and
+the usual consequence is `exit(0)` -- indistinguishable from a clean
+finish at a shell prompt.
+
+`_ctxreturn` therefore passes its stack pointer to C and does nothing
+else: *the frame layout belongs to `makecontext`, which is C, so the
+stub carries no magic number.* No offset into `ucontext_t` appears in
+assembly at all.
+
+**`UC_ARGMAX` is a cap and not a convention.** On an ABI that passes the
+first six arguments in registers the stack image does not change shape
+with `argc`, so the bookkeeping cell can sit at a fixed offset whatever
+the count -- how glibc does it. Here only argument zero is in a register
+(6c's `RARG` = BP) and the rest are contiguous from `8(FP)`, so the cell
+must follow a FIXED number of argument slots. Eight, chosen rather than
+derived; php passes none.
+
+**`mc_regs[3..7]` are left exactly as they are, and that is why POSIX
+makes `getcontext` a precondition of `makecontext`**: those registers
+may hold 6c's caches for global variables, so the right value for a new
+context is the caller's, which the required `getcontext` already put
+there. Zeroing them would hand the new context a null view of every
+register-cached global.
+
+**`stack_t` is defined in `<ucontext.h>` and not in `<signal.h>`.**
+POSIX puts it beside `sigaltstack`, and APExp has no `sigaltstack`: a
+probe compiling `stack_t ss;` against `<signal.h>` would answer yes and
+the next thing such a package does is call the function. *This tree's
+commonest bug shape with the sign flipped* -- not a stub answering
+success but a TYPE answering for a capability that is not there. The
+`external/` grep found 39 files; every C use is a consumer behind
+`HAVE_SIGALTSTACK`, and the two `config.h` hits are gnulib's
+replacement slot, which a plain typedef does not disturb.
+
+### Two instruments, and what each can and cannot say
+
+`sys/lib/tests/ucontext-test.c` is the switch, and **it has a real
+reference implementation**, which is rare here: glibc has ucontext, so
+all six sections were checked against a working one before the VM saw
+them. 0 failures on gcc. A failure on Plan 9 that also fails on glibc is
+the test's bug.
+
+`sys/lib/tests/makectx-xcheck.{c,sh}` is the frame arithmetic, on the
+host. **It compiles the real `ucontext.c` rather than a copy** -- no
+extraction, no markers to rot -- with stubs for the three assembly
+primitives, which is possible only because everything structural in that
+file is C. It exists because the off-by-one above fails *silently* on
+the VM and costs a rebuild to find. No external oracle, so its control
+is a deliberate break, run and reverted: the ucp cell off by one gives
+**3** failures, `SP` not one slot below gives **1**.
+
+***AND ITS FIRST RUN REPORTED ITS OWN STUB.*** Section 3 failed on "did
+not exit", because the real `_setmcontext` jumps away and never returns
+while the stub recorded and returned, falling through to `_ctxfinish`'s
+`exit(0)`. The stub `longjmp`s now, which is what "does not return"
+means in a host program. *The instrument reporting itself is the shape
+to recognise rather than argue with* -- the assertion was right and the
+fake was wrong.
+
+**What neither covers**: `uc_sigmask` is saved and restored through
+`sigprocmask`, which on APE is a read and a write of one global
+(`signal/sigprocmask.c`), and nothing here exercises a context switch
+with signals actually blocked. And the header could not be
+syntax-checked against the APE headers on the host -- they reach each
+other through absolute `/sys/include/ape/...` paths that resolve only
+inside `apexp-sh` -- so the gcc check symlinks the two ucontext headers
+into a scratch directory instead and supplies `sigset_t` from glibc's
+own `<bits/types/sigset_t.h>`.

docs/notes/php.md

@@ -239,7 +239,9 @@ version, then add one library at a time.
 
 ## Open, as a list
 
-* fibers: no context switch (§1) — **decide `ucontext` or boost asm**
+* fibers: `ucontext` is written (`docs/notes/libap.md`) and passes on
+  the host against glibc; **not yet run on the VM**, and php has not
+  been pointed at it (`ZEND_FIBER_UCONTEXT` in the `config.h`)
 * `main/php_config.h` not written yet; the inventory is derived, the
   answers are not
 * `memfd_create` is declared with no implementation, which is a libap

sys/include/ape/ucontext.h

@@ -0,0 +1,62 @@
+#ifndef __UCONTEXT_H
+#define __UCONTEXT_H
+
+#include <signal.h>
+#include <ucontext_arch.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * POSIX user contexts.
+ *
+ * WHERE IT CAME FROM.  php's `Zend/zend_fibers.c' is unconditional --
+ * `Fiber' is a language class -- and needs one of two context switches:
+ * boost.context's GAS assembly, or this.  APExp had neither.  See
+ * `docs/notes/php.md'.
+ *
+ * ***`stack_t' IS DEFINED HERE AND NOT IN <signal.h>, ON PURPOSE.***
+ * POSIX puts it in <signal.h>, beside `sigaltstack' -- and **APExp has
+ * no `sigaltstack'**.  Putting the type where POSIX puts it would make
+ * an autoconf probe that compiles `stack_t ss;' against <signal.h>
+ * answer YES, and the next thing such a package does is call
+ * `sigaltstack'.  *That is this tree's commonest bug shape with the
+ * sign flipped*: not a stub answering success, but a TYPE answering for
+ * a capability that is not there.  Here the type is reachable only by
+ * including <ucontext.h>, which is a deliberate act.
+ * The `external/' grep before adding the name found 39 files; every
+ * C use is a CONSUMER behind `HAVE_SIGALTSTACK', and the two `config.h'
+ * hits are gnulib's `/`*` #undef stack_t *`/' replacement slot, which a
+ * plain typedef does not disturb.
+ */
+typedef struct {
+	void  *ss_sp;
+	long   ss_size;
+	int    ss_flags;
+} stack_t;
+
+/*
+ * `uc_mcontext' IS FIRST and must stay first: `_getmcontext' and
+ * friends are handed `&ucp->uc_mcontext' by the C wrappers, so nothing
+ * in the assembly depends on this layout -- but keeping the machine
+ * state at offset 0 means a debugger looking at a `ucontext_t' finds
+ * the registers where it expects them.
+ */
+typedef struct __ucontext {
+	mcontext_t          uc_mcontext;
+	struct __ucontext  *uc_link;
+	stack_t             uc_stack;
+	sigset_t            uc_sigmask;
+} ucontext_t;
+
+extern int  getcontext(ucontext_t *);
+extern int  setcontext(const ucontext_t *);
+extern int  swapcontext(ucontext_t *, const ucontext_t *);
+extern void makecontext(ucontext_t *, void (*)(void), int, ...);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __UCONTEXT_H */