Claude/upgrade ape c library mm z gd

CLAUDE.md

@@ -303,8 +303,16 @@ it gates if a marker stops matching) and links them against a fake
 a crash in the function but the function freeing a segment the caller
 is still using*, which is invisible from inside and expensive from the
 VM. No external oracle, so its control is a deliberate break and has
-been run twice: the old start-address-only rule gives **8** failures,
-the old `_SEGDETACH` fallback gives **2**.
+been run **four times**: the old start-address-only rule gives **8**
+failures, the old `_SEGDETACH` fallback **2**, deleting the malloc
+arm's page round-up **1**, and `*basep = p` **3**.
+***AND THAT LAST ONE GAVE 0 THE FIRST TIME.*** Its sections read
+`mmap_table[0].base` and then checked that what was released matched
+it — both sides from the same field, so it held however wrong the
+field was. The harness wraps `malloc` now, as it already wrapped
+`free`. *The assertion had been phrased against the thing under test
+instead of against something independent of it*, which is the shape
+to look for when a break does not fire.
 **`hostkencc/clisp-mirror.sh` stages clisp's generated tree on the
 host** so the host kencc can compile and inspect it -- `--flags` prints
 the compile line, `--compile` builds all 31 TUs and GATES. *Its
@@ -469,18 +477,21 @@ regular and whether `select()` then reports them readable; the subject
 is what *rio* serves, so run it in a rio window with no wish running,
 and note that it never READS any of them, since reading `/dev/mouse` is
 the call that hangs),
-**`mmaptrim-probe`** (does a large `mmap` come back 2MB-ALIGNED here —
-which is what decides whether php's Zend allocator ever has to trim an
-over-sized mapping at all. **Two of its four sections can kill the
-process, so it takes an ARGUMENT**: no argument runs the tail trim, `3`
-runs the head trim, and they never share a run, because a section that
-can kill the process hides every section after it. The host is the
-control and has been run: both trims survive there, and **8 of 8
-aligned on Linux is why php works there**, while *section 1's
-over-sized mapping is misaligned on Linux too* — so section 1 is
-information about the trim SIZE and only section 0 can clear or condemn
-anything. Calls `_mmapmark`, since a probe built against stale library
-code reports a lie shaped like the platform's answer), and
+**`mmaptrim-probe`** (**ANSWERED, and the answer it was not built to
+ask is the one that mattered.** Section 0: *0 of 8* 2MB mappings come
+back 2MB-aligned, every one misaligned by exactly `0x1ff000` — a 2MB
+grid, off by one page — so php's Zend allocator does trim; sections 2
+and 3 say both trims survive. **But three of the eight `mmap` calls
+FAILED**, and 10MB is not a memory limit, so **section 0b** asks how
+many mappings a process may HOLD, at three sizes, *with the prediction
+written down first*: slots give the same count at every size, address
+space gives one falling as 1/size. It is **5 at 2MB and 5 at 16MB**,
+and 64 at a size below `SEG_THRESHOLD` — the control. **Two of its
+sections can kill the process, so it takes an ARGUMENT**: no argument
+runs the tail trim, `3` runs the head trim, never sharing a run,
+because a section that can kill the process hides every section after
+it. Calls `_mmapmark`, since a probe built against stale library code
+reports a lie shaped like the platform's answer), and
 `mkrecipe-probe/`, which is a whole mk DIRECTORY because the question is
 what `mk` hands `rc` (`mk -k all`; `-k` is not optional).
 
@@ -1121,9 +1132,12 @@ the 13-file `jit/ir` code generator; opcache only has to COMPILE, since
 `opcache.enable_cli` is off by default, and all three of its shared
 memory macros must be answered NO — *`shm_open` answering success with
 a file in `/tmp` over a per-process `mmap` is `HAVE_WORKING_MPROTECT`
-in its exact shape.* (3) the Zend allocator's partial `munmap`, which
-is **fixed** in libap with one measurement still open
-(`mmaptrim-probe` section 0).
+in its exact shape.* (3) the Zend allocator's memory, which is
+**CLOSED**: the partial `munmap` is fixed and both trims are measured
+to survive on the VM, and the segment-slot ceiling that would have
+capped the Zend heap at about 10MB is gone with `SEG_THRESHOLD` at
+64MB. *The alignment question answered no — 0 of 8 — so the trims do
+run; they just work now.*
 **No extension in the first build**, whatever its library's state.
 
 **Queued**: **tkdesk** gets its mkfile in `sys/src/ape/app` and is **not
@@ -1676,6 +1690,12 @@ the index, so that nothing here is a surprise.
   reason it scrubs `FD_ISTTY` and `FD_ISREG`. `FD_BUFFEREDX` is poison:
   `read()` and `select()` both answer EIO for it without touching the
   fd.
+- **A process gets FIVE spare segment slots, measured**, so `mmap`
+  cannot be backed by `segattach` in general: `mmaptrim-probe` section
+  0b maps until refused at three sizes and answers 5 at 2MB and 5 at
+  16MB -- *the same count for eight times the memory*. `SEG_THRESHOLD`
+  is 64MB for that reason and anything below it is `malloc`-backed,
+  page-aligned by hand since `malloc` is not.
 - `/dev/snarf` is the clipboard and has no concept of ownership.
 - Plan 9 has no loopback unless `ip/ipconfig loopback /dev/null 127.1`
   has been run -- and that belongs in the machine's startup, not in
@@ -1768,10 +1788,22 @@ segment that address is in; *the idiom both of those break is how every
 POSIX program gets an ALIGNED allocation*, php's Zend allocator
 included, and the unknown-address `_SEGDETACH` fallback is gone
 deliberately, because destroying a stranger's segment to free a mapping
-we are not sure we made is worse than leaking; **`ctime_r` declared in
+we are not sure we made is worse than leaking; **`mmap` spending a
+SEGMENT SLOT on every mapping over 1MB when a process has only five**
+— the prose above the constant already said "Plan9 allows only ~8
+segments (NSEG)" and the threshold was set to 1MB anyway, *the limit
+known and the constant chosen as though it were not*; 64MB now, which
+exposed the deviation it had been hiding, namely that the
+`malloc`-backed arm returned a pointer that was **not page-aligned**
+as POSIX requires; **`ctime_r` declared in
 `<time.h>` and implemented nowhere**, found by `apfunc-sweep.py` while
 answering php's `config.h` and already the composition of two functions
-that were there; `poll()`'s three POSIX violations and `select()` on a
+that were there — ***and writing a caller for it found `asctime_r`
+declared by NO header at all***, defined 120 lines down its own file
+and returning implicit `int` to everyone, which `apdecl-sweep.py`
+cannot report because it compares a call against a DECLARED return
+type and there was none (*the inverse question — what does libap define
+that no header declares — has no instrument*); `poll()`'s three POSIX violations and `select()` on a
 regular file; itcl's
 four `sfbug`s; kencc's `false` keyword, the `L`-suffix width cap, the
 wide case label, the compound-literal zero fill, and both bit-field

docs/notes/libap.md

@@ -3871,3 +3871,60 @@ header DECLARES?" is the inverse sweep and does not exist yet** --
 `time/*.c` by hand found `asctime_r` the only real omission there, the
 other 43 unmentioned names being statics or declared in
 `<sys/time.h>`/`<unistd.h>`.
+
+### And the slot ceiling was five, so mmap stopped using segments
+
+`mmaptrim-probe` section 0 failed three of its eight 2MB mappings.
+Five 2MB mappings is 10MB, which is nothing, so the limit was never
+memory. Section 0b separates the two readings, with the prediction
+written down before the run -- *slots give the same small count at
+every size, address space gives a count falling as 1/size* -- and
+9front/amd64 answers:
+
+```
+  1044480 bytes each: 64 live (hit the array's end)  (63 MB held)
+  2097152 bytes each:  5 live                        (10 MB held)
+ 16777216 bytes each:  5 live                        (80 MB held)
+```
+
+**The same count at eight times the size, for eight times the memory.**
+The limit is segment slots, and there are five. The smallest size is
+the control: it is below `SEG_THRESHOLD`, so it is malloc-backed,
+takes no slot, and runs to the array's end.
+
+***THE PROSE ABOVE THE CONSTANT ALREADY SAID SO.*** `__p9_syscall.c`
+opened with "Plan9 allows only ~8 segments per process (NSEG in the
+kernel). Using segattach for every mmap quickly exhausts this limit"
+-- and then set `SEG_THRESHOLD` to 1MB, which spends a slot on
+anything a program is at all likely to allocate. *The limit was known
+and the constant was chosen as though it were not.* At 1MB, php's Zend
+allocator -- 2MB chunks -- would have stopped at about 10MB of heap.
+
+64MB now: a program needs five simultaneous 64MB mappings to reach the
+ceiling, and the detachable path is kept for the only case worth a
+slot, a mapping big enough that leaving it in the data segment for the
+life of the process would matter.
+
+**That exposed a second deviation it had been hiding.** POSIX says
+`mmap` answers a page-aligned address; `malloc` does not, and the
+malloc arm returned the raw `malloc` pointer. That was live for
+everything under the old 1MB threshold and nothing had noticed, and
+raising the threshold would have made it the common case. The arm
+over-allocates by a page and rounds up now -- **expressible only
+because the munmap table already separates `base` from `addr`** for
+partial unmaps, so the two changes pay for each other.
+
+Sections 6, 7 and 8 of `munmaptrim-xcheck` cover the new arms, through
+the real `p9_mmap_tracked` with a `_SEGATTACH` fake that answers
+page-aligned memory.
+
+***AND ONE OF ITS NEW CONTROLS DID NOT FIRE THE FIRST TIME.***
+Breaking `*basep = base` to `*basep = p` -- a `free` of a pointer
+`malloc` never returned -- gave **0 failures**, because sections 6 and
+7 read `mmap_table[0].base` and then checked that what was released
+matched it. Both sides came from the same field, so the check held
+however wrong the field was. The harness wraps `malloc` now, as it
+already wrapped `free`, and the same break gives 3 failures. *A check
+that cannot fail is not a check*, and the way it got written is worth
+naming: the assertion was phrased against the thing under test instead
+of against something independent of it.

docs/notes/php.md

@@ -194,10 +194,28 @@ about to use, and a tail trim's interior address fell through to
 `p9_munmap` has a partial-unmap arm now; the mechanism, the instruments
 and the two deliberate breaks are in `docs/notes/libap.md`.
 
-**What is still open is the platform half**, and it is cheap:
-`sys/lib/tests/mmaptrim-probe.c` section 0 asks whether `_SEGATTACH`
-already returns 2MB-aligned addresses. If it does, `zend_mm_chunk_alloc_int`
-takes its first branch every time and nothing is trimmed or held.
+**The platform half is measured now and this item is CLOSED**, but not
+the way it was expected to close. `mmaptrim-probe` on 9front/amd64:
+
+* section 0 — **0 of 8** 2MB mappings are 2MB-aligned, every one
+  misaligned by exactly `0x1ff000`, a 2MB grid off by one page. So
+  `zend_mm_chunk_alloc_int` does take its trimming branch.
+* sections 2 and 3 — **both trims survive**. The partial-unmap arm
+  works on the machine as well as on the host.
+* section 1 — the over-sized mapping came back aligned, *contradicting
+  section 0's five*, so the placement rule is size-dependent and one
+  sample of it is not a rule. In zend's real path it means no head
+  trim and one tail trim.
+
+***AND THE RUN FOUND A CEILING THAT WOULD HAVE MATTERED MORE.*** Three
+of section 0's eight `mmap` calls failed, at 10MB total, which is no
+memory limit at all. Section 0b separated it: **five** live mappings
+at 2MB and **five** at 16MB — the same count for eight times the
+memory — so the limit was Plan 9 segment slots, one spent by
+`p9_mmap_impl` on every mapping at or above a 1MB `SEG_THRESHOLD`.
+**Zend grows its heap in 2MB chunks, so php would have died at about
+10MB of heap**, long before any trimming question arose.
+`SEG_THRESHOLD` is 64MB now; `docs/notes/libap.md` has it.
 
 ## Extensions: the ones with a ported library
 
@@ -222,7 +240,6 @@ version, then add one library at a time.
 ## Open, as a list
 
 * fibers: no context switch (§1) — **decide `ucontext` or boost asm**
-* `mmaptrim-probe` section 0 on the VM — the one open measurement
 * `main/php_config.h` not written yet; the inventory is derived, the
   answers are not
 * `memfd_create` is declared with no implementation, which is a libap
@@ -235,6 +252,12 @@ version, then add one library at a time.
 
 ## Closed
 
+* **The Zend allocator's memory, §3** — the partial `munmap` is fixed
+  and both trims are measured to survive on the VM, and the
+  segment-slot ceiling that would have capped the Zend heap at about
+  10MB is gone. *The alignment question answered no, so the trims do
+  run; they just work.*
+
 * **`ctime_r`** — declared in `<time.h>` and implemented nowhere, found
   by `apfunc-sweep.py` while answering php's `config.h`. It is the
   composition of `localtime_r` and `asctime_r`, both of which were