diff --git a/CLAUDE.md b/CLAUDE.md index 0b231f6eb..d93449a90 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -303,8 +303,16 @@ it gates if a marker stops matching) and links them against a fake a crash in the function but the function freeing a segment the caller is still using*, which is invisible from inside and expensive from the VM. No external oracle, so its control is a deliberate break and has -been run twice: the old start-address-only rule gives **8** failures, -the old `_SEGDETACH` fallback gives **2**. +been run **four times**: the old start-address-only rule gives **8** +failures, the old `_SEGDETACH` fallback **2**, deleting the malloc +arm's page round-up **1**, and `*basep = p` **3**. +***AND THAT LAST ONE GAVE 0 THE FIRST TIME.*** Its sections read +`mmap_table[0].base` and then checked that what was released matched +it — both sides from the same field, so it held however wrong the +field was. The harness wraps `malloc` now, as it already wrapped +`free`. *The assertion had been phrased against the thing under test +instead of against something independent of it*, which is the shape +to look for when a break does not fire. **`hostkencc/clisp-mirror.sh` stages clisp's generated tree on the host** so the host kencc can compile and inspect it -- `--flags` prints the compile line, `--compile` builds all 31 TUs and GATES. *Its @@ -469,18 +477,21 @@ regular and whether `select()` then reports them readable; the subject is what *rio* serves, so run it in a rio window with no wish running, and note that it never READS any of them, since reading `/dev/mouse` is the call that hangs), -**`mmaptrim-probe`** (does a large `mmap` come back 2MB-ALIGNED here — -which is what decides whether php's Zend allocator ever has to trim an -over-sized mapping at all. **Two of its four sections can kill the -process, so it takes an ARGUMENT**: no argument runs the tail trim, `3` -runs the head trim, and they never share a run, because a section that -can kill the process hides every section after it. The host is the -control and has been run: both trims survive there, and **8 of 8 -aligned on Linux is why php works there**, while *section 1's -over-sized mapping is misaligned on Linux too* — so section 1 is -information about the trim SIZE and only section 0 can clear or condemn -anything. Calls `_mmapmark`, since a probe built against stale library -code reports a lie shaped like the platform's answer), and +**`mmaptrim-probe`** (**ANSWERED, and the answer it was not built to +ask is the one that mattered.** Section 0: *0 of 8* 2MB mappings come +back 2MB-aligned, every one misaligned by exactly `0x1ff000` — a 2MB +grid, off by one page — so php's Zend allocator does trim; sections 2 +and 3 say both trims survive. **But three of the eight `mmap` calls +FAILED**, and 10MB is not a memory limit, so **section 0b** asks how +many mappings a process may HOLD, at three sizes, *with the prediction +written down first*: slots give the same count at every size, address +space gives one falling as 1/size. It is **5 at 2MB and 5 at 16MB**, +and 64 at a size below `SEG_THRESHOLD` — the control. **Two of its +sections can kill the process, so it takes an ARGUMENT**: no argument +runs the tail trim, `3` runs the head trim, never sharing a run, +because a section that can kill the process hides every section after +it. Calls `_mmapmark`, since a probe built against stale library code +reports a lie shaped like the platform's answer), and `mkrecipe-probe/`, which is a whole mk DIRECTORY because the question is what `mk` hands `rc` (`mk -k all`; `-k` is not optional). @@ -1121,9 +1132,12 @@ the 13-file `jit/ir` code generator; opcache only has to COMPILE, since `opcache.enable_cli` is off by default, and all three of its shared memory macros must be answered NO — *`shm_open` answering success with a file in `/tmp` over a per-process `mmap` is `HAVE_WORKING_MPROTECT` -in its exact shape.* (3) the Zend allocator's partial `munmap`, which -is **fixed** in libap with one measurement still open -(`mmaptrim-probe` section 0). +in its exact shape.* (3) the Zend allocator's memory, which is +**CLOSED**: the partial `munmap` is fixed and both trims are measured +to survive on the VM, and the segment-slot ceiling that would have +capped the Zend heap at about 10MB is gone with `SEG_THRESHOLD` at +64MB. *The alignment question answered no — 0 of 8 — so the trims do +run; they just work now.* **No extension in the first build**, whatever its library's state. **Queued**: **tkdesk** gets its mkfile in `sys/src/ape/app` and is **not @@ -1676,6 +1690,12 @@ the index, so that nothing here is a surprise. reason it scrubs `FD_ISTTY` and `FD_ISREG`. `FD_BUFFEREDX` is poison: `read()` and `select()` both answer EIO for it without touching the fd. +- **A process gets FIVE spare segment slots, measured**, so `mmap` + cannot be backed by `segattach` in general: `mmaptrim-probe` section + 0b maps until refused at three sizes and answers 5 at 2MB and 5 at + 16MB -- *the same count for eight times the memory*. `SEG_THRESHOLD` + is 64MB for that reason and anything below it is `malloc`-backed, + page-aligned by hand since `malloc` is not. - `/dev/snarf` is the clipboard and has no concept of ownership. - Plan 9 has no loopback unless `ip/ipconfig loopback /dev/null 127.1` has been run -- and that belongs in the machine's startup, not in @@ -1768,10 +1788,22 @@ segment that address is in; *the idiom both of those break is how every POSIX program gets an ALIGNED allocation*, php's Zend allocator included, and the unknown-address `_SEGDETACH` fallback is gone deliberately, because destroying a stranger's segment to free a mapping -we are not sure we made is worse than leaking; **`ctime_r` declared in +we are not sure we made is worse than leaking; **`mmap` spending a +SEGMENT SLOT on every mapping over 1MB when a process has only five** +— the prose above the constant already said "Plan9 allows only ~8 +segments (NSEG)" and the threshold was set to 1MB anyway, *the limit +known and the constant chosen as though it were not*; 64MB now, which +exposed the deviation it had been hiding, namely that the +`malloc`-backed arm returned a pointer that was **not page-aligned** +as POSIX requires; **`ctime_r` declared in `` and implemented nowhere**, found by `apfunc-sweep.py` while answering php's `config.h` and already the composition of two functions -that were there; `poll()`'s three POSIX violations and `select()` on a +that were there — ***and writing a caller for it found `asctime_r` +declared by NO header at all***, defined 120 lines down its own file +and returning implicit `int` to everyone, which `apdecl-sweep.py` +cannot report because it compares a call against a DECLARED return +type and there was none (*the inverse question — what does libap define +that no header declares — has no instrument*); `poll()`'s three POSIX violations and `select()` on a regular file; itcl's four `sfbug`s; kencc's `false` keyword, the `L`-suffix width cap, the wide case label, the compound-literal zero fill, and both bit-field diff --git a/docs/notes/libap.md b/docs/notes/libap.md index 34adf2a95..9f8a0668b 100644 --- a/docs/notes/libap.md +++ b/docs/notes/libap.md @@ -3871,3 +3871,60 @@ header DECLARES?" is the inverse sweep and does not exist yet** -- `time/*.c` by hand found `asctime_r` the only real omission there, the other 43 unmentioned names being statics or declared in ``/``. + +### And the slot ceiling was five, so mmap stopped using segments + +`mmaptrim-probe` section 0 failed three of its eight 2MB mappings. +Five 2MB mappings is 10MB, which is nothing, so the limit was never +memory. Section 0b separates the two readings, with the prediction +written down before the run -- *slots give the same small count at +every size, address space gives a count falling as 1/size* -- and +9front/amd64 answers: + +``` + 1044480 bytes each: 64 live (hit the array's end) (63 MB held) + 2097152 bytes each: 5 live (10 MB held) + 16777216 bytes each: 5 live (80 MB held) +``` + +**The same count at eight times the size, for eight times the memory.** +The limit is segment slots, and there are five. The smallest size is +the control: it is below `SEG_THRESHOLD`, so it is malloc-backed, +takes no slot, and runs to the array's end. + +***THE PROSE ABOVE THE CONSTANT ALREADY SAID SO.*** `__p9_syscall.c` +opened with "Plan9 allows only ~8 segments per process (NSEG in the +kernel). Using segattach for every mmap quickly exhausts this limit" +-- and then set `SEG_THRESHOLD` to 1MB, which spends a slot on +anything a program is at all likely to allocate. *The limit was known +and the constant was chosen as though it were not.* At 1MB, php's Zend +allocator -- 2MB chunks -- would have stopped at about 10MB of heap. + +64MB now: a program needs five simultaneous 64MB mappings to reach the +ceiling, and the detachable path is kept for the only case worth a +slot, a mapping big enough that leaving it in the data segment for the +life of the process would matter. + +**That exposed a second deviation it had been hiding.** POSIX says +`mmap` answers a page-aligned address; `malloc` does not, and the +malloc arm returned the raw `malloc` pointer. That was live for +everything under the old 1MB threshold and nothing had noticed, and +raising the threshold would have made it the common case. The arm +over-allocates by a page and rounds up now -- **expressible only +because the munmap table already separates `base` from `addr`** for +partial unmaps, so the two changes pay for each other. + +Sections 6, 7 and 8 of `munmaptrim-xcheck` cover the new arms, through +the real `p9_mmap_tracked` with a `_SEGATTACH` fake that answers +page-aligned memory. + +***AND ONE OF ITS NEW CONTROLS DID NOT FIRE THE FIRST TIME.*** +Breaking `*basep = base` to `*basep = p` -- a `free` of a pointer +`malloc` never returned -- gave **0 failures**, because sections 6 and +7 read `mmap_table[0].base` and then checked that what was released +matched it. Both sides came from the same field, so the check held +however wrong the field was. The harness wraps `malloc` now, as it +already wrapped `free`, and the same break gives 3 failures. *A check +that cannot fail is not a check*, and the way it got written is worth +naming: the assertion was phrased against the thing under test instead +of against something independent of it. diff --git a/docs/notes/php.md b/docs/notes/php.md index 9140c8752..03a9e94df 100644 --- a/docs/notes/php.md +++ b/docs/notes/php.md @@ -194,10 +194,28 @@ about to use, and a tail trim's interior address fell through to `p9_munmap` has a partial-unmap arm now; the mechanism, the instruments and the two deliberate breaks are in `docs/notes/libap.md`. -**What is still open is the platform half**, and it is cheap: -`sys/lib/tests/mmaptrim-probe.c` section 0 asks whether `_SEGATTACH` -already returns 2MB-aligned addresses. If it does, `zend_mm_chunk_alloc_int` -takes its first branch every time and nothing is trimmed or held. +**The platform half is measured now and this item is CLOSED**, but not +the way it was expected to close. `mmaptrim-probe` on 9front/amd64: + +* section 0 — **0 of 8** 2MB mappings are 2MB-aligned, every one + misaligned by exactly `0x1ff000`, a 2MB grid off by one page. So + `zend_mm_chunk_alloc_int` does take its trimming branch. +* sections 2 and 3 — **both trims survive**. The partial-unmap arm + works on the machine as well as on the host. +* section 1 — the over-sized mapping came back aligned, *contradicting + section 0's five*, so the placement rule is size-dependent and one + sample of it is not a rule. In zend's real path it means no head + trim and one tail trim. + +***AND THE RUN FOUND A CEILING THAT WOULD HAVE MATTERED MORE.*** Three +of section 0's eight `mmap` calls failed, at 10MB total, which is no +memory limit at all. Section 0b separated it: **five** live mappings +at 2MB and **five** at 16MB — the same count for eight times the +memory — so the limit was Plan 9 segment slots, one spent by +`p9_mmap_impl` on every mapping at or above a 1MB `SEG_THRESHOLD`. +**Zend grows its heap in 2MB chunks, so php would have died at about +10MB of heap**, long before any trimming question arose. +`SEG_THRESHOLD` is 64MB now; `docs/notes/libap.md` has it. ## Extensions: the ones with a ported library @@ -222,7 +240,6 @@ version, then add one library at a time. ## Open, as a list * fibers: no context switch (§1) — **decide `ucontext` or boost asm** -* `mmaptrim-probe` section 0 on the VM — the one open measurement * `main/php_config.h` not written yet; the inventory is derived, the answers are not * `memfd_create` is declared with no implementation, which is a libap @@ -235,6 +252,12 @@ version, then add one library at a time. ## Closed +* **The Zend allocator's memory, §3** — the partial `munmap` is fixed + and both trims are measured to survive on the VM, and the + segment-slot ceiling that would have capped the Zend heap at about + 10MB is gone. *The alignment question answered no, so the trims do + run; they just work.* + * **`ctime_r`** — declared in `` and implemented nowhere, found by `apfunc-sweep.py` while answering php's `config.h`. It is the composition of `localtime_r` and `asctime_r`, both of which were diff --git a/sys/lib/tests/munmaptrim-xcheck.c b/sys/lib/tests/munmaptrim-xcheck.c index 3261adad8..238b3da48 100644 --- a/sys/lib/tests/munmaptrim-xcheck.c +++ b/sys/lib/tests/munmaptrim-xcheck.c @@ -43,10 +43,10 @@ * do". Here it is a stub answering DESTRUCTION where the answer is * "not mine". * - * THE CONTROL IS A DELIBERATE BREAK, AND IT HAS BEEN RUN TWICE -- this - * check has no external oracle, so a clean 0 proves nothing until the - * instrument has been shown to notice. Both breaks were made in the real - * source, measured, and reverted: + * THE CONTROL IS A DELIBERATE BREAK, AND IT HAS BEEN RUN FOUR TIMES -- + * this check has no external oracle, so a clean 0 proves nothing until + * the instrument has been shown to notice. Each break was made in the + * real source, measured, and reverted: * * `if(len >= mmap_table[i].len)' -> `if(1)', i.e. the old rule that * matched on the start address alone: **8 failures**, in @@ -58,15 +58,34 @@ * in section 5, one of them for an address ONE PAGE PAST the end of a * live mapping. * + * the malloc arm's round-up deleted, `p = base': **1 failure**, in + * section 6. Only one, and correctly so: section 7 still passes, + * because zend aligns the chunk from whatever address it is given, + * so a non-page-aligned mmap breaks POSIX without breaking zend. + * + * `*basep = base' -> `*basep = p': **3 failures**, in + * sections 6 and 7 -- a `free' of a pointer `malloc' never returned. + * + * ***AND THAT LAST CONTROL FAILED TO FIRE THE FIRST TIME, 0 FAILURES.*** + * Sections 6 and 7 read `mmap_table[0].base' and then checked that what + * was released matched it -- which holds however wrong the field is, + * since both sides come from the same field. *A check that cannot + * fail.* They compare against the pointer `malloc' actually returned + * now, which the wrapper below records; the free was already wrapped + * for exactly this reason and the malloc was not. + * * Exit status is the failure count. */ +#define _GNU_SOURCE #include #include #include +#include +#include +#include #include - -#define SEG_THRESHOLD (1024*1024) +#include /* ---- the fakes the extracted code links against ---------------- */ @@ -74,6 +93,26 @@ static void *released[16]; static int released_seg[16]; static int nreleased; +static int nattached; + +/* + * `_SEGATTACH' models what Plan 9's does and no more: it answers a + * PAGE-ALIGNED, zeroed region that `_SEGDETACH' can later take back + * whole. It must return real memory, because the extracted + * `p9_mmap_impl' memsets what it is given -- a fake answering NULL + * would make the segment arm crash rather than be tested. + */ +static void * +_SEGATTACH(int attr, char *class, void *va, unsigned long len) +{ + void *p = NULL; + (void)attr; (void)class; (void)va; + if(posix_memalign(&p, 4096, len) != 0) + return (void*)-1; + nattached++; + return p; +} + static void * _SEGDETACH(void *p) { @@ -98,14 +137,36 @@ my_free(void *p) } } -/* `p9_mmap_tracked' is extracted too, but its allocator half is not - * wanted here; the harness fills the table through this instead, so the - * cases can name addresses that mean something when printed. */ +/* + * ***AND malloc IS WRAPPED FOR THE SAME REASON THE FREE IS, WHICH THE + * FIRST VERSION OF SECTIONS 6 AND 7 MISSED.*** Those sections read + * `mmap_table[0].base' and then checked that what was released matched + * it -- which is true however wrong `base' is, because both sides come + * from the same field. Breaking the real code's `*basep = base' to + * `*basep = p' on purpose produced **0 failures**: a check that cannot + * fail. The harness needs its own handle on the pointer malloc + * actually returned, and this is it. + */ +static void *last_malloc; +static int nmalloc; + +/* taken before the macro below hides the name */ +static void *(*real_malloc)(size_t) = malloc; + +#define malloc my_malloc +static void * +my_malloc(size_t n) +{ + last_malloc = real_malloc(n); + nmalloc++; + return last_malloc; +} /* ---- the real code ------------------------------------------- */ #include "munmaptrim-extract.h" #undef free +#undef malloc /* ---- the harness ---------------------------------------------- */ @@ -116,6 +177,7 @@ reset(void) { mmap_table_n = 0; nreleased = 0; + nattached = 0; } static void @@ -169,7 +231,20 @@ main(void) expect("entries left", mmap_table_n, 0); expect("releases", nreleased, 1); expect("released the base", released[0] == BASE, 1); - expect("released as a segment", released_seg[0], 1); + /* + * `track' derives is_seg from the length, as the real code does, + * so which release this expects is a FACT ABOUT THE THRESHOLD. + * It said `segment' while SEG_THRESHOLD was 1MB and the 2MB + * fixture was above it; at 64MB the same fixture is malloc-backed + * and it says the other thing. *Stated as a dependency rather + * than a constant*, because the first run after the threshold + * moved failed here and the failing line said nothing about + * why. The segment path is section 8's, through the real + * `p9_mmap_tracked'. + */ + expect("the 2MB fixture is below SEG_THRESHOLD", + CHUNK < SEG_THRESHOLD, 1); + expect("released as malloc, not segment", released_seg[0], 0); /* ---- 1: head trim, then free the rest ---- */ printf("\nSection 1: head trim keeps the tail live\n"); @@ -241,6 +316,102 @@ main(void) expect("releases, one past the end", nreleased, 0); expect("the real entry is still untouched", mmap_table_n, 1); + /* ---- 6: the malloc arm must answer a PAGE-ALIGNED address ---- */ + printf("\nSection 6: p9_mmap_tracked's malloc arm is page-aligned\n"); + reset(); + expect("the chunk size is below SEG_THRESHOLD", + CHUNK < SEG_THRESHOLD, 1); + { + void *a[4]; + int k, misaligned = 0, basesame = 0, badbase = 0; + + for(k = 0; k < 4; k++){ + a[k] = p9_mmap_tracked(NULL, CHUNK, 0, + MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); + if(a[k] == MAP_FAILED){ + printf(" FAIL mmap %d failed\n", k); + fails++; + continue; + } + if((uintptr_t)a[k] & (MMAP_PAGE - 1)) + misaligned++; + if(mmap_table[k].base == mmap_table[k].addr) + basesame++; + if(mmap_table[k].base != last_malloc) + badbase++; + /* the whole range must be writable and zeroed */ + if(((char*)a[k])[0] || ((char*)a[k])[CHUNK-1]) + printf(" (not zeroed at %d)\n", k); + memset(a[k], 0x3c, CHUNK); + } + expect("misaligned returns", misaligned, 0); + expect("base is the pointer malloc returned", badbase, 0); + expect("one malloc per mapping", nmalloc, 4); + expect("no segment slot spent", nattached, 0); + expect("entries recorded", mmap_table_n, 4); + /* base == addr only when malloc happened to land on a page; + * this is information, not a rule, so it is printed. */ + printf(" (base == addr for %d of 4 -- luck, not a rule)\n", + basesame); + for(k = 0; k < 4; k++) + p9_munmap(a[k], CHUNK); + expect("all released", nreleased, 4); + expect("released as malloc, not segment", released_seg[0], 0); + expect("entries left", mmap_table_n, 0); + } + + /* ---- 7: head trim on a malloc-backed mapping frees the BASE ---- */ + printf("\nSection 7: zend's sequence over the malloc arm\n"); + reset(); + { + char *q = p9_mmap_tracked(NULL, OVER, 0, + MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); + size_t o; + + if(q == MAP_FAILED){ + printf(" FAIL mmap failed\n"); + fails++; + } else { + void *b = last_malloc; /* NOT mmap_table[0].base: + * see the malloc wrapper */ + + o = (uintptr_t)q & (CHUNK-1); + o = o ? CHUNK - o : 0; + p9_munmap(q, o); + p = q + o; + expect("chunk is 2MB-aligned", + ((uintptr_t)p & (CHUNK-1)) == 0, 1); + expect("nothing released by the trim", nreleased, 0); + expect("base is the pointer malloc returned", + mmap_table[0].base == b, 1); + memset(p, 0x7e, CHUNK); /* the kept range must be live */ + p9_munmap(p, mmap_table[0].len); + expect("released", nreleased, 1); + expect("released the MALLOC pointer, not the chunk", + released[0] == b, 1); + expect("entries left", mmap_table_n, 0); + } + } + + /* ---- 8: at or above the threshold it is still a segment ---- */ + printf("\nSection 8: >= SEG_THRESHOLD still takes a segment\n"); + reset(); + { + void *big = p9_mmap_tracked(NULL, SEG_THRESHOLD, 0, + MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); + if(big == MAP_FAILED){ + printf(" FAIL mmap failed\n"); + fails++; + } else { + expect("one segment attached", nattached, 1); + expect("recorded as a segment", mmap_table[0].is_seg, 1); + expect("base == addr for a segment", + mmap_table[0].base == mmap_table[0].addr, 1); + p9_munmap(big, SEG_THRESHOLD); + expect("detached", released_seg[0], 1); + } + } + printf("\nmunmaptrim-xcheck: %d failures\n", fails); return fails; } diff --git a/sys/lib/tests/munmaptrim-xcheck.sh b/sys/lib/tests/munmaptrim-xcheck.sh index e10ff2430..2bedb0b64 100644 --- a/sys/lib/tests/munmaptrim-xcheck.sh +++ b/sys/lib/tests/munmaptrim-xcheck.sh @@ -35,19 +35,34 @@ fi { echo "/* GENERATED by munmaptrim-xcheck.sh -- do not edit. */" + grep '^#define SEG_THRESHOLD' "$SRC" + grep '^#define MMAP_PAGE' "$SRC" sed -n '/^static struct MmapEntry {/,/^static int mmap_table_cap/p' "$SRC" awk ' - /^p9_munmap\(void \*addr, size_t len\)$/ { print prev; inf = 1 } + /^p9_mmap_impl\(void \*addr, size_t len/ { print prev; inf = 1 } + /^p9_mmap_tracked\(void \*addr, size_t len/ { print prev; inf = 1 } + /^p9_munmap\(void \*addr, size_t len\)$/ { print prev; inf = 1 } inf { print } inf && /^}$/ { inf = 0 } { prev = $0 } ' "$SRC" } > "$OUT" -# gate: both halves must be there -grep -q 'struct MmapEntry' "$OUT" || { echo "munmaptrim-xcheck: table marker did not match" >&2; exit 2; } -grep -q 'p9_munmap' "$OUT" || { echo "munmaptrim-xcheck: p9_munmap marker did not match" >&2; exit 2; } -grep -q '^int$' "$OUT" || { echo "munmaptrim-xcheck: lost p9_munmap's return type" >&2; exit 2; } +# gate: every piece must be there, and the DEFINES are part of it -- +# the threshold decides which arm each case exercises, so a harness +# compiled without it would silently test one path twice. +for marker in '#define SEG_THRESHOLD' '#define MMAP_PAGE' 'struct MmapEntry' \ + 'p9_mmap_impl' 'p9_mmap_tracked' 'p9_munmap'; do + grep -q -- "$marker" "$OUT" || { + echo "munmaptrim-xcheck: marker '$marker' did not match" >&2 + exit 2 + } +done +# the return types live on their own line above each name +[ "$(grep -c '^static void \*$' "$OUT")" -ge 1 ] || { + echo "munmaptrim-xcheck: lost p9_mmap_impl's return type" >&2; exit 2; } +grep -q '^int$' "$OUT" || { + echo "munmaptrim-xcheck: lost p9_munmap's return type" >&2; exit 2; } cc=${CC:-gcc} $cc -Wall -Wno-unused-function -o "$(dirname "$0")/munmaptrim-xcheck" \ diff --git a/sys/src/ape/lib/ap/plan9/__p9_syscall.c b/sys/src/ape/lib/ap/plan9/__p9_syscall.c index 48ea98f93..a883de894 100644 --- a/sys/src/ape/lib/ap/plan9/__p9_syscall.c +++ b/sys/src/ape/lib/ap/plan9/__p9_syscall.c @@ -144,30 +144,70 @@ p9retl(long r) #endif /* - * Plan9 allows only ~8 segments per process (NSEG in the kernel). - * Using segattach for every mmap quickly exhausts this limit, causing - * "insufficient physical memory" errors. Strategy: + * Plan 9 gives a process a small fixed number of segments, so segattach + * cannot be the general backing for mmap. Strategy: * * anonymous, len < SEG_THRESHOLD: use malloc() (no new segment, sbrk-backed) - * anonymous, len >= SEG_THRESHOLD: use segattach() (page-aligned, zeroed) + * anonymous, len >= SEG_THRESHOLD: use segattach() (detachable) * file-backed: always read into malloc'd buffer (MAP_SHARED write-back * is not supported — changes are private) * - * Lowering SEG_THRESHOLD from 65536 to 1MB dramatically reduces the - * number of segattach calls for programs like bash that use mmap as - * a general allocator for medium-sized chunks. + * ***THE THRESHOLD IS 64MB BECAUSE THE NUMBER OF SLOTS IS FIVE, AND + * THAT IS MEASURED RATHER THAN RECALLED.*** `sys/lib/tests/ + * mmaptrim-probe.c' section 0b maps until it is refused, at three + * sizes, and on 9front/amd64 answers: + * + * 1044480 bytes each: 64 live (63 MB held) <- under the old + * threshold: malloc + * 2097152 bytes each: 5 live (10 MB held) + * 16777216 bytes each: 5 live (80 MB held) + * + * The same count at eight times the size, for eight times the memory, + * is what says the limit is SLOTS and not memory or address space -- + * the prediction was written down before the run, with the opposite + * observation (a count falling as 1/size) named as its refutation. + * The smallest size is the control and runs to the array's end. + * + * *The prose above this used to say "~8 segments (NSEG)" and the + * threshold was set to 1MB anyway*, which is the whole bug: the limit + * was known and the constant was chosen as though it were not. At 1MB + * a program could hold five mappings of a megabyte each and then fail, + * and php's Zend allocator -- which grows its heap in 2MB chunks -- + * would have stopped at about 10MB of heap. + * + * At 64MB a program needs five simultaneous 64MB mappings to reach the + * ceiling, and the detachable path is kept for the case it is actually + * worth a slot: a mapping large enough that leaving it in the data + * segment for the life of the process would matter. */ -#define SEG_THRESHOLD (1024*1024) /* 1 MB — use segattach only above this */ +#define SEG_THRESHOLD (64*1024*1024) /* use segattach only above this */ + +/* + * The page size mmap rounds to. `sysconf(_SC_PAGESIZE)' answers a + * hardcoded 4096 (`conf/sysconf.c'), so this is that constant and not + * an independent guess; if one moves, both must. + */ +#define MMAP_PAGE 4096 /* * p9_mmap_impl: core anonymous mmap emulation. - * Returns the allocated pointer on success, MAP_FAILED on error (sets errno). - * Uses void* throughout to preserve 64-bit addresses on amd64. + * Returns the mapping's address on success, MAP_FAILED on error (sets + * errno). Uses void* throughout to preserve 64-bit addresses on amd64. + * + * `*basep' is what must eventually be handed to _SEGDETACH()/free(), + * and is NOT always the address returned: **POSIX says mmap answers a + * page-aligned address and malloc does not**, so the malloc arm + * over-allocates by a page and rounds up. That deviation was live for + * everything under the old 1MB threshold and nothing had noticed; the + * threshold above makes it the common case, so it is fixed here rather + * than inherited. The base/address split the munmap table already + * carries for partial unmaps is exactly what makes it expressible. */ static void * -p9_mmap_impl(void *addr, size_t len, int prot, int flags, int fd, off_t off) +p9_mmap_impl(void *addr, size_t len, int prot, int flags, int fd, off_t off, + void **basep, int *segp) { - void *p; + void *p, *base; int is_seg; (void)prot; /* Plan9 has no per-page protection */ @@ -179,17 +219,19 @@ p9_mmap_impl(void *addr, size_t len, int prot, int flags, int fd, off_t off) is_seg = (len >= SEG_THRESHOLD); if(is_seg){ - p = _SEGATTACH(0, "memory", 0, len); + p = base = _SEGATTACH(0, "memory", 0, len); if(p == (void*)-1){ errno = ENOMEM; return MAP_FAILED; } } else { - p = malloc(len); - if(p == NULL){ + base = malloc(len + MMAP_PAGE - 1); + if(base == NULL){ errno = ENOMEM; return MAP_FAILED; } + p = (void*)(((uintptr_t)base + MMAP_PAGE - 1) + & ~(uintptr_t)(MMAP_PAGE - 1)); } memset(p, 0, len); @@ -200,12 +242,14 @@ p9_mmap_impl(void *addr, size_t len, int prot, int flags, int fd, off_t off) lseek(fd, off, SEEK_SET); n = read(fd, p, len); if(n < 0){ - if(is_seg) _SEGDETACH(p); - else free(p); + if(is_seg) _SEGDETACH(base); + else free(base); errno = EIO; return MAP_FAILED; } } + *basep = base; + *segp = is_seg; return p; } @@ -239,7 +283,9 @@ static int mmap_table_cap = 0; void * p9_mmap_tracked(void *addr, size_t len, int prot, int flags, int fd, off_t off) { - void *p = p9_mmap_impl(addr, len, prot, flags, fd, off); + void *base; + int is_seg; + void *p = p9_mmap_impl(addr, len, prot, flags, fd, off, &base, &is_seg); if(p == MAP_FAILED) return MAP_FAILED; @@ -257,10 +303,10 @@ p9_mmap_tracked(void *addr, size_t len, int prot, int flags, int fd, off_t off) * safe direction: see p9_munmap's tail. */ } if(mmap_table_n < mmap_table_cap){ - mmap_table[mmap_table_n].base = p; + mmap_table[mmap_table_n].base = base; mmap_table[mmap_table_n].addr = p; mmap_table[mmap_table_n].len = len; - mmap_table[mmap_table_n].is_seg = (len >= SEG_THRESHOLD); + mmap_table[mmap_table_n].is_seg = is_seg; mmap_table_n++; } return p;