diff --git a/CLAUDE.md b/CLAUDE.md index bd1485c95..e00ccfb0c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -366,8 +366,17 @@ which reads like the mkfile's rule and is not it (`ext/hash/hash_sha3.c` takes them too, from `ext/hash`), and it passed `CFLAGS` through, whose leading **`-c` means "compile only" to `pcc` and "print `const if`" to `6c`** -- dozens of lines that read -exactly like diagnostics. *It cannot ask about the LINK*, which stays -the VM's. +exactly like diagnostics. +***AND `--undefined` ASKS THE LINK'S QUESTION WITHOUT A LINKER, WHICH +THIS PARAGRAPH FIRST SAID IT COULD NOT***: `6c -S` names every symbol +each unit DEFINES (`<>` marks a file-local one) and REFERENCES, so +references minus global definitions minus each unit's own statics, +over all 515, is what `6l` would complain about -- handed to +`apfunc-sweep.py` for "does libap have it". **READ, do not gate**: the +sweep sees C function definitions only, so assembly reads as +`declared` and DATA as `absent` (ten of php's 283 are one or the +other, all present). *Its positive control is one command and names +the three symbols the VM's `6l` actually named.* ***AND `hostkencc/` BUILDS ALL TEN C COMPILERS WITH gcc***, from the shipped `sys/src/cmd/cc` and `[1-9kqv]c` plus go1.4's `lib9`/`libbio`, already vendored at `sys/src/external/go1.4` — which **compiled with gcc @@ -1195,15 +1204,22 @@ silent: **php — ***ALL 514 TRANSLATION UNITS COMPILE***, measured on the HOST, and the notes are in `docs/notes/php.md`.** 8.5.12, vendored at -`sys/src/external/php`; `sh sys/lib/tests/hostkencc/php-mirror.sh ---compile` answers `compiled 514, errors 0, missing 0`. **The LINK is -untouched and is the VM's**: the host has no libap and `hostkencc` -deliberately does not build the linkers, so the undefined-symbol class -— a `config.h` claiming a name libap only DECLARES — has not been -asked. *That question was checked by hand for the resolver family and -it held*: `res_query.c`'s `weak_alias(res_query, res_search)` is a -no-op on kencc, and `network/res_aliases.c` exists precisely to define -the strong forwarders. +`sys/src/external/php`; `--compile` answers `compiled 515, errors 0, +missing 0` and **`--undefined` says the program wants 283 symbols +from outside php, every one of which libap has** (273 the sweep +names outright; ten it misreports because they are assembly or DATA, +each opened by hand). **So the link is PREDICTED to succeed and has +not been run**, and the refutation condition is a name outside those +283 — which is what a newly reached `static inline` would be. +***`mk install` DID reach `6l` and named four***: `_volatile__`, from +`__asm__ __volatile__("")`, which kencc lexed as a function call +because it knew `__asm__` and not GNU's qualifiers (50 vendored files +use that spelling, and the two-line test for it found +`__extension__` eating the next character); and three lexbor +`lxb_dom_collection_*`, because **kencc emits an unused `static +inline` where gcc at -O2 discards it**, so a header-only helper +calling into a file upstream's `config.m4` does not list is a link +error here and silence there. Both fixed. **The first sweep returned EIGHT errors in six files and all six were fixed without a VM round trip** — three of them kencc's (a pointer assigned to a `bool`; `bool visible;` declaring nothing, because diff --git a/docs/notes/instruments.md b/docs/notes/instruments.md index 37691f059..e07b4a725 100644 --- a/docs/notes/instruments.md +++ b/docs/notes/instruments.md @@ -1190,9 +1190,34 @@ like diagnostics. *A flag one letter from another flag is checked rather than recalled, and these two meanings are in two different programs.* -**What it does not measure is the LINK** — the host has no libap and -`hostkencc` deliberately does not build the linkers — so the -undefined-symbol class stays the VM's. +***AND `--undefined` ASKS THE LINK'S QUESTION WITHOUT A LINKER, WHICH +THIS ENTRY FIRST SAID WAS IMPOSSIBLE.*** The original wording was +"the host has no libap and `hostkencc` deliberately does not build +the linkers — so the undefined-symbol class stays the VM's", and +neither fact is the obstacle it looks like. `6c -S` prints every +symbol a unit DEFINES (`TEXT`/`GLOBL`, `<>` marking a file-local one) +and every symbol it REFERENCES; across all 515 units, references +minus global definitions minus each unit's own statics is exactly +what `6l` would complain about. The result is handed to +`apfunc-sweep.py`, which says whether libap has each name. + +**Read that answer, do not gate on it.** The sweep looks for C +function definitions, so ten of the 283 come back wrong in two +predictable ways — `getcontext` and `swapcontext` are ASSEMBLY and +read as `declared`; `stdin`, `stdout`, `stderr`, `environ`, +`h_errno`, `_ctype` and `_errnoloc` are DATA and read as `absent`; +and `signal`, spelled `(*signal(int, void (*)(int)))(int)`, matches no +definition-shaped regex. All ten are present. *A sweep over function +names, asked about a variable, answers `absent` confidently* — the +caveat CLAUDE.md already carried for `HAVE_TM_ZONE`, arriving in its +other form. + +**Its positive control is a real failure and costs one command**: +`--undefined ext/lexbor/lexbor/dom/interfaces/node.c` narrows the +program to one unit and names `lxb_dom_collection_create`, `_init` +and `_destroy` among its 37 — *the three the VM's `6l` named*. A mode +that reports an empty set is indistinguishable from one that cannot +see, which is why the number is written down. ## `boolptr-test` — a pointer converted to `bool` diff --git a/docs/notes/kencc.md b/docs/notes/kencc.md index 0dc7c1494..3e83d78ac 100644 --- a/docs/notes/kencc.md +++ b/docs/notes/kencc.md @@ -3289,3 +3289,66 @@ one fault that cannot be found by reading the subject* — and this one also explains why `exits()` losing a buffered `diagbuf` was the first suspicion and the wrong one: go1.4's `libbio` registers an `atexit` flush, so a diagnostic would have appeared. + +## The link found two more in the same lexer block + +php's objects all compiled, and `6l` then named four undefined +symbols. Three were a build-list question (`docs/notes/php.md`); the +fourth was this compiler, and reproducing it exposed a second fault +beside it that had been there far longer. + +### `__asm__ __volatile__("")` called a function named `_volatile__` + +`Zend/zend_gdb.c:57` is + + ZEND_API zend_never_inline void __jit_debug_register_code(void) + { + __asm__ __volatile__(""); + } + +and the link answered + + __jit_debug_register_code: undefined: _volatile__ in ... + +`lex.c` makes `__asm__` invisible by consuming from the opening `(` to +the matching `)`. **GNU's grammar is `asm [volatile] [inline] [goto] ( +... )`**, and nothing here knew about the qualifiers: the scanner +looked for `(`, found `_`, and handed the identifier to the parser, +which called it. *A function nobody wrote, in an object file, found at +the link* — the compile said only `function not declared`, which in a +C89-tolerant compiler is a warning-shaped line in a sea of them. + +The qualifiers are skipped now — `volatile`, `__volatile`, +`__volatile__`, `inline`, `__inline`, `__inline__`, `goto` — and an +identifier that is none of them is **diagnosed**, because the +construct admits nothing else there. **50 vendored files use the +spelling**, 15 of them php's and the rest spread over pcre2, m4, +libpng, perl, libressl, gnulib, diffutils, coreutils, zlib and zstd, +so this was not a php question. + +### And `__extension__` had been eating the next character + +The `__asm__` fix needed a one-line test, and the test next to it did +not compile either: + + __extension__ int yy; + +became **`nt yy;`** — `6c -S` emitted `GLOBL nt+0(SB)`. The no-argument +branch of the same block ended `unget(ac); goto l0;`, and that is +wrong in a way that reads as right: `unget` only sets `peekc`, while +`l0` is BELOW the `if(peekc != IGN)` test at the top of `yylex`, so +the next `GETC` reads straight past the character. It is `c = ac; +goto l1;` now — lex the character already in hand — which is the idiom +the `__func__` case twenty lines down was already using. + +**Nothing in the tree had noticed**, and the reason is worth keeping: +`__extension__` is almost always followed by a token whose first +character is not load-bearing to the diagnosis, so the failure reads +as a syntax error in the code rather than as a lost character. *The +bug was found by writing a two-line test for a different bug*, which +is the cheapest way it could have been found and was not deliberate. + +Both fixes are in the block the string-literal fix of the previous +round also touched, and the controls are the same three: `agree.c` +identical, `--probe` ten `regpair-min` and eight `regpair-probe` with +`kc` and `vc` at `:242`, clisp's 31 TUs clean. diff --git a/docs/notes/php.md b/docs/notes/php.md index 48e07f943..0b3a364dc 100644 --- a/docs/notes/php.md +++ b/docs/notes/php.md @@ -707,11 +707,44 @@ unchanged: **sound for "does it compile" and "what does it diagnose", unsound for anything keyed on `long` being 32 bits**, since the host build is LP64. -**What it does NOT measure is the LINK**, which is the next class of -failure and is the VM's: a `config.h` claiming a name libap only -DECLARES fails there, far from the claim. The resolver family below is -exactly that shape caught early — `apfunc-sweep`'s middle state — and -it held, but it had to be checked rather than assumed. +***AND IT MEASURES THE LINK TOO, WHICH THIS SECTION FIRST SAID IT +COULD NOT.*** The claim was "the host has no libap and `hostkencc` +deliberately does not build the linkers, so this one is the VM's" — +and **neither is needed**. `6c -S` prints, per translation unit, every +symbol it defines (`TEXT`/`GLOBL`, with `<>` for a file-local one) and +every symbol it references; the program's undefined set is the +references minus the global definitions minus each unit's own statics, +over all of them at once. `php-mirror.sh --undefined` does that and +hands the result to `apfunc-sweep.py`. *A capability was written off +from the shape of the tool rather than from the question being asked* +— the same error as "kencc cannot be compiled on the build host", +which `cpp-xcheck` then disproved for a whole directory. + +### The link: four symbols, two causes + +`mk install` reached `6l` and named four undefined names. + +**`_volatile__`**, referenced by `__jit_debug_register_code`, is +kencc's: `__asm__ __volatile__("")` — the scanner knew `__asm__` and +not GNU's `[volatile] [inline] [goto]` qualifiers, so the qualifier +was lexed as an identifier and CALLED. 50 vendored files in this tree +use that spelling. Fixed in `cc/lex.c`, and *writing the two-line test +for it found that `__extension__` had been eating the first character +of whatever followed it* — both in `docs/notes/kencc.md`. + +**`lxb_dom_collection_create`, `_init` and `_destroy`**, referenced by +`lxb_dom_collection_make`, are a build-list question with a compiler +reason underneath. That function is a `static inline` in +`lexbor/dom/collection.h`, which `node.h` and `element.h` both +include, so **every one of the many units reaching those headers +emits it** — *kencc emits an unused static function and gcc at -O2 +discards it*. The three functions it calls live in +`lexbor/dom/collection.c`, and `ext/lexbor/config.m4:51-63` does not +list that file, because upstream never needs them. The mkfile +compiles it; it is lexbor's own code and self-consistent. **The +general form is worth keeping**: a header-only helper that calls into +a file the port does not compile is a link error here and silence on +gcc. ### The eight, and what each turned out to be @@ -777,10 +810,16 @@ cannot be found by reading the subject.* * fibers: **done** — `ucontext` is in libap and `ucontext-test` is 0 failures on the VM (`docs/notes/libap.md`). What remains is pointing php at it: `ZEND_FIBER_UCONTEXT` in the `config.h` -* **the LINK.** Every object compiles; nothing has been linked, and - the undefined-symbol class is untouched. `php-mirror.sh` cannot ask - — the host has no libap and `hostkencc` deliberately does not build - the linkers — so this one is the VM's. +* **the LINK: predicted to succeed, not yet run.** All 515 objects + compile and `--undefined` says the program wants **283 symbols from + outside php**, of which `apfunc-sweep` finds **273 defined in + libap** and ten that are present but invisible to it — `getcontext` + and `swapcontext` are assembly, `stdin`/`stdout`/`stderr`/`environ`/ + `h_errno`/`_ctype`/`_errnoloc` are DATA, and `signal` is spelled + `(*signal(int, void (*)(int)))(int)`, which no definition-shaped + regex matches. Each of the ten was opened. **So nothing is missing, + and the refutation condition is a name outside those 283** — which + is what a newly reached `static inline` would be. * the 474 unanswered questions in the template are still unanswered, and `cfghdr-sweep` gates on none of them. A compile is not evidence that an answer is right, only that it is consistent. diff --git a/sys/lib/tests/hostkencc/php-mirror.sh b/sys/lib/tests/hostkencc/php-mirror.sh index 4c4ab3c19..35ee981d4 100755 --- a/sys/lib/tests/hostkencc/php-mirror.sh +++ b/sys/lib/tests/hostkencc/php-mirror.sh @@ -1,12 +1,13 @@ #!/bin/sh # -# php-mirror.sh -- compile php's 513 translation units with the -# HOST-BUILT kencc, so a compile error costs a second instead of a VM -# round trip. +# php-mirror.sh -- compile php's translation units with the HOST-BUILT +# kencc, so a compile error -- or an undefined symbol -- costs a second +# instead of a VM round trip. # # sh php-mirror.sh --flags # print the compile flags and exit -# sh php-mirror.sh --compile # all 513; GATES on the error count +# sh php-mirror.sh --compile # all of them; GATES on the error count # sh php-mirror.sh --compile Zend/zend_lazy_objects.c # just these +# sh php-mirror.sh --undefined # every symbol the LINK will want # eval "$(sh php-mirror.sh --flags)"; $CC6 $CFL -S .../zend.c # # Depends on `hostkencc.sh' having run. `clisp-mirror.sh' is the @@ -37,7 +38,54 @@ # ABSOLUTE path (`"/sys/include/ape/stddef.h"'), which exists on the VM # and not here. # -# WHAT IT FOUND ON ITS FIRST RUN is in `docs/notes/php.md'. +# ***`--undefined' ASKS THE LINK'S QUESTION WITHOUT A LINKER, AND THIS +# FILE SAID IT COULD NOT.*** The first version's header ended "it +# cannot ask about the LINK, which stays the VM's", because the host +# has no libap and `hostkencc' deliberately does not build `[1-9]l'. +# **Neither is needed.** `6c -S' prints, for every translation unit, +# each symbol it DEFINES (`TEXT name+0(SB)', `GLOBL name+0(SB)' -- +# with `<>' when the definition is file-local) and each symbol it +# REFERENCES (anything else spelled `name+off(SB)'). The program's +# undefined set is then the references minus the global definitions +# minus that unit's own statics, computed over all of them at once -- +# which is the question `6l' answers, minus only the part that is +# libap's. +# +# For that last part it hands the list to `apfunc-sweep.py', which +# exists to say whether libap DEFINES a name, only DECLARES it, or has +# neither. **Read its answer rather than gating on it**, for two +# reasons the first run demonstrated: the sweep looks for FUNCTION +# definitions in `.c' files, so `getcontext' and `swapcontext' come +# back `declared' because they are ASSEMBLY (`arch/amd64/ucontext.s'), +# and `stdin', `stderr', `stdout', `environ', `h_errno', `_ctype' and +# `_errnoloc' come back `absent' because they are DATA. All ten are +# present; *a sweep over function names, asked about a variable, +# answers `absent' confidently* -- the caveat CLAUDE.md already +# carries, arriving here in its other form. `signal' is a third +# shape: defined, but spelled +# `(*signal(int sig, void (*func)(int)))(int)', which no +# definition-shaped regex matches. +# +# **ITS POSITIVE CONTROL IS A REAL FAILURE AND COSTS ONE COMMAND**: +# +# sh php-mirror.sh --undefined ext/lexbor/lexbor/dom/interfaces/node.c +# +# narrows the program to one unit, and the answer names +# `lxb_dom_collection_create', `_init' and `_destroy' among its 37 -- +# **the three the VM's `6l' named**, for the reason below. *A mode +# that reports an empty set is indistinguishable from one that cannot +# see*, which is why this is written down with its number. +# +# ***THOSE THREE ARE WHY `collection.c' IS IN THE MKFILE AND NOT IN +# UPSTREAM'S config.m4.*** `lxb_dom_collection_make' is a `static +# inline' in `lexbor/dom/collection.h', reached by every unit that +# includes `node.h' or `element.h'; **kencc emits an unused static +# function and gcc at -O2 discards it**, so upstream never needs the +# three functions it calls and this build does. That asymmetry is +# general: a header-only helper calling into a file the port does not +# compile is a link error here and silence there. +# +# WHAT IT FOUND is in `docs/notes/php.md'. set -e @@ -109,7 +157,7 @@ fi test -x "$HERE/out/bin/6c" || { echo "php-mirror: run hostkencc.sh first" >&2; exit 2; } -mkdir -p "$OUT/shadow" +mkdir -p "$OUT/shadow" "$OUT/asm" sed 's#"/sys/include/ape/#"#' \ "$ROOT/amd64/include/ape/stddef_arch.h" > "$OUT/shadow/stddef_arch.h" @@ -126,12 +174,14 @@ sources() { sed 's/$/.c/' } -if [ "${1:-}" != "--compile" ]; then +case "${1:-}" in +--compile) mode=compile; shift;; +--undefined) mode=undefined; shift;; +*) n=$(sources | wc -l) echo "php-mirror: staged $OUT; $n sources; now --compile" - exit 0 -fi -shift + exit 0;; +esac PATH="$HERE/out/bin:$PATH"; export PATH export objtype=amd64 @@ -172,6 +222,20 @@ for s in $list; do ext/hash/hash_sha3.c) extra=$KECCAKFLAGS;; ext/json/*) extra=-I$PHP/ext/json;; esac + if [ "$mode" = undefined ]; then + # `-S' rather than `-o': the assembly carries the symbol + # table this mode needs, and nothing else does without a + # reader for Plan 9 object files. + if $CC6 $extra $CFL -S "$src" \ + > "$OUT/asm/$(echo "$s" | tr / _).s" 2> "$OUT/t.log"; then + ok=$((ok+1)) + else + bad=$((bad+1)) + echo "==== $s" >> "$OUT/errors" + cat "$OUT/t.log" >> "$OUT/errors" + fi + continue + fi if $CC6 $extra $CFL -o "$OUT/t.6" "$src" > "$OUT/t.log" 2>&1; then ok=$((ok+1)) else @@ -180,6 +244,47 @@ for s in $list; do cat "$OUT/t.log" >> "$OUT/errors" fi done + +if [ "$mode" = undefined ]; then + echo "php-mirror: assembled $ok, errors $bad, missing $missing" + test "$bad" -eq 0 || { echo "php-mirror: fix the compile first" >&2; exit 1; } + python3 - "$OUT/asm" > "$OUT/undefined" <<'EOF' +import os, re, sys + +D = sys.argv[1] +# `TEXT name+0(SB)' and `GLOBL name+0(SB)' define; `name<>+0(SB)' +# defines it FILE-LOCALLY, which satisfies that unit and no other. +DEF = re.compile(r'^\t(?:TEXT|GLOBL)\t([A-Za-z_.][\w.]*)(<>)?\+') +REF = re.compile(r'(?name, "__extension__") == 0 || strcmp(s->name, "__asm__") == 0 || strcmp(s->name, "__asm") == 0)) { - int ac, depth; + int ac, depth, asmlike; + + asmlike = strcmp(s->name, "__asm__") == 0 + || strcmp(s->name, "__asm") == 0; /* * After reading the identifier above, peekc holds the char that * ended the identifier loop (e.g. the first '(' of '((...))'). @@ -1010,6 +1013,57 @@ yylex(void) } while(ac == ' ' || ac == '\t') ac = GETC(); + /* + * ***`__asm__ __volatile__("")' IS THE COMMON SPELLING AND + * NOTHING HERE KNEW THE QUALIFIER.*** GNU's grammar is + * `asm [volatile] [inline] [goto] ( ... )', and the loop + * below ran straight into `__volatile__', found no `(', and + * handed the identifier to the parser -- which called it: + * + * zend_gdb.c:59 function not declared: _volatile__ + * __jit_debug_register_code: undefined: _volatile__ + * + * at php's LINK, from + * `ZEND_API ... __jit_debug_register_code(void) + * { __asm__ __volatile__(""); }'. + * + * An identifier here is not valid C in any other reading -- + * the construct takes qualifiers and then a parenthesised + * argument -- so an unknown one is diagnosed rather than + * guessed at. + */ + if(asmlike && ac != '(' && (ac == '_' || isalpha(ac))) { + char qual[32]; + int qn; + + for(;;) { + qn = 0; + while(qn < (int)sizeof(qual)-1 && + (ac == '_' || isalnum(ac))) { + qual[qn++] = ac; + ac = GETC(); + } + qual[qn] = 0; + if(strcmp(qual, "volatile") != 0 + && strcmp(qual, "__volatile") != 0 + && strcmp(qual, "__volatile__") != 0 + && strcmp(qual, "inline") != 0 + && strcmp(qual, "__inline") != 0 + && strcmp(qual, "__inline__") != 0 + && strcmp(qual, "goto") != 0) { + yyerror("unexpected \"%s\" after %s", + qual, s->name); + break; + } + while(ac == ' ' || ac == '\t' || ac == '\n') { + if(ac == '\n') + lineno++; + ac = GETC(); + } + if(!(ac == '_' || isalpha(ac))) + break; + } + } if(ac == '(') { depth = 1; while(depth > 0) { @@ -1085,14 +1139,32 @@ yylex(void) if(ac == '(') depth++; else if(ac == ')') depth--; } - } else { - /* no argument list (e.g. bare __extension__); put char back */ - unget(ac); + /* Do NOT reset lasttok here: the swallowed keyword is + * invisible to the parser, so the previous type keyword + * (e.g. LDOUBLE before __asm__("sym") _Complex) must + * still be visible for _Complex combining. */ + goto l0; /* fetch next real token */ } - /* Do NOT reset lasttok here: the swallowed keyword is invisible - * to the parser, so the previous type keyword (e.g. LDOUBLE before - * __asm__("sym") _Complex) must still be visible for _Complex combining. */ - goto l0; /* fetch next real token */ + /* + * No argument list -- a bare `__extension__'. + * + * ***THIS USED TO BE `unget(ac); goto l0;' AND THAT LOSES THE + * CHARACTER.*** `unget' only sets `peekc', and `l0' is BELOW + * the `if(peekc != IGN)' test at the top of `yylex', so the + * next `GETC' reads straight past it. **`__extension__ int + * yy;' compiled as `nt yy;'** -- `6c -S' emitted + * `GLOBL nt+0(SB)' -- so `__extension__' has been eating the + * first character of whatever follows it for as long as it has + * been swallowed here. *Nothing in the tree had used it where + * the result was not also a syntax error*, which is why it + * surfaced only beside the `__volatile__' fault above. + * + * `l1' is the same entry point one step later: it lexes the + * character already in hand instead of fetching another, which + * is the idiom the `__func__' case below already uses. + */ + c = ac; + goto l1; } /* * C99 __func__ and GCC __FUNCTION__: predefined string literal