kencc: asm qualifiers and a lost character; php: the link's missing t…

CLAUDE.md

@@ -366,8 +366,17 @@ which reads like the mkfile's rule and is not it
 (`ext/hash/hash_sha3.c` takes them too, from `ext/hash`), and it
 passed `CFLAGS` through, whose leading **`-c` means "compile only" to
 `pcc` and "print `const if`" to `6c`** -- dozens of lines that read
-exactly like diagnostics. *It cannot ask about the LINK*, which stays
-the VM's.
+exactly like diagnostics.
+***AND `--undefined` ASKS THE LINK'S QUESTION WITHOUT A LINKER, WHICH
+THIS PARAGRAPH FIRST SAID IT COULD NOT***: `6c -S` names every symbol
+each unit DEFINES (`<>` marks a file-local one) and REFERENCES, so
+references minus global definitions minus each unit's own statics,
+over all 515, is what `6l` would complain about -- handed to
+`apfunc-sweep.py` for "does libap have it". **READ, do not gate**: the
+sweep sees C function definitions only, so assembly reads as
+`declared` and DATA as `absent` (ten of php's 283 are one or the
+other, all present). *Its positive control is one command and names
+the three symbols the VM's `6l` actually named.*
 ***AND `hostkencc/` BUILDS ALL TEN C COMPILERS WITH gcc***, from the
 shipped `sys/src/cmd/cc` and `[1-9kqv]c` plus go1.4's `lib9`/`libbio`,
 already vendored at `sys/src/external/go1.4` — which **compiled with gcc
@@ -1195,15 +1204,22 @@ silent:
 
 **php — ***ALL 514 TRANSLATION UNITS COMPILE***, measured on the HOST,
 and the notes are in `docs/notes/php.md`.** 8.5.12, vendored at
-`sys/src/external/php`; `sh sys/lib/tests/hostkencc/php-mirror.sh
---compile` answers `compiled 514, errors 0, missing 0`. **The LINK is
-untouched and is the VM's**: the host has no libap and `hostkencc`
-deliberately does not build the linkers, so the undefined-symbol class
-— a `config.h` claiming a name libap only DECLARES — has not been
-asked. *That question was checked by hand for the resolver family and
-it held*: `res_query.c`'s `weak_alias(res_query, res_search)` is a
-no-op on kencc, and `network/res_aliases.c` exists precisely to define
-the strong forwarders.
+`sys/src/external/php`; `--compile` answers `compiled 515, errors 0,
+missing 0` and **`--undefined` says the program wants 283 symbols
+from outside php, every one of which libap has** (273 the sweep
+names outright; ten it misreports because they are assembly or DATA,
+each opened by hand). **So the link is PREDICTED to succeed and has
+not been run**, and the refutation condition is a name outside those
+283 — which is what a newly reached `static inline` would be.
+***`mk install` DID reach `6l` and named four***: `_volatile__`, from
+`__asm__ __volatile__("")`, which kencc lexed as a function call
+because it knew `__asm__` and not GNU's qualifiers (50 vendored files
+use that spelling, and the two-line test for it found
+`__extension__` eating the next character); and three lexbor
+`lxb_dom_collection_*`, because **kencc emits an unused `static
+inline` where gcc at -O2 discards it**, so a header-only helper
+calling into a file upstream's `config.m4` does not list is a link
+error here and silence there. Both fixed.
 **The first sweep returned EIGHT errors in six files and all six were
 fixed without a VM round trip** — three of them kencc's (a pointer
 assigned to a `bool`; `bool visible;` declaring nothing, because

docs/notes/instruments.md

@@ -1190,9 +1190,34 @@ like diagnostics. *A flag one letter from another flag is checked
 rather than recalled, and these two meanings are in two different
 programs.*
 
-**What it does not measure is the LINK** — the host has no libap and
-`hostkencc` deliberately does not build the linkers — so the
-undefined-symbol class stays the VM's.
+***AND `--undefined` ASKS THE LINK'S QUESTION WITHOUT A LINKER, WHICH
+THIS ENTRY FIRST SAID WAS IMPOSSIBLE.*** The original wording was
+"the host has no libap and `hostkencc` deliberately does not build
+the linkers — so the undefined-symbol class stays the VM's", and
+neither fact is the obstacle it looks like. `6c -S` prints every
+symbol a unit DEFINES (`TEXT`/`GLOBL`, `<>` marking a file-local one)
+and every symbol it REFERENCES; across all 515 units, references
+minus global definitions minus each unit's own statics is exactly
+what `6l` would complain about. The result is handed to
+`apfunc-sweep.py`, which says whether libap has each name.
+
+**Read that answer, do not gate on it.** The sweep looks for C
+function definitions, so ten of the 283 come back wrong in two
+predictable ways — `getcontext` and `swapcontext` are ASSEMBLY and
+read as `declared`; `stdin`, `stdout`, `stderr`, `environ`,
+`h_errno`, `_ctype` and `_errnoloc` are DATA and read as `absent`;
+and `signal`, spelled `(*signal(int, void (*)(int)))(int)`, matches no
+definition-shaped regex. All ten are present. *A sweep over function
+names, asked about a variable, answers `absent` confidently* — the
+caveat CLAUDE.md already carried for `HAVE_TM_ZONE`, arriving in its
+other form.
+
+**Its positive control is a real failure and costs one command**:
+`--undefined ext/lexbor/lexbor/dom/interfaces/node.c` narrows the
+program to one unit and names `lxb_dom_collection_create`, `_init`
+and `_destroy` among its 37 — *the three the VM's `6l` named*. A mode
+that reports an empty set is indistinguishable from one that cannot
+see, which is why the number is written down.
 
 ## `boolptr-test` — a pointer converted to `bool`
 

docs/notes/kencc.md

@@ -3289,3 +3289,66 @@ one fault that cannot be found by reading the subject* — and this one
 also explains why `exits()` losing a buffered `diagbuf` was the first
 suspicion and the wrong one: go1.4's `libbio` registers an `atexit`
 flush, so a diagnostic would have appeared.
+
+## The link found two more in the same lexer block
+
+php's objects all compiled, and `6l` then named four undefined
+symbols. Three were a build-list question (`docs/notes/php.md`); the
+fourth was this compiler, and reproducing it exposed a second fault
+beside it that had been there far longer.
+
+### `__asm__ __volatile__("")` called a function named `_volatile__`
+
+`Zend/zend_gdb.c:57` is
+
+	ZEND_API zend_never_inline void __jit_debug_register_code(void)
+	{
+		__asm__ __volatile__("");
+	}
+
+and the link answered
+
+	__jit_debug_register_code: undefined: _volatile__ in ...
+
+`lex.c` makes `__asm__` invisible by consuming from the opening `(` to
+the matching `)`. **GNU's grammar is `asm [volatile] [inline] [goto] (
+... )`**, and nothing here knew about the qualifiers: the scanner
+looked for `(`, found `_`, and handed the identifier to the parser,
+which called it. *A function nobody wrote, in an object file, found at
+the link* — the compile said only `function not declared`, which in a
+C89-tolerant compiler is a warning-shaped line in a sea of them.
+
+The qualifiers are skipped now — `volatile`, `__volatile`,
+`__volatile__`, `inline`, `__inline`, `__inline__`, `goto` — and an
+identifier that is none of them is **diagnosed**, because the
+construct admits nothing else there. **50 vendored files use the
+spelling**, 15 of them php's and the rest spread over pcre2, m4,
+libpng, perl, libressl, gnulib, diffutils, coreutils, zlib and zstd,
+so this was not a php question.
+
+### And `__extension__` had been eating the next character
+
+The `__asm__` fix needed a one-line test, and the test next to it did
+not compile either:
+
+	__extension__ int yy;
+
+became **`nt yy;`** — `6c -S` emitted `GLOBL nt+0(SB)`. The no-argument
+branch of the same block ended `unget(ac); goto l0;`, and that is
+wrong in a way that reads as right: `unget` only sets `peekc`, while
+`l0` is BELOW the `if(peekc != IGN)` test at the top of `yylex`, so
+the next `GETC` reads straight past the character. It is `c = ac;
+goto l1;` now — lex the character already in hand — which is the idiom
+the `__func__` case twenty lines down was already using.
+
+**Nothing in the tree had noticed**, and the reason is worth keeping:
+`__extension__` is almost always followed by a token whose first
+character is not load-bearing to the diagnosis, so the failure reads
+as a syntax error in the code rather than as a lost character. *The
+bug was found by writing a two-line test for a different bug*, which
+is the cheapest way it could have been found and was not deliberate.
+
+Both fixes are in the block the string-literal fix of the previous
+round also touched, and the controls are the same three: `agree.c`
+identical, `--probe` ten `regpair-min` and eight `regpair-probe` with
+`kc` and `vc` at `:242`, clisp's 31 TUs clean.

docs/notes/php.md

@@ -707,11 +707,44 @@ unchanged: **sound for "does it compile" and "what does it diagnose",
 unsound for anything keyed on `long` being 32 bits**, since the host
 build is LP64.
 
-**What it does NOT measure is the LINK**, which is the next class of
-failure and is the VM's: a `config.h` claiming a name libap only
-DECLARES fails there, far from the claim. The resolver family below is
-exactly that shape caught early — `apfunc-sweep`'s middle state — and
-it held, but it had to be checked rather than assumed.
+***AND IT MEASURES THE LINK TOO, WHICH THIS SECTION FIRST SAID IT
+COULD NOT.*** The claim was "the host has no libap and `hostkencc`
+deliberately does not build the linkers, so this one is the VM's" —
+and **neither is needed**. `6c -S` prints, per translation unit, every
+symbol it defines (`TEXT`/`GLOBL`, with `<>` for a file-local one) and
+every symbol it references; the program's undefined set is the
+references minus the global definitions minus each unit's own statics,
+over all of them at once. `php-mirror.sh --undefined` does that and
+hands the result to `apfunc-sweep.py`. *A capability was written off
+from the shape of the tool rather than from the question being asked*
+— the same error as "kencc cannot be compiled on the build host",
+which `cpp-xcheck` then disproved for a whole directory.
+
+### The link: four symbols, two causes
+
+`mk install` reached `6l` and named four undefined names.
+
+**`_volatile__`**, referenced by `__jit_debug_register_code`, is
+kencc's: `__asm__ __volatile__("")` — the scanner knew `__asm__` and
+not GNU's `[volatile] [inline] [goto]` qualifiers, so the qualifier
+was lexed as an identifier and CALLED. 50 vendored files in this tree
+use that spelling. Fixed in `cc/lex.c`, and *writing the two-line test
+for it found that `__extension__` had been eating the first character
+of whatever followed it* — both in `docs/notes/kencc.md`.
+
+**`lxb_dom_collection_create`, `_init` and `_destroy`**, referenced by
+`lxb_dom_collection_make`, are a build-list question with a compiler
+reason underneath. That function is a `static inline` in
+`lexbor/dom/collection.h`, which `node.h` and `element.h` both
+include, so **every one of the many units reaching those headers
+emits it** — *kencc emits an unused static function and gcc at -O2
+discards it*. The three functions it calls live in
+`lexbor/dom/collection.c`, and `ext/lexbor/config.m4:51-63` does not
+list that file, because upstream never needs them. The mkfile
+compiles it; it is lexbor's own code and self-consistent. **The
+general form is worth keeping**: a header-only helper that calls into
+a file the port does not compile is a link error here and silence on
+gcc.
 
 ### The eight, and what each turned out to be
 
@@ -777,10 +810,16 @@ cannot be found by reading the subject.*
 * fibers: **done** — `ucontext` is in libap and `ucontext-test` is 0
   failures on the VM (`docs/notes/libap.md`). What remains is pointing
   php at it: `ZEND_FIBER_UCONTEXT` in the `config.h`
-* **the LINK.** Every object compiles; nothing has been linked, and
-  the undefined-symbol class is untouched. `php-mirror.sh` cannot ask
-  — the host has no libap and `hostkencc` deliberately does not build
-  the linkers — so this one is the VM's.
+* **the LINK: predicted to succeed, not yet run.** All 515 objects
+  compile and `--undefined` says the program wants **283 symbols from
+  outside php**, of which `apfunc-sweep` finds **273 defined in
+  libap** and ten that are present but invisible to it — `getcontext`
+  and `swapcontext` are assembly, `stdin`/`stdout`/`stderr`/`environ`/
+  `h_errno`/`_ctype`/`_errnoloc` are DATA, and `signal` is spelled
+  `(*signal(int, void (*)(int)))(int)`, which no definition-shaped
+  regex matches. Each of the ten was opened. **So nothing is missing,
+  and the refutation condition is a name outside those 283** — which
+  is what a newly reached `static inline` would be.
 * the 474 unanswered questions in the template are still unanswered,
   and `cfghdr-sweep` gates on none of them. A compile is not evidence
   that an answer is right, only that it is consistent.

sys/lib/tests/hostkencc/php-mirror.sh

@@ -1,12 +1,13 @@
 #!/bin/sh
 #
-# php-mirror.sh -- compile php's 513 translation units with the
-# HOST-BUILT kencc, so a compile error costs a second instead of a VM
-# round trip.
+# php-mirror.sh -- compile php's translation units with the HOST-BUILT
+# kencc, so a compile error -- or an undefined symbol -- costs a second
+# instead of a VM round trip.
 #
 #	sh php-mirror.sh --flags      # print the compile flags and exit
-#	sh php-mirror.sh --compile    # all 513; GATES on the error count
+#	sh php-mirror.sh --compile    # all of them; GATES on the error count
 #	sh php-mirror.sh --compile Zend/zend_lazy_objects.c   # just these
+#	sh php-mirror.sh --undefined  # every symbol the LINK will want
 #	eval "$(sh php-mirror.sh --flags)"; $CC6 $CFL -S .../zend.c
 #
 # Depends on `hostkencc.sh' having run.  `clisp-mirror.sh' is the
@@ -37,7 +38,54 @@
 # ABSOLUTE path (`"/sys/include/ape/stddef.h"'), which exists on the VM
 # and not here.
 #
-# WHAT IT FOUND ON ITS FIRST RUN is in `docs/notes/php.md'.
+# ***`--undefined' ASKS THE LINK'S QUESTION WITHOUT A LINKER, AND THIS
+# FILE SAID IT COULD NOT.***  The first version's header ended "it
+# cannot ask about the LINK, which stays the VM's", because the host
+# has no libap and `hostkencc' deliberately does not build `[1-9]l'.
+# **Neither is needed.**  `6c -S' prints, for every translation unit,
+# each symbol it DEFINES (`TEXT name+0(SB)', `GLOBL name+0(SB)' --
+# with `<>' when the definition is file-local) and each symbol it
+# REFERENCES (anything else spelled `name+off(SB)').  The program's
+# undefined set is then the references minus the global definitions
+# minus that unit's own statics, computed over all of them at once --
+# which is the question `6l' answers, minus only the part that is
+# libap's.
+#
+# For that last part it hands the list to `apfunc-sweep.py', which
+# exists to say whether libap DEFINES a name, only DECLARES it, or has
+# neither.  **Read its answer rather than gating on it**, for two
+# reasons the first run demonstrated: the sweep looks for FUNCTION
+# definitions in `.c' files, so `getcontext' and `swapcontext' come
+# back `declared' because they are ASSEMBLY (`arch/amd64/ucontext.s'),
+# and `stdin', `stderr', `stdout', `environ', `h_errno', `_ctype' and
+# `_errnoloc' come back `absent' because they are DATA.  All ten are
+# present; *a sweep over function names, asked about a variable,
+# answers `absent' confidently* -- the caveat CLAUDE.md already
+# carries, arriving here in its other form.  `signal' is a third
+# shape: defined, but spelled
+# `(*signal(int sig, void (*func)(int)))(int)', which no
+# definition-shaped regex matches.
+#
+# **ITS POSITIVE CONTROL IS A REAL FAILURE AND COSTS ONE COMMAND**:
+#
+#	sh php-mirror.sh --undefined ext/lexbor/lexbor/dom/interfaces/node.c
+#
+# narrows the program to one unit, and the answer names
+# `lxb_dom_collection_create', `_init' and `_destroy' among its 37 --
+# **the three the VM's `6l' named**, for the reason below.  *A mode
+# that reports an empty set is indistinguishable from one that cannot
+# see*, which is why this is written down with its number.
+#
+# ***THOSE THREE ARE WHY `collection.c' IS IN THE MKFILE AND NOT IN
+# UPSTREAM'S config.m4.***  `lxb_dom_collection_make' is a `static
+# inline' in `lexbor/dom/collection.h', reached by every unit that
+# includes `node.h' or `element.h'; **kencc emits an unused static
+# function and gcc at -O2 discards it**, so upstream never needs the
+# three functions it calls and this build does.  That asymmetry is
+# general: a header-only helper calling into a file the port does not
+# compile is a link error here and silence there.
+#
+# WHAT IT FOUND is in `docs/notes/php.md'.
 
 set -e
 
@@ -109,7 +157,7 @@ fi
 test -x "$HERE/out/bin/6c" || {
 	echo "php-mirror: run hostkencc.sh first" >&2; exit 2; }
 
-mkdir -p "$OUT/shadow"
+mkdir -p "$OUT/shadow" "$OUT/asm"
 sed 's#"/sys/include/ape/#"#' \
 	"$ROOT/amd64/include/ape/stddef_arch.h" > "$OUT/shadow/stddef_arch.h"
 
@@ -126,12 +174,14 @@ sources() {
 	sed 's/$/.c/'
 }
 
-if [ "${1:-}" != "--compile" ]; then
+case "${1:-}" in
+--compile)	mode=compile; shift;;
+--undefined)	mode=undefined; shift;;
+*)
 	n=$(sources | wc -l)
 	echo "php-mirror: staged $OUT; $n sources; now --compile"
-	exit 0
-fi
-shift
+	exit 0;;
+esac
 
 PATH="$HERE/out/bin:$PATH"; export PATH
 export objtype=amd64
@@ -172,6 +222,20 @@ for s in $list; do
 	ext/hash/hash_sha3.c)		extra=$KECCAKFLAGS;;
 	ext/json/*)			extra=-I$PHP/ext/json;;
 	esac
+	if [ "$mode" = undefined ]; then
+		# `-S' rather than `-o': the assembly carries the symbol
+		# table this mode needs, and nothing else does without a
+		# reader for Plan 9 object files.
+		if $CC6 $extra $CFL -S "$src" \
+		    > "$OUT/asm/$(echo "$s" | tr / _).s" 2> "$OUT/t.log"; then
+			ok=$((ok+1))
+		else
+			bad=$((bad+1))
+			echo "==== $s" >> "$OUT/errors"
+			cat "$OUT/t.log" >> "$OUT/errors"
+		fi
+		continue
+	fi
 	if $CC6 $extra $CFL -o "$OUT/t.6" "$src" > "$OUT/t.log" 2>&1; then
 		ok=$((ok+1))
 	else
@@ -180,6 +244,47 @@ for s in $list; do
 		cat "$OUT/t.log" >> "$OUT/errors"
 	fi
 done
+
+if [ "$mode" = undefined ]; then
+	echo "php-mirror: assembled $ok, errors $bad, missing $missing"
+	test "$bad" -eq 0 || { echo "php-mirror: fix the compile first" >&2; exit 1; }
+	python3 - "$OUT/asm" > "$OUT/undefined" <<'EOF'
+import os, re, sys
+
+D = sys.argv[1]
+# `TEXT name+0(SB)' and `GLOBL name+0(SB)' define; `name<>+0(SB)'
+# defines it FILE-LOCALLY, which satisfies that unit and no other.
+DEF = re.compile(r'^\t(?:TEXT|GLOBL)\t([A-Za-z_.][\w.]*)(<>)?\+')
+REF = re.compile(r'(?<![\w.<])([A-Za-z_.][\w.]*)\+-?\d+\(SB\)')
+
+gdef, statics, refs = set(), {}, {}
+for f in sorted(os.listdir(D)):
+    st, rf = set(), set()
+    for line in open(os.path.join(D, f), errors='ignore'):
+        m = DEF.match(line)
+        if m:
+            (st if m.group(2) else gdef).add(m.group(1))
+            continue
+        rf.update(REF.findall(line))
+    statics[f], refs[f] = st, rf
+
+und = set()
+for f, rf in refs.items():
+    und |= rf - gdef - statics[f]
+for n in sorted(und):
+    print(n)
+EOF
+	rm -rf "$OUT/asm"
+	n=$(wc -l < "$OUT/undefined")
+	echo "php-mirror: $n symbol(s) the link must find outside php"
+	echo "php-mirror: the list is in $OUT/undefined; asking libap --"
+	python3 "$ROOT/sys/lib/tests/apfunc-sweep.py" --file "$OUT/undefined" |
+		grep -v '	*defined   ' || true
+	echo "php-mirror: ^ anything here is READ, not a failure -- the"
+	echo "php-mirror:   sweep sees C function definitions only, so"
+	echo "php-mirror:   assembly reads as 'declared' and DATA as 'absent'."
+	exit 0
+fi
 echo "php-mirror: compiled $ok, errors $bad, missing $missing"
 echo "php-mirror: diagnostics in $OUT/errors"
 test "$bad" -eq 0 && test "$missing" -eq 0