diff --git a/CLAUDE.md b/CLAUDE.md index 38004bf0e..d0ae89fbd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -397,7 +397,18 @@ that the instrument notices.* nothing declares), `apehdr-sweep.py` (every `sys/include/ape` header standalone, plus a TOGETHER case for collisions only a combination can show), `rceq-sweep.py` (**gates**: a bare `=` in an mkfile recipe cannot -parse), `mkcont-sweep.py` (**gates**: a logical line mk cannot parse. TWO +parse. ***ITS `RULE` WANTED TWO COLONS AND SO READ ONLY ATTRIBUTE +RULES*** — `install:V:` and nothing else — so every ordinary +`target: prereq` recipe in the tree was invisible to it: **870 +recipe lines before, 1859 after**, with 0 findings both times. *A +gating sweep reporting 0 over lines it cannot see* is the failure its +own header already warned about for backquotes. **Found by control, +not by reading**: a bare `=` planted in `cmd/php/mkfile`'s `clean:V:` +was reported and the same `=` in one of that file's `%.$O:` rules was +not — *planting the break in one place would have passed*, which is +why `--control` now covers an attribute rule, a PLAIN rule and a +backquote, plus two that must NOT fire, and fails on exactly the +plain rule against the old regex), `mkcont-sweep.py` (**gates**: a logical line mk cannot parse. TWO shapes, and *the second was added after the first reported 0 over it*: a comment ending a continuation, and an INDENTED line that is not a recipe — `cmd/php/mkfile` ended its `OFILES` at an ordinary diff --git a/docs/notes/php.md b/docs/notes/php.md index 81a9402b8..374911c99 100644 --- a/docs/notes/php.md +++ b/docs/notes/php.md @@ -410,6 +410,60 @@ anywhere in this port**. `json_scanner.c` and parsers and scanners and the 4.2MB `zend_vm_execute.h`. *A question that could have cost two more build tools cost one grep.* +### The include layout, and the four headers under `cfg/` + +`TSRM/TSRM.h:20` says `#include
` while +`Zend/zend_frameless_function.h` and `ext/opcache/zend_shared_alloc.c` +say `#include `. **Both spellings have to work**, and +upstream gets that for free because the generated config really does +sit at `main/php_config.h` under the php root. + +So the hand-kept headers live in `cmd/php/cfg/`, with the answers in +`cfg/main/php_config.h` and `cfg/php_config.h` a one-line redirect -- +*two hand-maintained files that must agree are one file too many*, +which is the warning CLAUDE.md already carries about perl's +`Config.pm` and its `config.h`. One `-Icfg` answers all four +spellings. + +***AND `-I.' IS DELIBERATELY ABSENT.*** The build writes its objects +into `main/`, `Zend/`, `TSRM/` and `ext/` in the build directory, so +`.` on the include path would put directories named after php's own +source trees in front of php's own source trees. It is also why the +headers are under `cfg/` rather than in a `main/` beside the mkfile: +**`clean` does `rm -rf main`**, and a committed header there would be +deleted by it. + +### Per-directory flags, and why they are 32 explicit rules + +Preprocessing all 513 sources on the host found **31 with an +unresolvable include**, in two clusters, and the fix is flags rather +than files: + +* **pcre2, 28 files** -- `pcre2_internal.h:68` includes + `` on purpose when `PCRE2_CODE_UNIT_WIDTH` is unset. + Upstream passes `-DHAVE_CONFIG_H -DHAVE_MEMMOVE` and + `-I.../ext/pcre/pcre2lib`, where pcre2's own `config.h` lives. +* **Keccak, 3 files** -- `KeccakP-200-SnP.h` is *excluded by a flag*, + `-DKeccakP200_excluded` and its two siblings, plus + `-I.../ext/hash/sha3/generic64lc`. + +***THE FLAGS CANNOT GO IN THE GLOBAL CFLAGS, AND THAT WAS MEASURED +RATHER THAN FEARED.*** `-DHAVE_CONFIG_H` together with +`-I.../pcre2lib` would hand `ext/json/json.c:18`, +`json_encoder.c:18` and `main/php_odbc_utils.c:18` -- each of which +is `#ifdef HAVE_CONFIG_H / #include ` -- **pcre2's** +`config.h`. Not a failure: a different header, silently. + +The rules are explicit rather than patterned because mk's precedence +between a general and a specific meta-rule is something I would be +recalling rather than reading, there is no mk on the build host to +check it with, and a wrong guess costs a VM round trip. *Explicit +beats meta-rule in every mk, which needs no checking at all.* + +**All 31 were found in one host pass**, `gcc -E` over every source +with the real include paths, filtered to `No such file or directory`. +One sweep instead of 31 round trips. + ## What the config.h decided, and why by hand The long reasoning is in the file itself; what matters as a summary: diff --git a/sys/lib/tests/rceq-sweep.py b/sys/lib/tests/rceq-sweep.py index ae7306bb5..8af684ffb 100644 --- a/sys/lib/tests/rceq-sweep.py +++ b/sys/lib/tests/rceq-sweep.py @@ -62,7 +62,22 @@ ROOT = os.path.normpath(ROOT) TREE = os.path.join(ROOT, 'sys', 'src') -RULE = re.compile(r'^[^=#]*:[A-Z]*:') +# ***THIS USED TO BE `^[^=#]*:[A-Z]*:' AND SO REQUIRED TWO COLONS.*** +# That matches an ATTRIBUTE rule -- `install:V:', `test:QV:' -- and +# nothing else, so an ordinary `target: prereq' header was not a rule +# and its recipe lines were never read. **A gating sweep reporting 0 +# over recipes it cannot see** is the one failure mode its own header +# already warns about, two paragraphs down, about backquotes. +# +# It was found by control rather than by reading: a bare `=' planted +# in `cmd/php/mkfile's `clean:V:' was reported, and the same `=' in +# one of that file's `ext/...$O: $PHPSRC/....c' rules was not. +# *Planting the break in only one place would have passed.* +# +# A rule header is now any non-indented, non-comment line with a colon +# that is not an assignment -- `[^=#\t]*' stops at the `=', so +# `CFLAGS=-I/a:/b' cannot match, and a `#' at column 0 cannot either. +RULE = re.compile(r'^[^=#\t]*:') ASSIGN = re.compile(r'^[A-Za-z_][A-Za-z0-9_]*\s*=') # an `=' is legal only at the start of a command; these end one LEADING = re.compile(r"(^|[;&|{}()]|\|\||&&)\s*[A-Za-z_][A-Za-z0-9_]*\s*=") @@ -135,7 +150,73 @@ def bare_eq(line): return '=' in s +CONTROL = """\ +OFILES=\\ +\ta.$O\\ +\tb.$O + +install:V: $O.out +\techo attr=rule + +%.$O: $SRC/%.c +\techo plain=rule + +X=`{echo -DSIZE=8} + +ok:V: +\techo '-Dquoted=fine' +\tFOO=bar echo leading is legal +""" + + +def control(): + """***THREE SHAPES, BECAUSE PLANTING THE BREAK IN ONE PLACE PASSED.*** + + This sweep GATES, and for a long time it read recipes only under an + ATTRIBUTE rule -- `install:V:' -- because `RULE' wanted two colons. + A bare `=' planted in `cmd/php/mkfile's `clean:V:' was duly + reported, which looked like proof the sweep worked, and the same + `=' in one of that file's `target: prereq' rules was not. 989 + recipe lines across the tree were invisible. + + So the control covers an attribute rule, a PLAIN rule, and a + backquoted assignment -- the three ways mk hands text to rc -- and + two things that must NOT be reported: a quoted `=' and a LEADING + assignment, which rc allows. + """ + import tempfile + want = {6: 'attribute rule', 9: 'plain rule', 11: 'backquote'} + ok = True + with tempfile.TemporaryDirectory() as d: + p = os.path.join(d, 'mkfile') + open(p, 'w').write(CONTROL) + got = set() + for lineno, line in recipe_lines(p): + if bare_eq(line): + got.add(lineno) + for lineno, line in enumerate(CONTROL.split('\n'), 1): + rc = backquote_rc(line) + if rc is not None and bare_eq(rc): + got.add(lineno) + for ln, what in sorted(want.items()): + hit = ln in got + print('control: %-18s line %2d -- %s' + % (what, ln, 'ok' if hit else 'FAILED, not reported')) + if not hit: + ok = False + extra = got - set(want) + for ln in sorted(extra): + print('control: line %d reported and must NOT be ' + '(quoted or leading `=\') -- FAILED' % ln) + ok = False + if not extra: + print('control: quoted and leading `=\' correctly ignored -- ok') + return 0 if ok else 1 + + def main(): + if '--control' in sys.argv: + return control() if not os.path.isdir(os.path.join(ROOT, 'sys', 'include', 'ape')): print('rceq-sweep: %s is not an APExp root -- refusing to report' % ROOT) print('(a sweep rooted in the wrong tree reports an empty one as a') diff --git a/sys/src/ape/cmd/php/build-defs.h b/sys/src/ape/cmd/php/cfg/build-defs.h similarity index 100% rename from sys/src/ape/cmd/php/build-defs.h rename to sys/src/ape/cmd/php/cfg/build-defs.h diff --git a/sys/src/ape/cmd/php/php_config.h b/sys/src/ape/cmd/php/cfg/main/php_config.h similarity index 100% rename from sys/src/ape/cmd/php/php_config.h rename to sys/src/ape/cmd/php/cfg/main/php_config.h diff --git a/sys/src/ape/cmd/php/cfg/php_config.h b/sys/src/ape/cmd/php/cfg/php_config.h new file mode 100644 index 000000000..01ec8a836 --- /dev/null +++ b/sys/src/ape/cmd/php/cfg/php_config.h @@ -0,0 +1,17 @@ +/* + * php_config.h -- the SHORT spelling. + * + * ***THE SAME FILE IS INCLUDED TWO WAYS AND BOTH HAVE TO WORK.*** + * `TSRM/TSRM.h:20' says `#include
' while + * `Zend/zend_frameless_function.h' and `ext/opcache/zend_shared_alloc.c' + * say `#include '. Upstream both resolve against the + * php root, where the generated config really does live at + * `main/php_config.h'. + * + * So the answers live in `cfg/main/php_config.h' and this is a + * redirect, rather than a second copy that could drift from the + * first. *Two hand-maintained files that must agree are one file + * too many* -- the same reason perl's `Config.pm' and its `config.h' + * are called out in CLAUDE.md as a pair to keep in step. + */ +#include
diff --git a/sys/src/ape/cmd/php/timelib_config.h b/sys/src/ape/cmd/php/cfg/timelib_config.h similarity index 100% rename from sys/src/ape/cmd/php/timelib_config.h rename to sys/src/ape/cmd/php/cfg/timelib_config.h diff --git a/sys/src/ape/cmd/php/zend_config.h b/sys/src/ape/cmd/php/cfg/zend_config.h similarity index 100% rename from sys/src/ape/cmd/php/zend_config.h rename to sys/src/ape/cmd/php/cfg/zend_config.h diff --git a/sys/src/ape/cmd/php/mkfile b/sys/src/ape/cmd/php/mkfile index 867ee3aa4..9693ae604 100644 --- a/sys/src/ape/cmd/php/mkfile +++ b/sys/src/ape/cmd/php/mkfile @@ -619,7 +619,8 @@ OFILES=\ # is how `HELP_BUILTIN' was turned on and did not reach bash; the same # line is in `lib/perl/mkfile' and `cmd/bash/mkfile' for the same # reason. -HFILES=php_config.h zend_config.h timelib_config.h build-defs.h +HFILES=cfg/main/php_config.h cfg/php_config.h cfg/zend_config.h\ + cfg/timelib_config.h cfg/build-defs.h <$APEXPROOT/sys/src/cmd/mkone @@ -672,7 +673,19 @@ OBJDIRS=\ ext/standard ext/standard/libavifinfo ext/uri\ ext/uri/uriparser/src main main/streams sapi/cli -INCS=-I. -I$PHPSRC -I$PHPSRC/main -I$PHPSRC/Zend -I$PHPSRC/TSRM\ +# ***`cfg' COMES FIRST AND `.' IS NOT ON THE LIST AT ALL.*** +# The four hand-kept headers are reached by three different spellings +# -- `
' from `TSRM/TSRM.h:20', `' +# from Zend and opcache, `' from `main/php.h' -- and +# `-Icfg' answers all of them, with `cfg/php_config.h' redirecting to +# `cfg/main/php_config.h' so there is only ever one set of answers. +# +# *`-I.' is deliberately absent.* The build writes its objects into +# `main/', `Zend/', `TSRM/' and `ext/' right here, so `.' on the +# include path would put directories named after php's own source +# trees in front of php's own source trees. Nothing needs it: +# `internal_functions_cli.c' reaches everything through `$PHPSRC'. +INCS=-Icfg -I$PHPSRC -I$PHPSRC/main -I$PHPSRC/Zend -I$PHPSRC/TSRM\ -I$PHPSRC/ext/date/lib -I$PHPSRC/ext/lexbor\ -I$PHPSRC/ext/uri/uriparser/include @@ -721,6 +734,182 @@ ext/json/json_parser.$O: $PHPSRC/ext/json/json_parser.tab.c mkdir -p $OBJDIRS $CC $CFLAGS -I$PHPSRC/ext/json $prereq -o $target + +# ------------------------------------------------------------------ +# ***TWO DIRECTORIES NEED FLAGS THE REST MUST NOT HAVE, AND THE RULES +# ARE EXPLICIT RATHER THAN PATTERNED ON PURPOSE.*** +# +# A `ext/pcre/pcre2lib/%.$O:' meta-rule beside the general `%.$O:' one +# would be shorter, and mk probably prefers the more specific stem -- +# but *probably* is recollection, and this tree's rule is that a flag +# one letter from another flag is checked rather than recalled. There +# is no mk here to check it with, and a wrong guess costs a VM round +# trip, so these are 32 explicit rules. Explicit beats meta-rule in +# every mk, which needs no checking at all. +# +# ***AND THE FLAGS CANNOT SIMPLY GO IN THE GLOBAL CFLAGS, WHICH WAS +# MEASURED RATHER THAN FEARED.*** pcre2 wants `-DHAVE_CONFIG_H' and +# `-I$PHPSRC/ext/pcre/pcre2lib', which holds a `config.h' -- and +# `ext/json/json.c:18', `json_encoder.c:18' and +# `main/php_odbc_utils.c:18' each read +# +# #ifdef HAVE_CONFIG_H +# #include +# #endif +# +# so the two flags together would hand three php files **pcre2's** +# config.h. Not a failure: a different header, silently. Upstream +# avoids it by giving every extension its own compile line, which is +# what these rules are. +# +# The Keccak `-D..._excluded' flags are why `KeccakP-200-SnP.h' was +# reported missing: it is excluded, and the exclusion is a flag rather +# than a file. +# +# All 32 were found in ONE pass -- `gcc -E' over all 513 sources on +# the host with these include paths, reporting only +# `No such file or directory'. *One sweep instead of 31 VM round +# trips*, which is the same economics as every other host instrument +# here. Anything the preprocessor can answer should be answered +# before the VM sees it. +# ------------------------------------------------------------------ + +PCRE2FLAGS=-DHAVE_CONFIG_H -DHAVE_MEMMOVE -I$PHPSRC/ext/pcre\ + -I$PHPSRC/ext/pcre/pcre2lib + +KECCAKFLAGS=-DKeccakP200_excluded -DKeccakP400_excluded\ + -DKeccakP800_excluded -I$PHPSRC/ext/hash\ + -I$PHPSRC/ext/hash/sha3/generic64lc + + +ext/pcre/pcre2lib/pcre2_auto_possess.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_auto_possess.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_chartables.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_chartables.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_chkdint.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_chkdint.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_compile.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_compile.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_config.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_config.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_context.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_context.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_convert.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_convert.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_dfa_match.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_dfa_match.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_error.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_error.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_extuni.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_extuni.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_find_bracket.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_find_bracket.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_jit_compile.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_jit_compile.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_maketables.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_maketables.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_match.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_match.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_match_data.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_match_data.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_newline.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_newline.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_ord2utf.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_ord2utf.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_pattern_info.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_pattern_info.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_script_run.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_script_run.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_serialize.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_serialize.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_string_utils.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_string_utils.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_study.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_study.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_substitute.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_substitute.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_substring.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_substring.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_tables.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_tables.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_ucd.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_ucd.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_valid_utf.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_valid_utf.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/pcre/pcre2lib/pcre2_xclass.$O: $PHPSRC/ext/pcre/pcre2lib/pcre2_xclass.c + mkdir -p $OBJDIRS + $CC $CFLAGS $PCRE2FLAGS $prereq -o $target + +ext/hash/hash_sha3.$O: $PHPSRC/ext/hash/hash_sha3.c + mkdir -p $OBJDIRS + $CC $CFLAGS $KECCAKFLAGS $prereq -o $target + +ext/hash/sha3/generic64lc/KeccakHash.$O: $PHPSRC/ext/hash/sha3/generic64lc/KeccakHash.c + mkdir -p $OBJDIRS + $CC $CFLAGS $KECCAKFLAGS $prereq -o $target + +ext/hash/sha3/generic64lc/KeccakP-1600-opt64.$O: $PHPSRC/ext/hash/sha3/generic64lc/KeccakP-1600-opt64.c + mkdir -p $OBJDIRS + $CC $CFLAGS $KECCAKFLAGS $prereq -o $target + +ext/hash/sha3/generic64lc/KeccakSponge.$O: $PHPSRC/ext/hash/sha3/generic64lc/KeccakSponge.c + mkdir -p $OBJDIRS + $CC $CFLAGS $KECCAKFLAGS $prereq -o $target + # mkone's `clean' is `rm -f *.[$OS] ...', which cannot see an object in # a subdirectory -- and 512 of the 513 here are in one. Overriding it # means repeating what it does, the same as `cmd/bash/mkfile'.