diff --git a/README.md b/README.md index 676c3a960..d7e072bc7 100644 --- a/README.md +++ b/README.md @@ -1835,7 +1835,7 @@ and requires stdout, stderr and exit code to match on each. The gate count above are the dated counts snapshot in `docs/counts.tsv` (`python3 docs/counts_snapshot.py show` prints today's numbers beside it), so two open changes that each add a gate or a cap never conflict on a total. -Another gate derives the cap inventory. At the counts snapshot 2026-11-22 the tool had 454 compile-time caps and 10 ranking parameters. +Another gate derives the cap inventory. At the counts snapshot 2026-11-22 the tool had 456 compile-time caps and 10 ranking parameters. [`docs/LIMITS.md`](docs/LIMITS.md) lists each cap, its value, and whether the file discloses a truncation when the cap fires, and `python3 docs/limits_build.py --check` proves that list against `src/`. [`docs/TUNING.md`](docs/TUNING.md) lists the measured cost of the caps the sweep has measured, and names diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md index 0a950b665..bf55711ff 100644 --- a/docs/COMMANDS.md +++ b/docs/COMMANDS.md @@ -3828,7 +3828,7 @@ $ ./build/ripwire . --whereis=rankGraphTeleport --whereis-listing=all **Answers:** the dark-content dashboard: what is built but switched OFF in this repo the dark-content dashboard: what is BUILT but OFF in this repo. -Harvests all three gate patterns — #ifndef/#define header gates, CMake option(), and getenv() reads — and reports gate, kind (compile/cmake/env), default, the size of the code it guards (#if regions and their LOC), and its read sites. When a name is BOTH a header gate and a CMake option the CMake default WINS (that is what the build actually passes) and both sites are listed. A gate whose default IS another gate's name (#define F_WALLS F_ALL) is resolved: it inherits the master's default and rolls its guarded size up, so a master switch shows rather than a misleading loc="0". LIMITS: lexical, not preprocessed — a gate computed at configure time or set only in a CI script shows its in-repo default, never the value your build used. A gate needs a VALUE (#ifndef F / #define F 0) to be a gate: valueless pairs are include guards and are excluded, and a gate read as a VALUE (constexpr bool k = F != 0, then if constexpr) reports regions="0" honestly — its code is a C++ branch, not an #if region. Pair it with --flip=NAME below to size ONE gate instead of listing them all. BASH: in .sh/.bash/.zsh every $X / ${X...} expansion is an env read unless the same file assigned X before it (X=, export X=, local/declare/readonly, read, for, mapfile, getopts, printf -v, ((X=…)); read/select/getopts/mapfile also set REPLY/OPTARG/ MAPFILE). An assignment takes effect where its command ends; a prefix X=v cmd is not a write; a function's local/declare (and any later write of that name in it) hides reads only to that function's end, a write inside $( ) or ( ) only to that subshell's end, and a write in a pipeline component or a background job (cmd | read X, X=1 &, a piped while loop) nothing past its command (after shopt -s lastpipe the pipeline's last command runs in the script itself). Each read row carries form= (required ${X:?}/${X?}, default ${X:-d}, unset-default ${X-d}, settest ${X:+w}/${X+w} or a -n/-z operand of [ / [[ / test, assign ${X:=d}, plain $X), colon=1|0 (does an EMPTY value count as unset), default= the literal word, and test=n|z; write rows list the name's assignments repo-wide; ambient="1" tags a name Bash or the login always sets (PWD, HOME, BASH_SOURCE, RANDOM and the like). A shell file whose header is a GENERATED/DO-NOT-EDIT banner is skipped (generated_skipped=N). Bash LIMITS: lexical order within one file, no flow — a name set only by eval, by printf -v "$1", in a function defined below the read, or in a sourced file still reads as external (the write rows let a consumer that follows source join them). An expansion nested more than 32 deep in another's word is not read past that depth, and the root counts such files (nesting_cut=N). A file --flags cannot open or read to its end is counted as unreadable=N on the root: every count is then a floor. .zsh files are read with Bash's rules (no zsh-only expansion flags), and a script with no .sh/.bash/.zsh extension (a bare #!/bin/bash file) is not scanned. --flags --json emits the same rows as JSON (raise --limit to lift the site cap). +Harvests all three gate patterns — #ifndef/#define header gates, CMake option(), and getenv() reads — and reports gate, kind (compile/cmake/env), default, the size of the code it guards (#if regions and their LOC), and its read sites. When a name is BOTH a header gate and a CMake option the CMake default WINS (that is what the build actually passes) and both sites are listed. A gate whose default IS another gate's name (#define F_WALLS F_ALL) is resolved: it inherits the master's default and rolls its guarded size up, so a master switch shows rather than a misleading loc="0". LIMITS: lexical, not preprocessed — a gate computed at configure time or set only in a CI script shows its in-repo default, never the value your build used. A gate needs a VALUE (#ifndef F / #define F 0) to be a gate: valueless pairs are include guards and are excluded, and a gate read as a VALUE (constexpr bool k = F != 0, then if constexpr) reports regions="0" honestly — its code is a C++ branch, not an #if region. Pair it with --flip=NAME below to size ONE gate instead of listing them all. BASH: in .sh/.bash/.zsh every $X / ${X...} expansion is an env read unless the same file assigned X before it (X=, export X=, local/declare/readonly, read, for, mapfile, getopts, printf -v, ((X=…)); read/select/getopts/mapfile also set REPLY/OPTARG/ MAPFILE). An assignment takes effect where its command ends; a prefix X=v cmd is not a write; a function's local/declare (and any later write of that name in it) hides reads only to that function's end (a local run in a child shell, ( local X ), binds nothing past that shell, and declare -g writes the global), a write inside $( ) or ( ) only to that subshell's end, and a write in a pipeline component or a background job (cmd | read X, X=1 &, a piped while loop) nothing past its command (after shopt -s lastpipe the pipeline's last command runs in the script itself). Each read row carries form= (required ${X:?}/${X?}, default ${X:-d}, unset-default ${X-d}, settest ${X:+w}/${X+w} or a -n/-z operand of [ / [[ / test, assign ${X:=d}, plain $X), colon=1|0 (does an EMPTY value count as unset), default= the literal word, and test=n|z; write rows list the name's assignments repo-wide; ambient="1" tags a name Bash or the login always sets (PWD, HOME, BASH_SOURCE, RANDOM and the like). A shell file whose header is a GENERATED/DO-NOT-EDIT banner is skipped (generated_skipped=N). Bash LIMITS: lexical order within one file, no flow — a name set only by eval, by printf -v "$1", in a function defined below the read, or in a sourced file still reads as external (the write rows let a consumer that follows source join them). An expansion nested more than 32 deep in another's word is not read past that depth, and the root counts such files (nesting_cut=N). A file --flags cannot open or read to its end is counted as unreadable=N on the root, and one too big to read (over the crawl's --max-file-size, a data lane's own ceiling (YAML 512 KB), or --flags' own 64 MB cap) as oversize=N, with oversize_limit= the smallest ceiling that dropped one. JSON over its 256 KB ceiling is not counted: no gate is live in a .json file. With either count present, gates= is a floor, and dark_gates= and the per-kind counts are incomplete in either direction (a declaration in a missing file can reclassify a gate). .zsh files are read with Bash's rules (no zsh-only expansion flags), and a script with no .sh/.bash/.zsh extension (a bare #!/bin/bash file) is not scanned. --flags --json emits the same rows and root counts as JSON, read the same way (raise --limit to lift the site cap). **Try it** @@ -4956,7 +4956,7 @@ $ ./build/ripwire . --max-file-size=8K --top-k=3 ... [17 more line(s); run it to see the whole thing] ``` -**Shaped by:** `--expand`, `--skipped` +**Shaped by:** `--expand`, `--flags`, `--skipped` **Caveats (stated by the binary):** diff --git a/docs/LIMITS.md b/docs/LIMITS.md index 7d51a7f6b..3d3775b37 100644 --- a/docs/LIMITS.md +++ b/docs/LIMITS.md @@ -51,10 +51,10 @@ cap do not both rewrite the same line; the per-file tables are re-read from `src | total caps | files | caps whose file discloses | caps whose file discloses NOTHING | | --- | --- | --- | --- | -| 454 | 156 | 289 | **165** | +| 456 | 156 | 291 | **165** | Plus 10 ranking and apportionment parameters, in their own table below: they are not caps, they -are not counted as caps, and 454 + 10 is the 464 constants this generator parses out of `src/`. +are not counted as caps, and 456 + 10 is the 466 constants this generator parses out of `src/`. ## INDEXING, OUTPUT or BOUNDARY — which half of the answer a cap bounds @@ -72,8 +72,8 @@ None of them truncates anything, so none can be judged by `shown=`/`total=` and a disclosure — labelling them OUTPUT would ask for a `capped="1"` that could never honestly fire. The distinction was named in review on #108 and the rows below now carry it. -The `class` column below carries that answer where it is known. **146 of 454 caps are classified -(54 INDEXING, 46 OUTPUT, 46 BOUNDARY); the remaining 308 render `—`, which means NOT YET +The `class` column below carries that answer where it is known. **148 of 456 caps are classified +(56 INDEXING, 46 OUTPUT, 46 BOUNDARY); the remaining 308 render `—`, which means NOT YET CLASSIFIED — never "neither".** Classifications live in `docs/limits_classes.tsv`, a sidecar with a known expiry: the tag belongs on the declaration itself, and this file exists only because the round that @@ -274,6 +274,8 @@ Discloses: `capped`, `reads_capped`, `writes_capped` | --- | --- | --- | --- | | `kMaxAliasDepth` | `8` | INDEXING | — | | `kMaxEnvNameLen` | `128` | BOUNDARY | longest plausible environment-variable name | +| `kMaxFlagFileBytes` | `67108864` | INDEXING | 64 MiB read cap; a file past it counts in oversize= (#699) | +| `kMaxReadWholeBytes` | `4194304` | INDEXING | 4 MiB read cap for every verb but --flags; a file past it is disclosed unread | | `kMaxShellNesting` | `32` | INDEXING | — | | `kMaxSitesShown` | `8` | OUTPUT | sites per gate; a DEFAULT, raisable by --limit=N (effectiveRowCap), lifted by --detail | diff --git a/docs/counts.tsv b/docs/counts.tsv index bc12fe54f..c93a5da2f 100644 --- a/docs/counts.tsv +++ b/docs/counts.tsv @@ -5,11 +5,11 @@ taken 2026-11-22 gates.scripts 865 limits.boundary 46 -limits.caps 454 -limits.constants 464 -limits.disclosing 289 +limits.caps 456 +limits.constants 466 +limits.disclosing 291 limits.files 156 -limits.indexing 54 +limits.indexing 56 limits.output 46 limits.parameters 10 limits.silent 165 diff --git a/docs/limits_classes.tsv b/docs/limits_classes.tsv index 954856c6a..4c965480f 100644 --- a/docs/limits_classes.tsv +++ b/docs/limits_classes.tsv @@ -77,6 +77,7 @@ kMaxExtLen BOUNDARY kMaxExtentParens BOUNDARY kMaxFamily INDEXING kMaxFieldsShown OUTPUT +kMaxFlagFileBytes INDEXING kMaxFlipRows OUTPUT kMaxFnsShown OUTPUT kMaxHexDigits BOUNDARY @@ -96,6 +97,7 @@ kMaxPlanScout OUTPUT kMaxProbeCommits INDEXING kMaxQualifierHops INDEXING kMaxQueryNesting BOUNDARY +kMaxReadWholeBytes INDEXING kMaxRefs INDEXING kMaxRenameProbeFiles INDEXING kMaxSample INDEXING diff --git a/src/cli.h b/src/cli.h index 64d41f16b..35dd7313e 100644 --- a/src/cli.h +++ b/src/cli.h @@ -2359,7 +2359,9 @@ inline constexpr char kHelpTail[] = " getopts, printf -v, ((X=…)); read/select/getopts/mapfile also set REPLY/OPTARG/\n" " MAPFILE). An assignment takes effect where its command ends; a prefix X=v cmd is not a\n" " write; a function's local/declare (and any later write of that name in it) hides reads\n" - " only to that function's end, a write inside $( ) or ( ) only to that subshell's end,\n" + " only to that function's end (a local run in a child shell, ( local X ), binds nothing\n" + " past that shell, and declare -g writes the global), a write inside $( ) or ( ) only\n" + " to that subshell's end,\n" " and a write in a pipeline component or a background job (cmd | read X, X=1 &, a piped\n" " while loop) nothing past its command (after shopt -s lastpipe the pipeline's last\n" " command runs in the script itself). Each read row carries form= (required\n" @@ -2374,10 +2376,16 @@ inline constexpr char kHelpTail[] = " as external (the write rows let a consumer that follows source join them). An\n" " expansion nested more than 32 deep in another's word is not read past that depth, and\n" " the root counts such files (nesting_cut=N). A file --flags cannot open or read to its\n" - " end is counted as unreadable=N on the root: every count is then a floor. .zsh files\n" - " are read with Bash's rules (no zsh-only expansion flags), and a script with no\n" + " end is counted as unreadable=N on the root, and one too big to read (over the crawl's\n" + " --max-file-size, a data lane's own ceiling (YAML 512 KB), or --flags' own 64 MB cap)\n" + " as oversize=N, with oversize_limit= the smallest ceiling that dropped one. JSON over\n" + " its 256 KB ceiling is not counted: no gate is live in a .json file. With either count\n" + " present, gates= is a floor, and dark_gates= and the per-kind counts are incomplete\n" + " in either direction (a declaration in a missing file can reclassify a gate). .zsh\n" + " files are read with Bash's rules (no zsh-only expansion flags), and a script with no\n" " .sh/.bash/.zsh extension (a bare #!/bin/bash file) is not scanned.\n" - " --flags --json emits the same rows as JSON (raise --limit to lift the site cap).\n" + " --flags --json emits the same rows and root counts as JSON, read the same way (raise\n" + " --limit to lift the site cap).\n" " --flip=NAME (with --flags) the blast radius of turning ONE gate on\n" " (with --flags) the BLAST RADIUS of turning ONE gate ON: which code becomes live, how\n" " much, which SYMBOLS hold it, what those transitively reach, and which TESTS cover it —\n" diff --git a/src/darkflags.h b/src/darkflags.h index a67d03938..8196390e2 100644 --- a/src/darkflags.h +++ b/src/darkflags.h @@ -63,7 +63,13 @@ namespace rw namespace darkflags { -constexpr std::size_t kMaxFlagFileBytes = 4u << 20; // 4 MB — past this a "source" file is generated data +// 64 MiB — past this a "source" file is generated data. A runaway guard, not a filter: GitHub warns at 50 MB and refuses +// 100 MB, and the crawl's own default ceiling (--max-file-size, 4 MB) drops a file long before this one does. A file +// either ceiling drops is counted in oversize= on the root (#699): gates= is then a floor, the other counts incomplete. +constexpr std::size_t kMaxFlagFileBytes = 67108864; // 64 MiB read cap; a file past it counts in oversize= (#699) +// Every OTHER readWhole caller (--doc-drift, subject anchors, lint, layout, field uses, ...) keeps the 4 MiB read cap its +// legend names ("unreadable or over 4 MiB"): #699 raised the --flags cap only, so computeFlags passes kMaxFlagFileBytes. +constexpr std::size_t kMaxReadWholeBytes = 4194304; // 4 MiB read cap for every verb but --flags; a file past it is disclosed unread constexpr std::size_t kMaxSitesShown = 8; // sites per gate; a DEFAULT, raisable by --limit=N (effectiveRowCap), lifted by --detail constexpr std::size_t kMaxEnvNameLen = 128; // longest plausible environment-variable name @@ -1287,6 +1293,7 @@ struct ShellFunctionWrite std::size_t write = 0; // fh.writes index std::uint32_t function = 0; // the innermost function body it sits in bool isLocal = false; // it IS the `local` / in-function `declare` that binds the name + bool isGlobal = false; // `declare -g`: it sets the global even while a `local` binds the name (#701) }; // The scan-time narrowing chain over fh.writes (shellNarrowFrom): next[k] == k while write k has not been narrowed, @@ -1298,6 +1305,20 @@ struct ShellNarrowChain std::pair last{ 0, 0 }; }; +// A write waiting for its command to END, which is where it takes effect (FileHarvest::Read::effLine/col): `X=v` +// at command position is dropped if a command word follows (`X=v cmd` reaches only cmd), and every write — a +// keyword's too — commits at the separator, the line end or the `)` that closes its command, so a right-hand +// side that spans lines (`X='a` + `'${X:?}`) still reads the caller's X. function: the function a `local` binds +// to. isImplicit: the REPLY / MAPFILE a `read` / `mapfile` writes when it names no variable, withdrawn when it does. +// isGlobal: a `declare -g` write, which sets the global even inside a function that made the name local. +struct ShellPending +{ + FileHarvest::Read write; + std::uint32_t function = UINT32_MAX; // ShellScan::kNoFunction + bool isImplicit = false; + bool isGlobal = false; +}; + // The scanner's state across lines: quoting and command context nest (`"$( cmd "$X" )"`), and both `'…'` and `"…"` // may span lines, so this is per FILE, like forEachCodeByte's block-comment flag. struct ShellScan @@ -1306,17 +1327,8 @@ struct ShellScan enum class Kw : std::uint8_t { None, Decl, Export, Read, MapFile, ForName, GetOpts, PrintfV, Case, Shopt }; static constexpr std::uint32_t kNoFunction = UINT32_MAX; - // A write waiting for its command to END, which is where it takes effect (FileHarvest::Read::effLine/col): `X=v` - // at command position is dropped if a command word follows (`X=v cmd` reaches only cmd), and every write — a - // keyword's too — commits at the separator, the line end or the `)` that closes its command, so a right-hand - // side that spans lines (`X='a` + `'${X:?}`) still reads the caller's X. function: the function a `local` binds - // to. isImplicit: the REPLY / MAPFILE a `read` / `mapfile` writes when it names no variable, withdrawn when it does. - struct Pending - { - FileHarvest::Read write; - std::uint32_t function = kNoFunction; - bool isImplicit = false; - }; + using Pending = ShellPending; + static_assert( kNoFunction == UINT32_MAX, "ShellPending::function defaults to kNoFunction, spelled UINT32_MAX there" ); struct Frame { @@ -1335,6 +1347,7 @@ struct ShellScan Kw kw = Kw::None; bool isDeclLocal = false; // local, or declare/typeset inside a function: the write ends with it bool isDeclKeep = false; // a bare NAME keeps its inherited value (readonly, -x, -g, top-level declare) + bool isDeclGlobal = false; // declare -g / typeset -g: the write is the global's, whatever the function made local char optArg = 0; // the option letter whose argument the next word is (read -p, mapfile -d, …) std::uint8_t wordsLeft = 0; // getopts: optstring to skip before the name std::vector pending; // this command's writes, until it ends @@ -1450,7 +1463,7 @@ inline void shellDeclOption( ShellScan::Frame& f, std::string_view opt ) f.kw = ShellScan::Kw::None; return; } - if( c == 'g' ) { f.isDeclLocal = false; } + if( c == 'g' ) { f.isDeclLocal = false; f.isDeclGlobal = true; } if( c == 'g' || c == 'x' ) { f.isDeclKeep = true; } } } @@ -1600,6 +1613,7 @@ inline void shellCommandWord( std::string_view line, std::size_t at, const Shell f.kw = kw->kw; f.wordsLeft = kw->wordsLeft; f.isDeclLocal = isScopable && ( w.word == "local" || isInFunction ); + f.isDeclGlobal = false; f.isDeclKeep = kw->kw == ShellScan::Kw::Decl && !f.isDeclLocal; // readonly X / top-level declare X keep X if( kw->kw == ShellScan::Kw::Read || kw->kw == ShellScan::Kw::MapFile || w.word == "select" ) { @@ -1616,6 +1630,7 @@ inline void shellKeywordWrite( std::size_t at, const ShellWord& w, ShellScan::Kw const bool isLocal = kwBefore == ShellScan::Kw::Decl && f.isDeclLocal && st.function() != ShellScan::kNoFunction; std::erase_if( f.pending, []( const ShellScan::Pending& p ) { return p.isImplicit; } ); shellHoldWrite( f, w.name, lineNo, at, isLocal ? st.function() : ShellScan::kNoFunction ); + f.pending.back().isGlobal = kwBefore == ShellScan::Kw::Decl && f.isDeclGlobal; if( kwBefore == ShellScan::Kw::GetOpts ) { shellHoldWrite( f, "OPTARG", lineNo, at, ShellScan::kNoFunction ); @@ -1890,7 +1905,7 @@ inline void shellCommitPending( ShellScan& st, ShellScan::Frame& f, FileHarvest& } if( enclosing != ShellScan::kNoFunction ) { - st.functionWrites.push_back( ShellFunctionWrite{ fh.writes.size(), enclosing, p.function != ShellScan::kNoFunction } ); + st.functionWrites.push_back( ShellFunctionWrite{ fh.writes.size(), enclosing, p.function != ShellScan::kNoFunction, p.isGlobal } ); } fh.writes.push_back( std::move( p.write ) ); } @@ -2325,10 +2340,39 @@ inline std::vector shellShadowingWrites( const FileHarvest& f return shadowing; } +// One (function, name) group of shellCloseScopes, in effect order: [from, to) of `inFunction`. A `local` binds the name +// from where it takes effect to where its own shell ends — the function's `}`, or sooner when it ran in a child shell +// inside the function (`( local X )`, `$( local X )`, `local X | cat`), whose end the scan already narrowed it to (#700). +// A later write in effect inside a binding sets the local, so it ends at the function's `}` too; a write past every +// binding, or a `declare -g` (#701), sets the global and keeps the scope it has. +inline void shellNarrowBoundWrites( const std::vector& inFunction, std::size_t from, std::size_t to, + std::pair functionEnd, FileHarvest& fh ) +{ + using Pos = std::pair; + Pos bindEnd{ 0, 0 }; // the furthest end of a local binding met so far (exclusive) + for( std::size_t k = from; k < to; ++k ) + { + FileHarvest::Read& w = fh.writes[ inFunction[k].write ]; + if( inFunction[k].isGlobal ) + { + continue; + } + if( inFunction[k].isLocal ) + { + bindEnd = std::max( bindEnd, std::min( Pos{ w.scopeLine, w.scopeCol }, functionEnd ) ); + } + if( Pos{ w.effLine, w.col } < bindEnd ) + { + shellNarrowScope( w, functionEnd.first, functionEnd.second ); + } + } +} + // Each function-local write stops shadowing at its function's `}` (a function the file never closes runs to the end), // or sooner when a subshell inside the function already ended it. A LATER write in that function to a name it made // local (`local X; X=v`, `read X`, `: ${X:=d}`) sets the local too, so it ends there as well; before the `local`, a -// write is still the script's. The function's writes are grouped by (function, name) in effect order: one sort. +// write is still the script's, and so is one after a `local` that ran in a child shell or one made by `declare -g` +// (shellNarrowBoundWrites). The function's writes are grouped by (function, name) in effect order: one sort. inline void shellCloseScopes( const ShellScan& shell, FileHarvest& fh ) { std::vector inFunction = shell.functionWrites; @@ -2338,19 +2382,17 @@ inline void shellCloseScopes( const ShellScan& shell, FileHarvest& fh ) const FileHarvest::Read& y = fh.writes[ b.write ]; return std::tie( a.function, x.name, x.effLine, x.col, a.write ) < std::tie( b.function, y.name, y.effLine, y.col, b.write ); } ); - for( std::size_t k = 0; k < inFunction.size(); ) + for( std::size_t from = 0; from < inFunction.size(); ) { - const std::uint32_t function = inFunction[k].function; - const std::string& name = fh.writes[ inFunction[k].write ].name; - bool isLocal = false; - for( ; k < inFunction.size() && inFunction[k].function == function && fh.writes[ inFunction[k].write ].name == name; ++k ) + const std::uint32_t function = inFunction[ from ].function; + const std::string& name = fh.writes[ inFunction[ from ].write ].name; + std::size_t to = from; + while( to < inFunction.size() && inFunction[ to ].function == function && fh.writes[ inFunction[ to ].write ].name == name ) { - isLocal = isLocal || inFunction[k].isLocal; - if( isLocal ) - { - shellNarrowScope( fh.writes[ inFunction[k].write ], shell.functionEnds[ function ].first, shell.functionEnds[ function ].second ); - } + ++to; } + shellNarrowBoundWrites( inFunction, from, to, shell.functionEnds[ function ], fh ); + from = to; } for( const auto& [ writeIndex, function ] : shell.scopedWrites ) { @@ -2542,14 +2584,16 @@ inline FileHarvest harvestFile( std::string_view bytes, std::string_view path, b // ── tree walk + resolution ─────────────────────────────────────────────────────────────────────────────── -// Why readWhole returned nothing: the file could not be opened or read to its end, or it grew past kMaxFlagFileBytes. +// Why readWhole returned nothing: the file could not be opened or read to its end, or it grew past its read cap. enum class ReadWholeFail : std::uint8_t { None, Unreadable, TooLarge }; -// The whole file, or nullopt when it cannot be opened, cannot be read to its end, or grows past kMaxFlagFileBytes -// (`why`, when given, says which). A read error part-way through is a failure, never a shorter file: fread's short -// count alone cannot tell an error from the end, so the stream's error flag decides (#641 merge review — the prefix -// used to be harvested as if it were the whole file). An empty file is an engaged empty string. -inline std::optional readWhole( const std::string& path, ReadWholeFail* why = nullptr ) +// The whole file, or nullopt when it cannot be opened, cannot be read to its end, or grows past maxBytes +// (kMaxReadWholeBytes unless the caller names its own cap; `why`, when given, says which). A read error part-way +// through is a failure, never a shorter file: fread's short count alone cannot tell an error from the end, so the +// stream's error flag decides (#641 merge review — the prefix used to be harvested as if it were the whole file). +// An empty file is an engaged empty string. +inline std::optional readWhole( const std::string& path, ReadWholeFail* why = nullptr, + std::size_t maxBytes = kMaxReadWholeBytes ) { const auto fail = [ & ]( ReadWholeFail reason ) -> std::optional { @@ -2570,7 +2614,7 @@ inline std::optional readWhole( const std::string& path, ReadWholeF { n = std::fread( buf, 1, sizeof( buf ), fp ); out.append( buf, n ); - isTooLarge = out.size() > kMaxFlagFileBytes; + isTooLarge = out.size() > maxBytes; } const bool isReadError = std::ferror( fp ) != 0; std::fclose( fp ); @@ -2749,6 +2793,19 @@ struct FlagsResult void disclose( DisclosureWhy ) noexcept { ++files; } }; UnreadableFiles unreadable; + // The DISCLOSE sink for a file whose gates are missing because it is too big (#699): the crawl dropped it over its + // size ceiling (IngestResult::skippedOversize, a file this scan would have read), or readWhole refused it past + // kMaxFlagFileBytes (ReadWholeFail::TooLarge). Every count on the root is then a floor. limitBytes: the smallest + // ceiling that dropped one. Absent from the XML when zero. + struct OversizeFiles + { + enum class DisclosureWhy : std::uint8_t { CrawlCeiling, FlagsReadCap }; + std::uint64_t files = 0; + std::uint64_t limitBytes = 0; + void disclose( DisclosureWhy ) noexcept { ++files; } + void noteLimit( std::uint64_t bytes ) noexcept { limitBytes = limitBytes == 0 ? bytes : std::min( limitBytes, bytes ); } + }; + OversizeFiles oversize; }; // Fold one harvested definition into the gate table, applying the cmake-beats-compile override rule. @@ -2915,6 +2972,40 @@ inline void finishGate( Gate& g ) && std::any_of( g.reads.begin(), g.reads.end(), []( const Site& r ) { return r.sh.form != ShellForm::None; } ); } +// The files the crawl dropped over a size ceiling (--max-file-size, or a data lane's own) whose gates are missing too +// (#699): every one this scan would have read, except JSON. harvestFile reads no prose file, and a .json file is data +// with no code in it — no #if, option() or env read can be live there — so the crawl's 256 KB JSON data ceiling, +// which fires on fixtures and dumps in most repositories, hides no gate. +inline FlagsResult::OversizeFiles crawlOversize( const IngestResult& ing ) +{ + FlagsResult::OversizeFiles oversize; + for( const SkippedOversize& s : ing.skippedOversize ) + { + if( !lineSyntaxFor( s.path ).isProse && docparse::lowerExtOf( s.path ) != ".json" ) + { + DISCLOSE( oversize, FlagsResult::OversizeFiles::DisclosureWhy::CrawlCeiling, + "flags: the crawl dropped a file over its size ceiling — its gates are missing (oversize=)" ); + oversize.noteLimit( s.limitBytes ); + } + } + return oversize; +} + +// A file readWhole refused: its gates are missing, counted in unreadable= (it could not be opened or read to its end) +// or in oversize= (it grew past kMaxFlagFileBytes). +inline void discloseUnread( ReadWholeFail why, FlagsResult::UnreadableFiles& unreadable, FlagsResult::OversizeFiles& oversize ) +{ + if( why == ReadWholeFail::TooLarge ) + { + DISCLOSE( oversize, FlagsResult::OversizeFiles::DisclosureWhy::FlagsReadCap, + "flags: a file grew past kMaxFlagFileBytes — its gates are missing (oversize=)" ); + oversize.noteLimit( kMaxFlagFileBytes ); + return; + } + DISCLOSE( unreadable, FlagsResult::UnreadableFiles::DisclosureWhy::OpenOrReadFailed, + "flags: a file could not be opened or read to its end — its gates are missing (unreadable=)" ); +} + // Harvest every file, then resolve: gate identities first (so a read site can be attributed only to a name // that is actually a declared gate), then regions and reads, then alias chains. // `keepUnreadGates` (default false = the `--flags` contract, byte-identical): normally a DECLARED name with @@ -2929,18 +3020,15 @@ inline FlagsResult computeFlags( const IngestResult& ing, const std::string& roo std::uint64_t generatedSkipped = 0; std::uint64_t nestingCut = 0; FlagsResult::UnreadableFiles unreadable; + FlagsResult::OversizeFiles oversize = crawlOversize( ing ); const auto scan = [ & ]( const std::string& full, bool isCMake ) { ReadWholeFail why = ReadWholeFail::None; - const std::optional bytes = readWhole( full, &why ); + const std::optional bytes = readWhole( full, &why, kMaxFlagFileBytes ); if( !bytes ) { - if( why == ReadWholeFail::Unreadable ) - { - DISCLOSE( unreadable, FlagsResult::UnreadableFiles::DisclosureWhy::OpenOrReadFailed, - "flags: a file could not be opened or read to its end — its gates are missing (unreadable=)" ); - } + discloseUnread( why, unreadable, oversize ); return; } std::string rel( relForHash( full, root ) ); @@ -2992,6 +3080,7 @@ inline FlagsResult computeFlags( const IngestResult& ing, const std::string& roo res.generatedSkipped = generatedSkipped; res.nestingCut = nestingCut; res.unreadable = unreadable; + res.oversize = oversize; std::size_t filterNameHits = 0; for( auto& [ name, g ] : gates ) { @@ -3196,6 +3285,40 @@ inline std::string flagsNextInvocation( const FlagsResult& res, std::size_t maxS return widestCut == 0 ? std::string() : flags + std::string( extra ) + " --limit=" + std::to_string( widestCut ); } +// The legend for the root's counts of files whose gates are missing (unreadable=, oversize=): present only when one is. +inline void writeMissingFilesLegend( std::FILE* out, const FlagsResult& res ) +{ + if( res.unreadable.files != 0 ) + { + rw::emitRaw( out, "" ); + } + if( res.oversize.files != 0 ) + { + rw::emitTo( out, "", + kMaxYamlConfigBytes >> 10, kMaxFlagFileBytes >> 20, kMaxJsonConfigBytes >> 10 ); + } +} + +// The root's scan-cut attributes, each absent when zero: generated_skipped=, nesting_cut=, unreadable=, oversize= with +// oversize_limit=. +inline std::string flagsScanCutAttrs( const FlagsResult& res ) +{ + const auto count = []( std::string_view name, std::uint64_t n ) + { + return n == 0 ? std::string() : " " + std::string( name ) + "=\"" + std::to_string( n ) + "\""; + }; + return count( "generated_skipped", res.generatedSkipped ) + count( "nesting_cut", res.nestingCut ) + + count( "unreadable", res.unreadable.files ) + count( "oversize", res.oversize.files ) + + ( res.oversize.files == 0 ? std::string() : " oversize_limit=\"" + std::to_string( res.oversize.limitBytes ) + "\"" ); +} + inline void writeFlags( std::FILE* out, const FlagsResult& res, std::size_t maxSites, int pageOffset = 0 ) { std::vector esc; @@ -3212,11 +3335,7 @@ inline void writeFlags( std::FILE* out, const FlagsResult& res, std::size_t maxS "verb's own harvest scan (source + CMakeLists files it read looking for gates) — a wider crawl than the " "map's indexed corpus, so it will not equal the map's files= -->" ); writeShellLegend( out, res ); - if( res.unreadable.files != 0 ) - { - rw::emitRaw( out, "" ); - } + writeMissingFilesLegend( out, res ); // C1 F-07: the site listing's own vocabulary, DEFINED where the reader meets it (legendcoveragecheck's // rule). Emitted unconditionally because it describes a listing every run carries, unlike the attributes // themselves, which appear only on a gate that was actually cut. @@ -3247,10 +3366,7 @@ inline void writeFlags( std::FILE* out, const FlagsResult& res, std::size_t maxS : " escaped_root=\"" + std::to_string( res.escapedRoot ) + "\""; // Absent when false: a repo whose CMake sub-walk always succeeds keeps a byte-identical report. const std::string fgCmakeFailedAttr = res.cmakeScanFailed ? " cmake_scan_failed=\"1\"" : std::string(); - const std::string fgGeneratedAttr = ( res.generatedSkipped == 0 ? std::string() - : " generated_skipped=\"" + std::to_string( res.generatedSkipped ) + "\"" ) - + ( res.nestingCut == 0 ? std::string() : " nesting_cut=\"" + std::to_string( res.nestingCut ) + "\"" ) - + ( res.unreadable.files == 0 ? std::string() : " unreadable=\"" + std::to_string( res.unreadable.files ) + "\"" ); + const std::string fgGeneratedAttr = flagsScanCutAttrs( res ); rw::emitTo( out, "", res.gates.size(), res.dark, res.compileCount, res.cmakeCount, res.envCount, res.filesScanned, fgGeneratedAttr.c_str(), fgEscapedAttr.c_str(), fgCmakeFailedAttr.c_str(), fgFilterAttr.c_str(), flagsNext.c_str() ); @@ -3336,6 +3452,10 @@ inline void writeFlagsJson( std::FILE* out, const FlagsResult& res, std::size_t if( res.generatedSkipped != 0 ) { doc += ",\"generated_skipped\":" + std::to_string( res.generatedSkipped ); } if( res.nestingCut != 0 ) { doc += ",\"nesting_cut\":" + std::to_string( res.nestingCut ); } if( res.unreadable.files != 0 ) { doc += ",\"unreadable\":" + std::to_string( res.unreadable.files ); } + if( res.oversize.files != 0 ) + { + doc += ",\"oversize\":" + std::to_string( res.oversize.files ) + ",\"oversize_limit\":" + std::to_string( res.oversize.limitBytes ); + } if( res.escapedRoot != 0 ) { doc += ",\"escaped_root\":" + std::to_string( res.escapedRoot ); } if( res.cmakeScanFailed ) { doc += ",\"cmake_scan_failed\":true"; } if( !res.filter.empty() ) { doc += ",\"filter\":" + q( res.filter ); } diff --git a/test/bashenvcheck.sh b/test/bashenvcheck.sh index d7ecd2968..771f24e87 100755 --- a/test/bashenvcheck.sh +++ b/test/bashenvcheck.sh @@ -39,10 +39,20 @@ # redirection after an assignment; '…' in an unquoted expansion word vs in "…"; <<"FU\"EOF" # and the #685 review's two: `cmd | ( … )` ends its pipeline at the newline, and `&` backgrounds # the whole && / || list (a `;` or the `&` itself ends that list) +# scope.sh #700 / #701 (reported by jflowers): a `local` that ran in a child shell (`( local X )`, `$( local +# X )`, `local X | cat`) binds nothing past that shell, and `declare -g X=v` writes the global even +# after `local X` -> SC_SUB SC_CMD SC_PIPE SC_GLOB SC_GONLY absent. Negatives stay requirements: a +# plain `local X; X=2` (SC_PLAIN), a write after a child shell's `declare -g X=v` (SC_REL: the local +# still binds), a real `local` after a child-shell one (SC_TWICE) -- each red under a guard-off build +# -- plus SC_INNER / SC_NOLOC, writes inside a subshell, which the subshell's own end already scopes # shopt.sh the #688 review's input: a `shopt -u lastpipe` in a child shell (`( … )`, `$( … )`, `&`, a # group before `|`) leaves the script's lastpipe on (SH_*_KEEP absent); a foreground `shopt -u` # turns it off and a child's `shopt -s` does not turn it back on (SH_FG_OFF, SH_SUB_OFF present) # Generated in $TMP: a mode-000 shell file must be disclosed as unreadable="1" (root, legend, JSON), never skipped. +# Generated in $TMP (#699, reported by jflowers): a shell file over the crawl's 4 MB ceiling and a .cmake file over +# --flags' own 64 MiB read cap are disclosed as oversize="1" with oversize_limit= that ceiling (root, legend, JSON); a +# file at the cap and one the crawl ceiling admits are read; prose and JSON over a ceiling are not counted (--flags reads +# no prose, and JSON is data: no gate can be live in it). # Generated in $TMP: a 100,000-level ${X:-${X:-…}} line (the review's crash input) must finish, exit 0 and disclose # nesting_cut="1". 160,000 `(X=1)` lines then 160,000 `: "$X"` (2.1 MB) must finish under a 20 s cap (about 0.3 s # linear; the quadratic lookup it guards took about 56 s). @@ -128,7 +138,7 @@ if gate NOT_AMBIENT_SUPPLIED | grep -q ' ambient='; then no "NOT_AMBIENT_SUPPLIE # ── 5. GENERATED: a bundle with a GENERATED … DO NOT EDIT banner is skipped whole, and disclosed ───────── if has GEN_ONLY_IN_BUNDLE; then no "generated bundle harvested: $( gate GEN_ONLY_IN_BUNDLE )"; else ok 'generated: bundle.sh skipped (GEN_ONLY_IN_BUNDLE absent)'; fi if gate GEN_SHARED | grep -q 'reads="1" p="bundle.src.sh" l="4"'; then ok 'generated: GEN_SHARED comes from bundle.src.sh only (its "generated by" prose is not a banner)'; else no "GEN_SHARED wrong: $( gate GEN_SHARED )"; fi -if printf '%s' "$OUT" | grep -q ']* files="13" generated_skipped="1"'; then ok 'root: files="13" still counts the bundle, generated_skipped="1" discloses it'; else no "root wrong: $( printf '%s' "$OUT" | grep -oE ']*>' )"; fi +if printf '%s' "$OUT" | grep -q ']* files="14" generated_skipped="1"'; then ok 'root: files="14" still counts the bundle, generated_skipped="1" discloses it'; else no "root wrong: $( printf '%s' "$OUT" | grep -oE ']*>' )"; fi # ── 10. REVIEW EDGES (#641): nothing here may hide a requirement or call a plain read a set test ───────── row EDGE_ECHO_FLAG '' @@ -254,6 +264,81 @@ else if grep -q 'unreadable=' "$TMP/x"; then no "unreadable: the readable fixture carries unreadable="; else ok 'unreadable: absent when every file was read'; fi fi +# ── 14. SCOPE (#700 / #701, reported by jflowers): a child shell's `local` and a `declare -g` leave the name global ── +for n in SC_SUB SC_CMD SC_PIPE SC_GLOB SC_GONLY; do + if has "$n"; then no "scope: $n should be absent (its later write sets the global): $( gate "$n" )"; else ok "scope: $n absent"; fi +done +for nl in SC_PLAIN:21 SC_INNER:24 SC_NOLOC:27 SC_REL:30 SC_TWICE:33; do + row "${nl%%:*}" "" +done + +# ── 15. OVERSIZE (#699, reported by jflowers): a file too big to read is disclosed, never silently missing ── +has_in(){ grep -q ""$TMP/big/small.sh" && printf '#!/bin/bash\n: "${OV_BIG:?}"\n' >"$TMP/big/big.sh" \ + && pad "$TMP/big/big.sh" 5000000 && printf '# Big prose\n' >"$TMP/big/notes.md" && pad "$TMP/big/notes.md" 5000000 \ + && python3 -c 'import sys; open( sys.argv[1], "w" ).write( "[" + ",".join( ["\"process.env.OV_JSON\""] * 20000 ) + "]" )' "$TMP/big/data.json" \ + || no "could not write the oversize fixture" +"$BIN" "$TMP/big" --flags --no-cache --legend=full >"$TMP/bigx" 2>/dev/null || no "--flags over a 5 MB shell file exited non-zero" +"$BIN" "$TMP/big" --flags --no-cache --json >"$TMP/bigj" 2>/dev/null || no "--flags --json over a 5 MB shell file exited non-zero" +if grep -q ']* files="1" [^>]*oversize="1" oversize_limit="4194304"' "$TMP/bigx"; then ok 'oversize: a 5 MB shell file over the crawl ceiling -> files="1" oversize="1" oversize_limit="4194304" (the .md and the 400 KB .json are not counted)'; else no "oversize: crawl-ceiling root wrong: $( grep -oE ']*>' "$TMP/bigx" )"; fi +if grep -q '