Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 64 additions & 1 deletion .github/workflows/corpus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ jobs:
rc=0
/tmp/kibble -json "$dir" > "/tmp/${name}.json" || rc=$?
# Exit 1 is kibble reporting failures and exit 3 is kibble
# reporting that every row is a skip. Both are verdicts with a
# reporting that it tested no claim. Both are verdicts with a
# JSON array behind them, and a 0,0,0 pin is exactly the shape
# that exits 3. Any other nonzero is kibble unable to do its job,
# and blaming the corpus for that would be the same mistake as
Expand Down Expand Up @@ -172,3 +172,66 @@ jobs:
echo "::endgroup::"
done < corpus/mutations.tsv
exit "$status"
# Each recall row is a real bug a person found where kibble tested
# nothing. A caught=no row is a score, printed and never fatal, so the
# misses stay visible without turning the job red for work not yet
# built. A caught=yes row is a regression guard: losing that catch fails
# the job. KIBBLE and WORK let the same loop run on a laptop.
- name: Run kibble against the recall corpus
if: always()
shell: bash
run: |
set -uo pipefail
KIBBLE="${KIBBLE:-/tmp/kibble}"
WORK="${WORK:-/tmp/recall}"
test -x "$KIBBLE" || go build -o "$KIBBLE" .
mkdir -p "$WORK"
status=0
rows=0
caught=0
while IFS=$'\t' read -r name url sha doc line want proof expect; do
case "$name" in ""|\#*) continue ;; esac
rows=$((rows + 1))
dir="$WORK/${name}-${sha:0:12}"
out="$WORK/${name}-${sha:0:12}.json"
# Two rows can name one repository. It runs once.
if [ ! -s "$out" ]; then
echo "::group::${name} at ${sha:0:12}"
if [ ! -d "$dir/.git" ]; then
git init -q "$dir"
git -C "$dir" fetch -q --depth 1 "$url" "$sha"
git -C "$dir" checkout -q FETCH_HEAD
fi
rc=0
"$KIBBLE" -json "$dir" > "$out" || rc=$?
# Exit 1 reports failures and exit 3 reports nothing tested.
# Both are verdicts. Anything else is kibble unable to run.
if { [ "$rc" -gt 1 ] && [ "$rc" -ne 3 ]; } || ! jq -e 'type=="array"' "$out" >/dev/null 2>&1; then
echo "${name}: kibble itself failed (exit ${rc})"
rm -f "$out"
fi
echo "::endgroup::"
fi
hit=0
if [ -s "$out" ]; then
hit=$(jq -r --arg doc "$doc" --arg want "$want" --arg proof "$proof" '
($want | split("|")) as $ws
| [ (.[] | select((.status as $s | $ws | index($s)) and ((.detail // "") | contains($proof)))),
(.[] | .steps[]? | select((.doc // "") | endswith($doc)) | .lines[]
| select((.status as $s | $ws | index($s))
and (((.cmd // "") + "\n" + (.detail // "")) | contains($proof)))) ]
| length' "$out")
fi
where="${name} ${doc}:${line} ${want} naming \"${proof}\""
if [ "$hit" -gt 0 ]; then
caught=$((caught + 1))
echo "${where}: caught"
elif [ "$expect" = "yes" ]; then
echo "::error::${where}: no longer caught, and the row says it was"
status=1
else
echo "${where}: not caught yet"
fi
done < corpus/recall.tsv
echo "${caught} of ${rows} recall rows caught"
exit "$status"
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,12 +189,15 @@ it, so a CI log can always say which kibble produced its verdict.

The exit code tells a script what happened:

- `0` means at least one check ran and none failed.
- `0` means at least one documented claim was tested and none failed.
- `1` means the run failed. `-strict` widens what counts as a failure.
- `2` means a usage error, such as no path and no README in the current directory,
Docker unavailable, or less than 2 GiB free in Docker.
- `3` means nothing was checked: every row is a skip, so no install ran and no example
could run either. The action fails on purpose, since a check that ran nothing must not pass.
- `3` means nothing was tested: no row and no example line reached `VERIFIED`, `FAIL`,
`GAP`, or `DRIFT`. A brew formula lookup, a recipe with no binary to smoke-test, a
timeout, and a skip all leave the documents unchecked, and the summary names up to three
of those causes. The action fails on purpose, since a check that tested nothing must not
pass.
- `130` means the run was interrupted.

## What it has found
Expand Down
27 changes: 27 additions & 0 deletions corpus/recall.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# The recall corpus: real documentation bugs found by hand in repositories
# where kibble tested nothing, each pinned to the commit the bug was found at
# and to the line a reader trips on. repos.tsv proves kibble does not invent
# failures and mutations.tsv proves it catches rot planted on purpose. This
# file holds what neither can: rot that was already there and that kibble
# walked past, reading the repository as fine.
#
# A row is caught when the run reports the wanted status on that bug: an
# example line in the named document whose status is wanted and whose
# command or detail contains the proof, or a top-level row whose status is
# wanted and whose detail contains the proof. The JSON carries no line
# number, so the line column says where a person should look and the proof
# pins identity, as in mutations.tsv. Wanted statuses are separated by "|".
#
# caught=no rows are the scoreboard: the job prints whether each is caught now
# and never fails on them. Once a change catches a row, flip it to caught=yes,
# and from then on the job fails if the line stops reaching its status. Every
# bug found by hand becomes a row here before any fix for it is built.
#
# The first four came from the 2026-10-22 hand check of ten repositories
# kibble had run nothing on. Columns are tab-separated:
# name, clone URL, pinned commit, document, line, wanted status, proof, caught.
jq https://github.com/jqlang/jq.git 94b00aa807a51a8f97bc9345c03e32982e344678 README.md 56 FAIL libjq.so.1 no
goreleaser https://github.com/goreleaser/goreleaser.git a9fb415389a2e55906ae457d9f76bb77b3cb6056 www/content/getting-started/quick-start.md 138 FAIL doesn't contain any tags no
pyenv https://github.com/pyenv/pyenv.git 7d65a4683c91b29f7501da4c8737d4e786f8cad2 README.md 187 DRIFT|FAIL --install no
shellcheck https://github.com/koalaman/shellcheck.git cd9072643932ee2008e454e2cd990e25c1dc9d49 README.md 412 FAIL exec foo; echo "Done!" no
shellcheck https://github.com/koalaman/shellcheck.git cd9072643932ee2008e454e2cd990e25c1dc9d49 README.md 451 FAIL cat file | grep foo no
22 changes: 20 additions & 2 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,19 @@ reports which steps a brand-new user could actually complete.
`directories.bin` are not read.
- Runs each `git clone` recipe too: the clone and the build lines that follow it in the
same code block, with GitHub SSH remotes rewritten to HTTPS for the keyless container.
- Runs a source build when a block configures the checkout (`./configure` or a cmake
generate step) and installs it (`make install`, `ninja install`, or `cmake --install`).
The recipe is the block as written, from its first command through the install and on
to the first blank line, comment, or teardown, with a leading `sudo` dropped. It runs in
`buildpack-deps:stable` on the committed tree, restored from the objects and refs of
the checkout's `.git` directory, which are streamed in so submodules resolve. The git
config, hooks, and reflogs stay on the host, since a config can hold a token. The
installed binary is smoke-tested in its own process. A loader error there, such as a
library `make install` placed where the loader does not look until `ldconfig` runs, is
a `FAIL`, because no argument starts that binary. When the document names `ldconfig` or
`LD_LIBRARY_PATH` outside the block, the same error is `BUILT`: the reader was told, and
kibble ran only the block. Any other failed smoke test stays `BUILT`. A block that also
clones, installs system packages, or pipes a remote script is not a source build.
- Picks the image from the toolchain the step assumes, so a Rust project builds with
`cargo` and a Node project with `npm`, and finds the installed binary wherever that
toolchain puts it, even when it is not named after the package.
Expand Down Expand Up @@ -81,8 +94,13 @@ honestly rather than as if nothing were documented.
When the documents hold no install of the tool itself that kibble extracts, the install
check is a `SKIP` with the reason `no-install-route`. The detail names the README, how
many install documents were read, and the routes seen, such as brew, apt, or none. It is
not a `GAP`, because the documents may describe routes kibble does not run. When every
row is a skip, the run exits 3, so a check that ran nothing does not read as a pass.
not a `GAP`, because the documents may describe routes kibble does not run. When no row
and no example line reached `VERIFIED`, `FAIL`, `GAP`, or `DRIFT`, the run exits 3 and
the summary says nothing was tested and why, so a check that tested nothing does not read
as a pass. A brew formula lookup (`EXISTS`), a recipe with no binary (`RAN`), and a
`TIMEOUT` count as nothing tested: each leaves every documented claim unsettled. Before
this rule only an all-skip run exited 3, and a repository whose rows were a formula lookup
and skipped examples exited 0 having tested nothing.

A package install runs the documented line verbatim, so what kibble verifies is the
command a reader would actually type, flags and all. The bin directory is compared before
Expand Down
8 changes: 8 additions & 0 deletions docs/SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@ memory and process counts, `no-new-privileges`, and every Linux capability dropp
the handful `apt` needs to install the packages the docs depend on. Nothing from the host
is mounted in; the repository is streamed in as a tar of its working tree.

A documented source build (`./configure`, `make`, `make install`) is the one case that
receives more than the working tree. It is sent the objects and refs of the checkout's
`.git` directory, so the container can restore the committed tree and resolve submodules
the way a fresh clone does. That is the repository's local history, including commits and
stashes that were never pushed. Nothing else in `.git` is sent: not `config`, where a CI
checkout stores the token it fetched with and a remote URL can carry a password, not
hooks, not reflogs, and not a submodule's git directory.

The boundary that stays open is the network, because verifying an install *is* fetching
it: `go install`, `cargo install`, and `npm install -g` are network operations. A
malicious documented command can therefore reach out from inside the container.
Expand Down
Loading
Loading