Skip to content

Report untested runs and catch a broken source build - #32

Merged
dcadolph merged 4 commits into
mainfrom
feat/recall
Nov 15, 2026
Merged

dcadolph merged 4 commits into
mainfrom
feat/recall

Conversation

@dcadolph

Copy link
Copy Markdown
Owner

kibble tested nothing on 55 of the 84 repositories in the last two sweeps and said so nowhere. A hand check of ten such repositories found a real documentation bug in four. This is the first part of closing that gap: make silence visible, keep score on the bugs kibble walked past, and catch the first of them.

Exit 3 when nothing was tested

A run exits 3 when no row and no example line reached VERIFIED, FAIL, GAP, or DRIFT. Before this only an all-skip run did, so a repository whose rows were a brew formula lookup, a recipe with no binary to smoke-test, or a timeout exited 0 having tested nothing. The summary gains one line that says nothing was tested and names up to three causes. Exit 1 still wins over exit 3, and JSON output is unchanged.

Recall corpus

A third corpus file beside the pinned repositories and the planted mutations: real bugs found by hand where kibble tested nothing, each pinned to its commit, document, and line. The corpus workflow runs every row. A row marked not yet caught prints its state and never fails the job. A row marked caught fails the job if the catch is lost. It starts with five rows from four repositories, all not yet caught, so the misses stay on the scoreboard until a change earns them.

Stand-in paths to the tool resolve to the tool

A command written as a stand-in path to a binary the session installs, in the forms pyenv's README uses for its own flag, is planned as that binary instead of being skipped as a placeholder. Its flags feed the flag check. A stand-in path to anything else stays a placeholder. Over the 87 pinned clones no plan moves. pyenv itself is still not caught, because its only runnable install leaves the binary off PATH, and that is a separate change.

Source builds

A shell block that configures the checkout (an autotools configure or a cmake generate step) and then installs it (make install, ninja install, or cmake install) is an install route. The recipe is the block as written from its first command through the install and on to the first blank line, comment, or teardown, with a leading sudo dropped. It runs in the Debian build image on the committed tree, and the installed binary is smoke-tested in a process of its own.

A dynamic loader refusing to start that binary is a FAIL, because no argument gets it to run. That is the jq bug: the documented install leaves a binary that cannot find its own library until ldconfig runs. jq at its pinned commit now reads FAIL naming the library, and jq today, whose block ends with ldconfig, reads VERIFIED. When the document names ldconfig or the library path variable outside the block, the same error is BUILT, since the reader was told and kibble ran only the block. Any other failed smoke test stays BUILT. A block that clones, installs system packages, or pipes a remote script is not a source build.

What the container receives

This is the one route that is sent more than the working tree, and it is the part to read closely. The build needs a git repository so submodules resolve, so the container is sent the objects and refs of the checkout's git directory and restores the committed tree into a repository it creates itself. Nothing else from the git directory is sent: not the config, where a CI checkout stores the token it fetched with and a remote URL can carry a password, not hooks, not reflogs, and not a submodule's git directory. The security document says what this means in plain terms, including that local history a user never pushed is part of what is sent.

Verification

  • Unit tests cover each rule with a paired case that must not match. Every new test was run with its rule switched off and failed.
  • The container test builds a small C project whose binary needs its own shared library: FAIL without ldconfig, VERIFIED with it. It plants a token in the host's git config and fails the recipe if the token reaches the container. With the whole git directory sent, that test fails.
  • jq at its pinned commit: FAIL naming the missing library, exit 1.
  • The six corpus pins hold, and the one repository that tests nothing still exits 3.
  • Plans over the 87 pinned clones are unchanged apart from pyenv and jq.
  • Build, vet, race tests, and lint are clean.

Not in this change

Building the tool from the checkout when no documented install provides it, the release-binary route, tutorials that belong in a fresh repository, the installers kibble refuses, and claim checks beyond exit codes each follow separately. The jq recall row flips to caught in the gate that follows this merge.

@dcadolph
dcadolph merged commit 64562d1 into main Nov 15, 2026
3 checks passed
@dcadolph
dcadolph deleted the feat/recall branch November 15, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant