Repository navigation
Report untested runs and catch a broken source build - #32
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kibble tested nothing on 55 of the 84 repositories in the last two sweeps and said so nowhere. A hand check of ten such repositories found a real documentation bug in four. This is the first part of closing that gap: make silence visible, keep score on the bugs kibble walked past, and catch the first of them.
Exit 3 when nothing was tested
A run exits 3 when no row and no example line reached VERIFIED, FAIL, GAP, or DRIFT. Before this only an all-skip run did, so a repository whose rows were a brew formula lookup, a recipe with no binary to smoke-test, or a timeout exited 0 having tested nothing. The summary gains one line that says nothing was tested and names up to three causes. Exit 1 still wins over exit 3, and JSON output is unchanged.
Recall corpus
A third corpus file beside the pinned repositories and the planted mutations: real bugs found by hand where kibble tested nothing, each pinned to its commit, document, and line. The corpus workflow runs every row. A row marked not yet caught prints its state and never fails the job. A row marked caught fails the job if the catch is lost. It starts with five rows from four repositories, all not yet caught, so the misses stay on the scoreboard until a change earns them.
Stand-in paths to the tool resolve to the tool
A command written as a stand-in path to a binary the session installs, in the forms pyenv's README uses for its own flag, is planned as that binary instead of being skipped as a placeholder. Its flags feed the flag check. A stand-in path to anything else stays a placeholder. Over the 87 pinned clones no plan moves. pyenv itself is still not caught, because its only runnable install leaves the binary off PATH, and that is a separate change.
Source builds
A shell block that configures the checkout (an autotools configure or a cmake generate step) and then installs it (make install, ninja install, or cmake install) is an install route. The recipe is the block as written from its first command through the install and on to the first blank line, comment, or teardown, with a leading sudo dropped. It runs in the Debian build image on the committed tree, and the installed binary is smoke-tested in a process of its own.
A dynamic loader refusing to start that binary is a FAIL, because no argument gets it to run. That is the jq bug: the documented install leaves a binary that cannot find its own library until ldconfig runs. jq at its pinned commit now reads FAIL naming the library, and jq today, whose block ends with ldconfig, reads VERIFIED. When the document names ldconfig or the library path variable outside the block, the same error is BUILT, since the reader was told and kibble ran only the block. Any other failed smoke test stays BUILT. A block that clones, installs system packages, or pipes a remote script is not a source build.
What the container receives
This is the one route that is sent more than the working tree, and it is the part to read closely. The build needs a git repository so submodules resolve, so the container is sent the objects and refs of the checkout's git directory and restores the committed tree into a repository it creates itself. Nothing else from the git directory is sent: not the config, where a CI checkout stores the token it fetched with and a remote URL can carry a password, not hooks, not reflogs, and not a submodule's git directory. The security document says what this means in plain terms, including that local history a user never pushed is part of what is sent.
Verification
Not in this change
Building the tool from the checkout when no documented install provides it, the release-binary route, tutorials that belong in a fresh repository, the installers kibble refuses, and claim checks beyond exit codes each follow separately. The jq recall row flips to caught in the gate that follows this merge.