Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,10 +254,20 @@ verdict claims the documented step works:
tool itself that kibble extracts. The detail names the README, how many install
documents were read, and the routes seen. It is not a `GAP`, since the documents may
describe routes kibble does not run, such as apt or brew.
- An example row whose detail opens with "built from the checkout, the documented
install was not tested" ran against the tool kibble built itself, because no
documented install gave the session a binary. kibble builds only a tool it can name:
a Go main package at the root or under `cmd/`, a Rust package's one binary, or a
Python project's one console script, or whichever of these is named for the
repository. The examples' verdicts stand, but a release can differ from the checkout,
and the install rows keep their own verdicts. A checkout that does not build reads
`BLOCKED`, since the failure is kibble's stand-in and not the document.
- `BLOCKED` with the reason `inherited-state` means a line that would have been a plain
`FAIL` ran in a directory an earlier section entered with `cd`, `pushd`, or `popd`.
Documents `cd` once in setup and stay there, so the failure may come from what that
directory holds rather than from what the line says.
directory holds rather than from what the line says. Each collapsed `<details>` element
is a section of its own, since a README offers alternative setups that way under one
heading and a reader opens only one.
- `BLOCKED` with the reason `checkout-state` means a line that would have been a plain
`FAIL` ran in the repository's own checkout and its output says the target already
exists or is already initialized. The collision is with the checkout kibble chose, so
Expand All @@ -266,7 +276,16 @@ verdict claims the documented step works:
already held the other. So does a documented tool saying the top of the checkout is not
a repository of its own kind and naming its own init command, when no earlier line of
the document that passed made or cloned one. That line was not tested, and it is not
counted as passing.
counted as passing. When a scaffold line such as `goreleaser init` is stopped this way,
the same tool's later lines in that document read the checkout's project and not the
one the document creates, so their failures and their documented nonzero exits read
`checkout-state` as well. A usage error or a crash still stands.
- `BLOCKED` with the reason `environment-changed` means a line crashed and afterwards
the documented tool failed `--version`, though it passed when the session began.
Something the session ran changed the environment the tool runs from, as a project
command can remove the tool's own dependencies, so the crash says nothing about the
line. A crash the tool survives stays `FAIL`. A tool whose `--version` fails from the
start is never probed.
- A passing line that writes a titled fence to its file reads `VERIFIED` "wrote X as the
document shows it". The document supplied the content, so only the write is claimed.
- An install whose product is a shell function the documented sourcing defines is
Expand Down
197 changes: 197 additions & 0 deletions checkoutbuild.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
package main

import (
"go/parser"
"go/token"
"os"
"path"
"path/filepath"
"strings"

kplan "github.com/dcadolph/kibble/internal/plan"
)

// checkoutBuild returns an install that builds the repository's own tool from
// the checkout, for a repository whose documented installs give an example
// session nothing to run against. Without it one missing install route cost
// every example in the repository, and on the 11-11 sweeps that silence was
// the largest single reason a repository came back with nothing tested.
//
// It builds only what it can name without guessing: a Go main package at the
// root or under cmd/, a Rust package with one binary or one named for the
// repository, or a Python project with one console script or one named for it.
// When two ecosystems both offer a tool, the one named for the repository wins,
// and with no such name nothing is built, since running the examples against
// the wrong binary would convict a document of someone else's failure.
func checkoutBuild(dir, repo string) (kplan.PlanInstall, bool) {
var found []kplan.PlanInstall
for _, try := range []func(dir, repo string) (kplan.PlanInstall, bool){
goCheckoutBuild, rustCheckoutBuild, pythonCheckoutBuild,
} {
if in, ok := try(dir, repo); ok {
found = append(found, in)
}
}
if len(found) == 1 {
return found[0], true
}
for _, in := range found {
if normalizeName(in.Binary) == normalizeName(repo) {
return in, true
}
}
return kplan.PlanInstall{}, false
}

// goCheckoutBuild builds a Go module's main package with go install, from the
// root when the root is one, else from the cmd/ directory pickBinary selects.
func goCheckoutBuild(dir, repo string) (kplan.PlanInstall, bool) {
body, err := os.ReadFile(filepath.Join(dir, "go.mod"))
if err != nil {
return kplan.PlanInstall{}, false
}
m := reGoModule.FindSubmatch(body)
if m == nil {
return kplan.PlanInstall{}, false
}
base := path.Base(stripMajor(string(m[1])))
if isMainPackage(dir) {
if !reBinName.MatchString(base) {
return kplan.PlanInstall{}, false
}
return kplan.PlanInstall{Cmd: "go install .", Ecosystem: "go", Binary: base, FromCheckout: true}, true
}
ents, err := os.ReadDir(filepath.Join(dir, "cmd"))
if err != nil {
return kplan.PlanInstall{}, false
}
var mains []string
for _, e := range ents {
if e.IsDir() && isMainPackage(filepath.Join(dir, "cmd", e.Name())) {
mains = append(mains, e.Name())
}
}
name, ok := pickBinary(mains, repo, base)
if !ok {
return kplan.PlanInstall{}, false
}
return kplan.PlanInstall{Cmd: "go install ./cmd/" + name, Ecosystem: "go", Binary: name, FromCheckout: true}, true
}

// isMainPackage reports whether dir holds a non-test Go file of package main.
func isMainPackage(dir string) bool {
ents, err := os.ReadDir(dir)
if err != nil {
return false
}
fset := token.NewFileSet()
for _, e := range ents {
name := e.Name()
if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {
continue
}
f, err := parser.ParseFile(fset, filepath.Join(dir, name), nil, parser.PackageClauseOnly)
if err == nil && f.Name.Name == "main" {
return true
}
}
return false
}

// rustCheckoutBuild builds a Rust package's binary with cargo install --path,
// locked when the checkout carries a lock file, as a release build would be. A
// workspace root with no package of its own is left alone.
func rustCheckoutBuild(dir, repo string) (kplan.PlanInstall, bool) {
body, err := os.ReadFile(filepath.Join(dir, "Cargo.toml"))
if err != nil {
return kplan.PlanInstall{}, false
}
pkgs := sectionValues(body, map[string]string{"package": "name"})
if len(pkgs) == 0 {
return kplan.PlanInstall{}, false
}
bins := sectionValues(body, map[string]string{"[bin]": "name"})
if len(bins) == 0 {
if _, err := os.Stat(filepath.Join(dir, "src", "main.rs")); err != nil {
return kplan.PlanInstall{}, false
}
bins = pkgs[:1]
}
name, ok := pickBinary(bins, repo, pkgs[0])
if !ok {
return kplan.PlanInstall{}, false
}
cmd := "cargo install --path . --bin " + name
if _, err := os.Stat(filepath.Join(dir, "Cargo.lock")); err == nil {
cmd += " --locked"
}
return kplan.PlanInstall{Cmd: cmd, Ecosystem: "rust", Binary: name, FromCheckout: true}, true
}

// checkoutVenv is the virtual environment a Python tool built from the
// checkout is installed into.
const checkoutVenv = "/opt/kibble-tool"

// pythonCheckoutBuild installs a Python project from the checkout with pip,
// when its pyproject.toml declares the console script to run. The install goes
// into an environment of its own with the script linked onto PATH, the way pipx
// does it. Installed into the image's Python, the tool shared an environment
// with whatever its examples manage: poetry's `poetry sync --only dev`, run in
// poetry's checkout, removed poetry's own dependencies, and the 39 lines after
// it crashed on a missing module.
func pythonCheckoutBuild(dir, repo string) (kplan.PlanInstall, bool) {
body, err := os.ReadFile(filepath.Join(dir, "pyproject.toml"))
if err != nil {
return kplan.PlanInstall{}, false
}
scripts := sectionValues(body, map[string]string{"project.scripts": "*", "tool.poetry.scripts": "*"})
names := sectionValues(body, map[string]string{"project": "name", "tool.poetry": "name"})
names = append(names, "")
name, ok := pickBinary(scripts, repo, names[0])
if !ok {
return kplan.PlanInstall{}, false
}
cmd := "python3 -m venv " + checkoutVenv + " && " + checkoutVenv + "/bin/pip install --quiet . && " +
"ln -sf " + checkoutVenv + "/bin/" + name + " /usr/local/bin/" + name
return kplan.PlanInstall{Cmd: cmd, Ecosystem: "python", Binary: name, FromCheckout: true}, true
}

// pickBinary returns the candidate named for the repository or its package,
// else the only candidate, else nothing. A name that cannot sit safely in a
// shell command is never returned.
func pickBinary(cands []string, names ...string) (string, bool) {
pick := ""
for _, c := range cands {
for _, n := range names {
if n != "" && normalizeName(c) == normalizeName(n) {
pick = c
break
}
}
if pick != "" {
break
}
}
if pick == "" && len(cands) == 1 {
pick = cands[0]
}
if pick == "" || !reBinName.MatchString(pick) {
return "", false
}
return pick, true
}

// checkoutNote labels every example result run against a tool kibble built
// from the checkout, since a reader installs a release and the two can differ.
const checkoutNote = "built from the checkout, the documented install was not tested"

// fromCheckout reports whether a plan's tool is one kibble built from the
// checkout rather than one a documented install provided.
func fromCheckout(plan *kplan.Plan) bool {
for _, in := range plan.Installs {
if in.FromCheckout {
return true
}
}
return false
}
Loading
Loading