Skip to content

💡 feat: expose Test Engine runs and failed executions in the CLI #965

Description

@elijahchancey

Is your feature request related to a problem?

Test Engine results are not reachable from the CLI. For a pipeline whose failures are mostly
test failures, that is the part of the build you actually want, and it is the one part the CLI
cannot show you.

The gap is visible in what the raw log gives you instead: a test runner's stdout, where
finding which test failed and why means scrolling past everything that passed. Test Engine has
already parsed that into individual executions with names, durations, failure messages and
stack traces.

The official MCP server exposes this — get_build_test_engine_runs, list_tests_for_build,
get_failed_executions (with an expanded form carrying stack traces), get_test_run and
get_test. The CLI has no equivalent for any of them.

Describe the solution you'd like.

Enough surface to answer "which tests failed in this build, and why":

  • bk test-run list -p <pipeline> --build <n> — the Test Engine runs attached to a build
  • bk test-run failures <run-id> — failed executions with names and failure messages,
    --expanded for stack traces
  • participation in --max-tokens, since stack traces are exactly the payload that needs a
    budget

Scoping it to failures first would cover most of the value; full test listing and per-test
history matter less from a terminal.

Describe alternatives you've considered.

Reading the job log and parsing the runner's output. That is what the CLI forces today, and it
is per-framework, brittle, and loses the structure Test Engine already produced.

bk api against the Test Engine endpoints works but means hand-constructing paths and
reimplementing the output handling the rest of the CLI provides.

Additional context

Version bk 3.55.0. Related: #963 (a build failure summary would naturally include failed
test executions).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions