Skip to content

MCP: list, read and compare test runs (phase 4) - #124

Merged
wardviaene merged 3 commits into
mainfrom
ci-mcp-runs
Nov 19, 2026
Merged

wardviaene merged 3 commits into
mainfrom
ci-mcp-runs

Conversation

@wardviaene

Copy link
Copy Markdown
Member

What

  • SDK: embeddedci.benchpod.runs.api. RunsApi(server=None) over a ServerApi: list_runs(suite, dut, build, pod, ref, limit=20, before) (GET /api/runs; pod maps to device, limit capped at 200), get_run(run_id), compare_runs(run_id, with_="previous") (previous, a run id, or baseline:<branch>), list_run_files, get_run_file(run_id, name) and case_trace(run, case_id, errors_only=False, limit=200). Helpers: summarize_run (outcome, counts, health, pod, tests, failed cases with the message's first line, measurements outside limits), find_case (exact id or a unique end of it), trace_lines (NDJSON to objects, long req/resp shortened). Module-level shortcuts use BENCHPOD_API_KEY, else the benchpod login session, else GitHub Actions OIDC.
  • MCP tools (cloud_* like cloud_list_devices, same credentials):
    • cloud_runs_list(suite, dut, build, pod, ref, limit=20, before)
    • cloud_run_get(run_id, full=false): summary + uploaded files, the whole run.json with full
    • cloud_run_compare(run_id, compare_with="previous"): with is a Python keyword, so the parameter is compare_with
    • cloud_run_trace(run_id, case_id, errors_only=false, limit=200)
      An SDK without the runs API gets a clear tool error (the tools are registered either way; their tests skip on the minimum SDK).
  • MCP tool snapshot, README tool table and overview, guide (server instructions), both CHANGELOGs; SDK README "Reading runs back".

Tests

packages/embeddedci/tests/test_runs_api.py and packages/embeddedci-mcp/tests/test_run_tools.py against a fake runs API (urlopen patched; shared fake in tests/runs_api_fake.py): filters and auth header, get/compare/files, not found, summary contents, trace trimming, errors only, limit and truncation, no trace for a passed case, ambiguous and missing case ids. SDK, MCP and scripts suites pass. Locally test_bundled_copy_matches_the_firmware fails because firmware main moved to features revision 7 (see below); it skips in CI.

The bundled features table and error catalog are not synced in this PR: firmware revisions 6 and 7 (la_5v, la_pull_strength, fact vref_out, BP0028/BP0029) need SDK code (set_la_voltage(5.0), pull strength), not just data.

🤖 Generated with Claude Code

wardviaene and others added 3 commits November 19, 2026 06:43
RunsApi over a ServerApi: list_runs (suite, dut, build, pod, ref, limit,
before), get_run, compare_runs (previous, a run id, baseline:<branch>),
list_run_files, get_run_file and case_trace; summarize_run, find_case and
trace_lines shape the documents. Module-level shortcuts use
BENCHPOD_API_KEY, else the benchpod login session, else GitHub Actions
OIDC. Tests against a fake runs API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read the test runs on embeddedci.com through the SDK's runs API, with the
cloud_* tools' credentials: list runs, a run's summary (failed cases with
the message's first line, measurements outside limits, files; full=true
for the whole run.json), the diff against an earlier run (compare_with:
previous, a run id, baseline:<branch>), and a failed case's pod calls.
An SDK without the runs API gets a clear tool error. Tool snapshot,
README table, guide and CHANGELOG updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wardviaene
wardviaene merged commit f981552 into main Nov 19, 2026
20 checks passed
@wardviaene wardviaene mentioned this pull request Nov 19, 2026
wardviaene added a commit that referenced this pull request Nov 19, 2026
Test run recording (#118), pod lease on every transport (#119), run upload
and build_report on runs (#122), console log level (#121), runs API (#124).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@wardviaene
wardviaene deleted the ci-mcp-runs branch November 20, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant