Repository navigation
MCP: list, read and compare test runs (phase 4) - #124
Merged
Merged
Conversation
RunsApi over a ServerApi: list_runs (suite, dut, build, pod, ref, limit, before), get_run, compare_runs (previous, a run id, baseline:<branch>), list_run_files, get_run_file and case_trace; summarize_run, find_case and trace_lines shape the documents. Module-level shortcuts use BENCHPOD_API_KEY, else the benchpod login session, else GitHub Actions OIDC. Tests against a fake runs API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read the test runs on embeddedci.com through the SDK's runs API, with the cloud_* tools' credentials: list runs, a run's summary (failed cases with the message's first line, measurements outside limits, files; full=true for the whole run.json), the diff against an earlier run (compare_with: previous, a run id, baseline:<branch>), and a failed case's pod calls. An SDK without the runs API gets a clear tool error. Tool snapshot, README table, guide and CHANGELOG updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
embeddedci.benchpod.runs.api.RunsApi(server=None)over aServerApi:list_runs(suite, dut, build, pod, ref, limit=20, before)(GET /api/runs;podmaps todevice, limit capped at 200),get_run(run_id),compare_runs(run_id, with_="previous")(previous, a run id, orbaseline:<branch>),list_run_files,get_run_file(run_id, name)andcase_trace(run, case_id, errors_only=False, limit=200). Helpers:summarize_run(outcome, counts, health, pod, tests, failed cases with the message's first line, measurements outside limits),find_case(exact id or a unique end of it),trace_lines(NDJSON to objects, long req/resp shortened). Module-level shortcuts useBENCHPOD_API_KEY, else thebenchpod loginsession, else GitHub Actions OIDC.cloud_list_devices, same credentials):cloud_runs_list(suite, dut, build, pod, ref, limit=20, before)cloud_run_get(run_id, full=false): summary + uploaded files, the whole run.json withfullcloud_run_compare(run_id, compare_with="previous"):withis a Python keyword, so the parameter iscompare_withcloud_run_trace(run_id, case_id, errors_only=false, limit=200)An SDK without the runs API gets a clear tool error (the tools are registered either way; their tests skip on the minimum SDK).
Tests
packages/embeddedci/tests/test_runs_api.pyandpackages/embeddedci-mcp/tests/test_run_tools.pyagainst a fake runs API (urlopen patched; shared fake intests/runs_api_fake.py): filters and auth header, get/compare/files, not found, summary contents, trace trimming, errors only, limit and truncation, no trace for a passed case, ambiguous and missing case ids. SDK, MCP and scripts suites pass. Locallytest_bundled_copy_matches_the_firmwarefails because firmware main moved to features revision 7 (see below); it skips in CI.The bundled features table and error catalog are not synced in this PR: firmware revisions 6 and 7 (
la_5v,la_pull_strength, factvref_out, BP0028/BP0029) need SDK code (set_la_voltage(5.0), pull strength), not just data.🤖 Generated with Claude Code