Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions packages/embeddedci-mcp/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## Unreleased

- New tools `cloud_runs_list`, `cloud_run_get`, `cloud_run_compare` and `cloud_run_trace`: list
the test runs on embeddedci.com, read one (the failed cases with their message's first line, the
measurements outside their limits), see what changed since an earlier run, and read a failed
case's pod calls. They need an SDK with `embeddedci.benchpod.runs.api` (newer than 3.0.3).
- New tool `log_level`: read the pod's console log level, or set it with `level` (`error`,
`warn`, `info`, `debug`; `info` is the default, `debug` prints every command trace on the pod's
console). Firmware newer than 3.10.12; older firmware gets a clean unsupported-feature error.
Expand Down
9 changes: 8 additions & 1 deletion packages/embeddedci-mcp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,10 @@ stored wiring profile. `BENCHPOD_API_KEY` wins when set, and
inside GitHub Actions the SDK uses OIDC instead.

`cloud_list_devices` lists the pods on your account and which are online, without connecting to
one; pass a result's `connection` to `connect`. The session token behind a cloud connection is
one; pass a result's `connection` to `connect`. `cloud_runs_list`, `cloud_run_get`,
`cloud_run_compare` and `cloud_run_trace` read the test runs uploaded to embeddedci.com, so an agent
can see why a run failed (the failed cases, the measurements outside their limits, the pod calls of
a failed test) and what changed since an earlier run, without the bench. The session token behind a cloud connection is
renewed before it expires, so a long session keeps working.

A pod is shared, so `connect` takes an exclusive lease, waiting up to `--lease-wait` seconds if
Expand Down Expand Up @@ -192,6 +195,10 @@ description, ranges and enums, so an agent sees more than this table.
| Tool | Purpose | Parameters |
| --- | --- | --- |
| `cloud_list_devices` | The pods on your embeddedci.com organization and whether each is online, without connecting. | none |
| `cloud_runs_list` | Test runs uploaded to embeddedci.com (`benchpod ci`, `pytest --embeddedci-upload`, `build_report`), newest first, with outcome and counts; `before` is the previous page's `next_before`. | `suite`, `dut`, `build`, `pod`, `ref`, `limit=20`, `before` |
| `cloud_run_get` | One test run: outcome, counts, health, pod and tests, the failed cases with the first line of their message, the measurements outside their limits and the uploaded files; `full=true` adds the whole `run.json`. | **`run_id`**, `full=false` |
| `cloud_run_compare` | What changed between an earlier run and this one: newly failing and fixed cases, measurements, slower tests and pod calls, and the inputs that differ (pod firmware, wiring, DUT firmware). `compare_with`: `previous`, a run id, or `baseline:<branch>`. | **`run_id`**, `compare_with="previous"` |
| `cloud_run_trace` | The pod calls of one test case from its uploaded trace (only failed and error cases have one): what was sent, how long it took, the answer or error; long requests and replies are shortened. `case_id` may be a unique end of the id. | **`run_id`**, **`case_id`**, `errors_only=false`, `limit=200` |

### Wiring profile

Expand Down
2 changes: 1 addition & 1 deletion packages/embeddedci-mcp/src/embeddedci_mcp/guide.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
2. The I/O voltage (1.8 or 3.3, the DUT's) comes from the wiring profile's la_mv: connect sets it, so status normally reports one. When status warns that none is selected, the profile sets none: ask the user for the board's I/O voltage and set it in the profile (set_wiring with la_mv, then save_wiring), or call set_la_voltage for this session. The pod refuses flash, UART, LA capture, pull resistors and sensor emulation until it is set.
3. status shows firmware, capabilities, the LA voltage and open sessions.

Cloud pods: cloud_list_devices lists the pods registered on embeddedci.com (name, online) without connecting to any; connect to one with its `connection` ('embeddedci:<name>'). Tools named cloud_* talk to embeddedci.com, not to a pod. Cloud access (cloud_* and connect to 'embeddedci:<name>') authenticates with BENCHPOD_API_KEY, else the user's `benchpod login` session; if neither exists, ask the user to run `benchpod login`.
Cloud pods: cloud_list_devices lists the pods registered on embeddedci.com (name, online) without connecting to any; connect to one with its `connection` ('embeddedci:<name>'). Tools named cloud_* talk to embeddedci.com, not to a pod. Test runs (benchpod ci, pytest --embeddedci-upload, build_report): cloud_runs_list finds them, cloud_run_get shows the failed cases and measurements outside their limits, cloud_run_trace the pod calls of a failed case, cloud_run_compare what changed since an earlier run. Cloud access (cloud_* and connect to 'embeddedci:<name>') authenticates with BENCHPOD_API_KEY, else the user's `benchpod login` session; if neither exists, ask the user to run `benchpod login`.

Wiring: the pod has 14 generic LA channels (LA1-LA14) and nothing is fixed. Call `wiring` first: it is the bench's profile, normally one of the named profiles stored on the pod itself (the same ones the web app shows; `profile_name`, the pod's default unless connect(wiring_profile=...) named another): which DUT signal is on which channel, the I/O voltage, the target-power rail, the UART baud, the SWD target. Every channel, baud, rail or SWD argument you omit comes from it, and channel arguments also accept its names ("READY", "uart_rx"). What the profile does not set is undefined: when a tool says a pin is missing, ask the user which LA channel it is on, never guess. Confirm the wiring with the user if it looks wrong. `set_wiring` changes it for this connection only: nothing is stored until save_wiring (or the server's --wiring-autosave). Once the wiring works on the board, ask the user whether to save it, and under which name when it is a different board (save_wiring(name='nrf52840')). `unsaved_changes` in the result lists what is not stored; it is lost on disconnect. load_wiring switches to another stored profile, set_default_wiring changes the pod's default (ask first). power_on applies the profile first (its I/O voltage, then the rail) and records it on the pod. Refusals carry a stable code (BP0009 ...) with the fix; follow it. Target power is internal 5 V (efuse 1) or external (efuse 2). The DUT's reset line goes to the pod's own reset pin: flash(nreset=true), reset_target.

Expand Down
44 changes: 44 additions & 0 deletions packages/embeddedci-mcp/src/embeddedci_mcp/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -801,6 +801,50 @@ class CloudDeviceList(BaseModel):
devices: List[CloudDevice]


class CloudRunList(BaseModel):
auth: Literal["api_key", "benchpod_login"]
runs: List[Dict[str, Any]] = Field(description=(
"Newest first: id, url, suite, dut, outcome, valid, counts, started, duration_ms, build_id, "
"device_name, tests_ref, tests_sha, firmware, github_run_url."))
next_before: Optional[str] = Field(None, description=(
"More runs exist: pass this as `before` for the next page."))
org_pods: Optional[int] = Field(None, description="How many pods the organization has.")


class CloudRun(BaseModel):
auth: Literal["api_key", "benchpod_login"]
summary: Dict[str, Any] = Field(description=(
"id, url, suite, dut, outcome, counts, valid, health_reasons, pod, tests, build_id, "
"failed_cases (id, outcome, the message's first line, whether a trace was uploaded) and "
"measurements_out_of_limits."))
files: List[Dict[str, Any]] = Field(default_factory=list, description=(
"The uploaded files (name, size_bytes, expired): junit.xml, the traces of failed cases."))
run: Optional[Dict[str, Any]] = Field(None, description="The full run.json, with full=true.")


class CloudRunCompare(BaseModel):
auth: Literal["api_key", "benchpod_login"]
found: bool = Field(description="False when there is no earlier run to compare with.")
message: Optional[str] = None
diff: Dict[str, Any] = Field(default_factory=dict, description=(
"a (the earlier run), b (this one), comparable, reasons, flags, inputs_changed, cases "
"(newly_failing, fixed, still_failing, new, removed, newly_skipped), measurements, slower, "
"calls_slower, a_run (the earlier run's summary)."))


class CloudRunTrace(BaseModel):
auth: Literal["api_key", "benchpod_login"]
case: str
trace: str = Field(description="The trace file's name in the run.")
lines: List[Dict[str, Any]] = Field(description=(
"One object per pod call: t (s since the run started), op, cmd, req, ms, ok, resp or "
"error; long req/resp are shortened."))
total: int
errors: int = Field(description="Calls that failed (ok false).")
shown: int
truncated: bool = Field(description="More lines exist than `limit` returned.")


class RecordingResult(BaseModel):
id: str
name: str
Expand Down
101 changes: 101 additions & 0 deletions packages/embeddedci-mcp/src/embeddedci_mcp/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -2611,6 +2611,107 @@ def op() -> m.CloudDeviceList:
return await _call(op, lock=False)


def _runs_api() -> Tuple[Any, str]:
"""The SDK's runs API over the same credentials as the other cloud_* tools."""
try:
from embeddedci.benchpod.runs.api import RunsApi
except ImportError as exc: # an SDK older than the runs API
raise ToolError("reading test runs needs a newer embeddedci SDK (embeddedci.benchpod.runs.api); "
"upgrade the embeddedci package") from exc
server, auth = _cloud_api()
return RunsApi(server), auth


def _lookup(fn: Callable[[], T]) -> Callable[[], T]:
def op() -> T:
try:
return fn()
except LookupError as exc:
raise ToolError(str(exc)) from exc
return op


@mcp.tool(annotations=_ann("List test runs", read_only=True, cloud=True))
async def cloud_runs_list(
suite: Annotated[Optional[str], Field(description="Only this suite (the run's key.suite).")] = None,
dut: Annotated[Optional[str], Field(description="Only this board under test ('none' for pod-only suites).")] = None,
build: Annotated[Optional[str], Field(description="Only the runs linked to this build id.")] = None,
pod: Annotated[Optional[str], Field(description="Only the runs on this pod (name or id).")] = None,
ref: Annotated[Optional[str], Field(description="Only the runs of this git branch or ref of the tests.")] = None,
limit: Annotated[int, Field(ge=1, le=200, description="How many runs, newest first.")] = 20,
before: Annotated[Optional[str], Field(description="The next_before of the previous page.")] = None,
) -> m.CloudRunList:
"""Test runs uploaded to embeddedci.com (benchpod ci, pytest --embeddedci-upload, build_report),
newest first, with their outcome and counts. Needs no pod connection; authenticates like
cloud_list_devices."""
def op() -> m.CloudRunList:
api, auth = _runs_api()
data = api.list_runs(suite=suite, dut=dut, build=build, pod=pod, ref=ref, limit=limit, before=before)
return m.CloudRunList(auth=auth, runs=list(data.get("runs") or []),
next_before=data.get("next_before"), org_pods=data.get("org_pods"))

return await _call(op, lock=False)


@mcp.tool(annotations=_ann("Read a test run", read_only=True, cloud=True))
async def cloud_run_get(
run_id: str,
full: Annotated[bool, Field(description="Also return the whole run.json (every case, inputs, calls).")] = False,
) -> m.CloudRun:
"""One test run: outcome, counts, health, the pod and tests it ran with, the failed cases with
the first line of their message, and the measurements outside their limits. cloud_run_trace
reads a failed case's pod calls; cloud_run_compare shows what changed since an earlier run."""
def op() -> m.CloudRun:
from embeddedci.benchpod.runs.api import summarize_run

api, auth = _runs_api()
doc = api.get_run(run_id)
return m.CloudRun(auth=auth, summary=summarize_run(doc), files=list(doc.get("files") or []),
run=doc if full else None)

return await _call(op, lock=False)


@mcp.tool(annotations=_ann("Compare test runs", read_only=True, cloud=True))
async def cloud_run_compare(
run_id: str,
compare_with: Annotated[str, Field(description=(
"'previous' (the newest earlier valid run of the same suite and DUT), a run id, or "
"'baseline:<branch>' (the newest valid run on that branch)."))] = "previous",
) -> m.CloudRunCompare:
"""What changed between an earlier run (a) and this one (b): newly failing and fixed cases,
measurements that moved or broke a limit, slower tests and pod calls, and the inputs that differ
(pod firmware, wiring, DUT firmware), so the likely cause sits next to the change."""
def op() -> m.CloudRunCompare:
api, auth = _runs_api()
data = api.compare_runs(run_id, compare_with)
found = bool(data.get("found", True))
return m.CloudRunCompare(auth=auth, found=found, message=data.get("message"),
diff={k: v for k, v in data.items() if k not in ("found", "message")})

return await _call(op, lock=False)


@mcp.tool(annotations=_ann("Read a test case's pod calls", read_only=True, cloud=True))
async def cloud_run_trace(
run_id: str,
case_id: Annotated[str, Field(description=(
"The case id (pytest node id), or a unique end of it ('test_ack')."))],
errors_only: Annotated[bool, Field(description="Only the calls that failed.")] = False,
limit: Annotated[int, Field(ge=1, le=2000, description="How many lines, from the start.")] = 200,
) -> m.CloudRunTrace:
"""Every call a test case made to the pod (what it sent, how long it took, what the pod
answered), from the trace uploaded with the run. Only failed and error cases have one."""
def op() -> m.CloudRunTrace:
api, auth = _runs_api()
t = api.case_trace(run_id, case_id, errors_only=errors_only, limit=limit)
return m.CloudRunTrace(auth=auth, case=t["case"], trace=t["trace"], lines=t["lines"],
total=t["total"], errors=t["errors"], shown=t["shown"],
truncated=t["truncated"])

return await _call(_lookup(op), lock=False)


@mcp.tool(annotations=_ann("Replay a library waveform", destructive=True, cloud=True))
async def replay_waveform(
waveform_id: str,
Expand Down
68 changes: 68 additions & 0 deletions packages/embeddedci-mcp/tests/test_run_tools.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
"""cloud_runs_list, cloud_run_get, cloud_run_compare and cloud_run_trace against a fake runs API
(the SDK tests' fake, urlopen patched: no network)."""

from __future__ import annotations

import importlib.util
import urllib.request
from pathlib import Path

import pytest
from mcp.server.fastmcp.exceptions import ToolError

from conftest import call

pytest.importorskip("embeddedci.benchpod.runs.api", reason="an SDK without the runs API")

_FAKE = Path(__file__).resolve().parents[2] / "embeddedci" / "tests" / "runs_api_fake.py"
_spec = importlib.util.spec_from_file_location("runs_api_fake_for_mcp", _FAKE)
fake_mod = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(fake_mod) # type: ignore[union-attr]


@pytest.fixture
def server(monkeypatch):
fake = fake_mod.FakeRunsServer()
monkeypatch.setattr(urllib.request, "urlopen", fake.urlopen)
monkeypatch.setenv("BENCHPOD_API_KEY", "eci_test")
monkeypatch.delenv("BENCHPOD_API_BASE", raising=False)
return fake


def test_runs_list(server):
out = call("cloud_runs_list", suite="hil", pod="benchpod-abc123", limit=5)
assert out["auth"] == "api_key" and out["runs"][0]["id"] == "r-2"
assert out["next_before"] and out["org_pods"] == 2
assert server.requests[0]["query"] == {"suite": "hil", "device": "benchpod-abc123", "limit": "5"}
assert server.requests[0]["auth"] == "ApiKey eci_test"


def test_run_get(server):
out = call("cloud_run_get", run_id="r-2")
s = out["summary"]
assert s["outcome"] == "failed" and s["failed_cases"][0]["message"] == "ValueError: bad CRC: 6f72"
assert s["measurements_out_of_limits"][0]["name"] == "settle_ms"
assert out["run"] is None and [f["name"] for f in out["files"]][0] == "junit.xml"
assert call("cloud_run_get", run_id="r-2", full=True)["run"]["results"]["cases"]


def test_run_compare(server):
out = call("cloud_run_compare", run_id="r-2")
assert out["found"] and out["diff"]["cases"]["newly_failing"] == ["tests/test_can.py::test_ack"]
assert server.requests[-1]["query"] == {"with": "previous"}
none = call("cloud_run_compare", run_id="r-2", compare_with="baseline:none")
assert none["found"] is False and "no earlier" in none["message"]
assert server.requests[-1]["query"] == {"with": "baseline:none"}


def test_run_trace(server):
out = call("cloud_run_trace", run_id="r-2", case_id="test_ack", errors_only=True)
assert out["case"] == "tests/test_can.py::test_ack" and out["total"] == 3
assert [ln["cmd"] for ln in out["lines"]] == ["can_write"] and out["lines"][0]["ok"] is False
with pytest.raises(ToolError, match="no trace was uploaded"):
call("cloud_run_trace", run_id="r-2", case_id="test_ok")


def test_unknown_run(server):
with pytest.raises(ToolError, match="run not found"):
call("cloud_run_get", run_id="nope")
4 changes: 4 additions & 0 deletions packages/embeddedci-mcp/tests/test_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@
"can_open", "can_write", "can_read", "can_respond", "can_status", "can_close",
"la_step", "command",
"cloud_list_devices",
"cloud_runs_list",
"cloud_run_get",
"cloud_run_compare",
"cloud_run_trace",
"vision_cloud_boards", "vision_connect", "vision_disconnect", "vision_status", "vision_look", "vision_ocr", "vision_find",
"vision_wait_for", "vision_compare", "vision_screen_health", "vision_calibrate", "vision_add_reference",
"vision_press", "vision_tap",
Expand Down
Loading