Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ Thanks for helping out. Issues and pull requests are welcome.
make test PYTHON=.venv/bin/python
```

- If you have a BenchPod, run the hardware tier too: `make e2e POD=<pod host>`.
- If you have a BenchPod, run the hardware tier too: `make e2e POD=<pod host>`, or record it as a
test run with `cd packages/embeddedci/tests && benchpod ci --pod <your pod>`
(`packages/embeddedci/tests/benchpod-ci.yaml`; see `packages/embeddedci/tests/e2e/README.md`).
- The 2.x SDK API and MCP tool surface are frozen (snapshot tests fail on any change). Additions
are fine; renames and removals wait for a major version.
- Keep commits small and focused, and describe what changed and why.
Expand Down
17 changes: 17 additions & 0 deletions packages/embeddedci/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,23 @@
`bp.set_log_level(level)` sets it (`LOG_LEVELS`: `error`, `warn`, `info`, `debug`; default
`info`, stored on the pod). Firmware newer than 3.10.12 (feature `log_level`); older firmware
raises `UnsupportedFeatureError`.
- **Upload test runs:** `--embeddedci-upload` (also the `embeddedci_upload` ini option and
`EMBEDDEDCI_UPLOAD=1`) uploads the recorded run to embeddedci.com when the session ends:
`run.json`, `junit.xml` and the traces of the failed and error tests. Credentials:
`BENCHPOD_API_KEY`, else the `benchpod login` session, else GitHub Actions OIDC. A failed upload
is a warning. Not under `benchpod ci`, which uploads the run itself.
`embeddedci.benchpod.runs.upload.upload_run(folder)` uploads a folder later.
- **`build_report` on test runs:** in GitHub Actions a session whose tests use `build_report`
records its run and uploads it linked to the build the fixture created, so the build shows every
test's result. The fixture still sends the pass/fail to the build for older servers.
`BuildReporter.api_base` and `BuildReporter.session_token` are new.
- Run folder file names always pass the server's rule (`[A-Za-z0-9._-]`, no leading dot): a case
id starting with `.` gets a trace name without it, and attachment names keep ASCII letters only.
- `tests/benchpod-ci.yaml` runs this repository's hardware tests with `benchpod ci`.
- Under `benchpod ci`, a stop of the runner's lease from the web app (`BP0026` on a lease joined
through `BENCHPOD_LEASE`) ends the pytest session after the test that saw it (exit code 2); the
step's `parts/<step>.json` still has the cases so far, `lease_lost` and the reason. With the SDK's
own lease the session goes on without it, as before.
- Test run recording: a failed case's `message` starts with the crash line (`ValueError: bad CRC:
6f72`), then a blank line and pytest's full report, still capped at 16 KB.
- **Pod lease on the LAN and the USB network:** on a pod with the `lease` feature (features table
Expand Down
37 changes: 35 additions & 2 deletions packages/embeddedci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -508,6 +508,7 @@ The `benchpod` fixture is a `BenchPod` instance, not the module — import const
| `--embeddedci-suite` | — | `pytest` | suite name recorded in `run.json` |
| `--embeddedci-dut` | — | `""` | name of the board under test recorded in `run.json` |
| `--embeddedci-keep-traces` | `EMBEDDEDCI_KEEP_TRACES` | `all` | which pod-call traces to keep: `all`, `failed` or `none` |
| `--embeddedci-upload` | `EMBEDDEDCI_UPLOAD` | off | upload the recorded run to embeddedci.com at the end (also the `embeddedci_upload` ini option), see [Uploading a run](#uploading-a-run) |

Connection resolution order: `--benchpod-connection` → `benchpod_connection` ini option →
`BENCHPOD_CONNECTION` → `--benchpod-discover`.
Expand Down Expand Up @@ -624,6 +625,25 @@ test passes to pytest's `record_property` are recorded too (with source `record_
values are left out). A test keeps at most 200 measurements; the rest are dropped and counted in
`health.measurements_dropped`.

**Uploading a run.** `--embeddedci-upload` (or `embeddedci_upload = true` in the ini file, or
`EMBEDDEDCI_UPLOAD=1`) uploads the run to embeddedci.com when the session ends, where it shows
next to the build it tested and can be compared with earlier runs. Without `--embeddedci-run-dir` it
records into `./embeddedci-runs/<started UTC>-<suite>/` first. What goes up: `run.json`,
`junit.xml`, and the traces (with their attachments) of the failed and error tests only; the
folder on disk keeps what `--embeddedci-keep-traces` says. Credentials: `BENCHPOD_API_KEY`, else
the `benchpod login` session, else the workflow's OIDC token in GitHub Actions (`id-token: write`);
`--benchpod-api-base` / `BENCHPOD_API_BASE` picks the server. The run's URL is printed at the end.
A failed upload is a warning: it never changes the test result. Under `benchpod ci` the plugin does
not upload; the runner uploads the whole run.

The upload can be done later, or from another machine, with the same code:

```python
from embeddedci.benchpod.runs.upload import upload_run

print(upload_run("embeddedci-runs/20261118T140211Z-hil")["url"])
```

**Inside `benchpod ci`.** The `benchpod ci` runner hands its run folder to the `pytest` steps it
starts through the environment; the plugin then writes `parts/<step>.json` (the cases, health
counters, tool versions and pod) and its traces into that folder, and the runner merges them into
Expand Down Expand Up @@ -2299,7 +2319,9 @@ the lease length.
starts), `BenchPod` joins the lease with that id instead of taking its own: on the LAN and the USB
network it becomes an owner of the pod's lease, over the cloud it uses the id for the server's
lease. Either way it never renews or releases it; the runner does. On a pod without the `lease`
feature it logs one warning and runs without a lease.
feature it logs one warning and runs without a lease. When the runner's lease is stopped from the
web app, the pytest plugin ends the session after the test that saw it (exit code 2, "run stopped
by ..."); the cases so far, `health.lease_lost` and the reason still go to the run folder.

Over the cloud, ordinary commands go through the server's command channel rather than the byte
tunnel, so they keep working while a UART session holds the tunnel.
Expand Down Expand Up @@ -2334,7 +2356,18 @@ def test_boots(benchpod, firmware, build_report):

Reporting is active only inside GitHub Actions with a mintable session token (`id-token: write`,
repo trusted in the web app). Everywhere else the fixture yields an inert reporter
(`build_report.active` is false), so the same test runs unchanged. `--benchpod-build-target` /
(`build_report.active` is false), so the same test runs unchanged.

**Test results on the build.** When `build_report` reports, the session's tests are also recorded
as a [test run](#test-run-recording) and uploaded at the end, linked to the build the fixture
created (`inputs.dut_firmware.build_id`): every test's outcome, duration, measurements and pod-call
timings, and the traces of the failed tests. The build page shows them under Tests. The fixture
still sends the pass/fail with the build, so older servers keep showing it; current servers keep
the build's own status for the build and show the tests from the run, so a failed test no longer
turns the build red. With several tests using the fixture, the run is linked to the first build. The run is uploaded
with the reporter's own credentials (`build_report.api_base`, `build_report.session_token`).
Recording starts by itself in GitHub Actions when a collected test uses `build_report`; pass
`--embeddedci-run-dir` to choose the folder. `--benchpod-build-target` /
`BENCHPOD_BUILD_TARGET` records a platform id. Outside pytest, `make_build_reporter()` returns the
same `BuildReporter` / `NoopBuildReporter`.

Expand Down
15 changes: 14 additions & 1 deletion packages/embeddedci/src/embeddedci/benchpod/ci.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,16 @@ def active(self) -> bool:
def build_id(self) -> Optional[str]:
return self._build_id

@property
def api_base(self) -> str:
"""The embeddedci server this reporter talks to."""
return self._api_base

@property
def session_token(self) -> str:
"""The session token the reporter uses (the test run is uploaded with it too)."""
return self._token

# -- public API used by tests -------------------------------------------------

def record_wiring(
Expand Down Expand Up @@ -169,7 +179,10 @@ def set_result(self, success: bool, reason: str = "") -> None:
self._result = (bool(success), reason)

def finalize(self) -> None:
"""Create the build if needed and post the final pytest status. Idempotent."""
"""Create the build if needed and post the final pytest status. Idempotent.

The status call is kept for servers that predate test runs; newer servers keep the build's
own status and show the test results from the run linked to the build instead."""
if self._finalized:
return
self._finalized = True
Expand Down
3 changes: 3 additions & 0 deletions packages/embeddedci/src/embeddedci/benchpod/pod_lease.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,9 @@ def _mark_lost(self, by: str) -> bool:
run = active_run()
if run is not None:
run.lease_lost(by)
if self.joined:
# The runner's lease (benchpod ci) was stopped: the whole run ends.
run.request_stop(f"run stopped by {by or 'someone'}: the pod's lease was stopped")
except Exception: # noqa: BLE001 - never fail the session over the run record
pass
return True
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -533,6 +533,10 @@ def test_boots(benchpod, firmware, build_report):
... # the pytest pass/fail is captured automatically

The build status (pass/fail) is captured automatically from the test result at teardown.

The session's test run is recorded too and uploaded at the end, linked to the build: every
test's outcome, measurements and pod-call timings, and the traces of the failed tests (see
``--embeddedci-upload``). The build page shows it under Tests.
"""
from .ci import make_build_reporter

Expand All @@ -556,6 +560,11 @@ def test_boots(benchpod, firmware, build_report):
if log_text:
reporter.upload_logs("pytest.log", log_text)
reporter.finalize()
# The session's test run is uploaded at the end, linked to this build.
runs = request.config.pluginmanager.get_plugin(_runs.PLUGIN_NAME)
build_id = getattr(reporter, "build_id", None)
if runs is not None and getattr(reporter, "active", False) and build_id:
runs.link_build(build_id, reporter.api_base, reporter.session_token)


def _collect_pytest_log(node: "pytest.Item") -> str:
Expand Down
18 changes: 17 additions & 1 deletion packages/embeddedci/src/embeddedci/benchpod/runs/active.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,16 @@ def default_run_dir(started: float, suite: str = "", base: Union[str, "os.PathLi
return Path(base) / "embeddedci-runs" / f"{folder_stamp(started)}-{case_slug_simple(suite or 'pytest')}"


_SIMPLE_OK = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._-")


def case_slug_simple(text: str) -> str:
return "".join(c if c.isalnum() or c in "._-" else "_" for c in text)[:80] or "pytest"
"""``text`` as a short file name part: ASCII ``[A-Za-z0-9._-]`` only (the server's file name
rule), every other character ``_``, no leading dot, at most 80 characters."""
out = "".join(c if c in _SIMPLE_OK else "_" for c in text)[:80]
if out.startswith("."):
out = "_" + out[1:]
return out or "pytest"


@dataclass
Expand Down Expand Up @@ -140,6 +148,8 @@ def __init__(self, run_dir: Union[str, "os.PathLike[str]"], *, step: str = "pyte
self.handover = handover
#: The suite name (``key.suite``); a pod lease taken during the run uses it as purpose.
self.suite = ""
#: Why the run must end now (the runner's lease was stopped), or "".
self.stop_reason = ""
self.started = time.time()
self.t0 = float(t0) if t0 is not None else self.started
self.run_id = run_id or new_run_id(self.started)
Expand Down Expand Up @@ -181,6 +191,12 @@ def lease_lost(self, by: str = "") -> None:
self.health["lease_lost"] = True
self.add_reason(f"the pod lease was stopped by {by or 'someone'} during the run")

def request_stop(self, reason: str) -> None:
"""End the run after the current test (the pytest plugin exits the session)."""
with self._lock:
if not self.stop_reason:
self.stop_reason = reason

def add_reason(self, reason: str, *, invalid: bool = True) -> None:
"""A note on the run's health; ``invalid`` (default) marks the run as not a valid measurement."""
with self._lock:
Expand Down
Loading