diff --git a/packages/embeddedci/CHANGELOG.md b/packages/embeddedci/CHANGELOG.md index 9da753e..b9cc987 100644 --- a/packages/embeddedci/CHANGELOG.md +++ b/packages/embeddedci/CHANGELOG.md @@ -2,6 +2,16 @@ ## Unreleased +- **Test run recording:** `pytest --embeddedci-run-dir DIR` (or `EMBEDDEDCI_RUN_DIR`) writes a run + folder: `run.json` (schema version 1: outcome, counts, every test case with its measurements and + per-command call timings, and the firmware, tests, pod, wiring and tool versions it ran with), + `junit.xml`, and a trace of every pod call per test (`trace/`, large payloads in `att/`, secrets + redacted, 50 MB per run). `--embeddedci-suite`, `--embeddedci-dut` and `--embeddedci-keep-traces` + (`all`, `failed`, `none`) set the rest. Off by default. Under `benchpod ci` (`EMBEDDEDCI_RUN_DIR` + with `EMBEDDEDCI_RUN_STEP`) the plugin writes `parts/.json` for the runner to merge. +- **`measure` fixture:** `measure(name, value, unit="", min=None, max=None, drift=None)` records a + number for the test; a value outside `min`/`max` fails it. Numeric `record_property` values are + recorded as measurements too. - USB console: a firmware log line printed right after the prompt (`[sig] SPI1 link at ... for gateware 58`, on every `status` up to firmware 3.10.12) no longer makes the serial transport time out waiting for the prompt. A prompt followed only by complete `[tag] ...` lines ends the diff --git a/packages/embeddedci/README.md b/packages/embeddedci/README.md index 3a72d5a..be36305 100644 --- a/packages/embeddedci/README.md +++ b/packages/embeddedci/README.md @@ -32,7 +32,7 @@ No pin numbers: the SWD channels, the UART pair, the baud and the power rail com [wiring profile](#wiring-profile). Pass them explicitly when you want to override it. **Contents:** [Install](#install) · [Quick start](#quick-start) · -[API conventions](#api-conventions-and-stability) · [pytest plugin](#pytest-plugin) · +[API conventions](#api-conventions-and-stability) · [pytest plugin](#pytest-plugin) · [Test runs](#test-run-recording) · [Power and reset](#power-reset-and-power-monitoring) · [Flashing](#flashing) · [UART](#uart) · [Pull resistors](#la-pull-resistors) · [Emulated sensors](#emulated-sensors) · [Analog paths](#analog-paths-dc-output-and-single-readings) · [Captures](#captures) · @@ -487,6 +487,10 @@ The `benchpod` fixture is a `BenchPod` instance, not the module — import const | `--benchpod-no-lease` | — | off | don't lock the cloud device (only when nothing else can use it) | | `--benchpod-build-target` | `BENCHPOD_BUILD_TARGET` | — | platform id recorded by `build_report` (e.g. `stm32f4`) | | `--benchpod-sensor-library` | `BENCHPOD_SENSOR_LIBRARY` | the pinned version | sensor library version for `benchpod_sensors` (`latest` or a release) | +| `--embeddedci-run-dir` | `EMBEDDEDCI_RUN_DIR` | off | record the run into this folder (`auto`: `./embeddedci-runs/-/`), see [Test run recording](#test-run-recording) | +| `--embeddedci-suite` | — | `pytest` | suite name recorded in `run.json` | +| `--embeddedci-dut` | — | `""` | name of the board under test recorded in `run.json` | +| `--embeddedci-keep-traces` | `EMBEDDEDCI_KEEP_TRACES` | `all` | which pod-call traces to keep: `all`, `failed` or `none` | Connection resolution order: `--benchpod-connection` → `benchpod_connection` ini option → `BENCHPOD_CONNECTION` → `--benchpod-discover`. @@ -518,6 +522,7 @@ benchpod_connection = 192.168.1.213 | `benchpod_pins` / `pins` | session | `pin_1` … `pin_14`, `efuse`, and `has_pullup()`, `has_pulldown()`, `pull_ohms()`, `pullup_ohms()`, `pull_direction()` | | `firmware` | function | the `--benchpod-firmware` path (skips when unset) | | `build_report` | function | a build reporter — see [Build reporting](#build-reporting) | +| `measure` | function | `measure(name, value, unit="", min=None, max=None, drift=None)`: records a number for the test; a value outside `min`/`max` fails it, see [Test run recording](#test-run-recording) | `benchpod` is shared by the whole session, so state a test leaves behind (a running DAC, engaged pull-ups, a running sensor) carries into the next one — use the teardown fixtures. @@ -542,6 +547,79 @@ nothing changes. firmware that does not announce the flag ran on analog boards only and counts as analog, like `BenchPod.can_analog`. +### Test run recording + +With `--embeddedci-run-dir DIR` the plugin records the whole run into one folder: the outcome of +every test, the numbers tests measured, and every call each test made to a pod, with the +provenance needed to compare two runs (the firmware file's hash, the tests' git commit, the pod's +firmware, gateware and calibration, the wiring profile, tool versions). It needs no account and +writes nothing anywhere else. Recording is off unless the option or `EMBEDDEDCI_RUN_DIR` is set, so +an existing suite writes nothing new. + +```bash +pytest --embeddedci-run-dir=auto --embeddedci-suite=process-node-hil --embeddedci-dut=process-node \ + --embeddedci-keep-traces=failed +``` + +``` +/ + run.json the record (schema_version 1), written last + junit.xml the same tests as JUnit XML, for CI test reporters + trace/.ndjson the pod calls of one test, one JSON object per line + att/-.json large requests and replies a trace line points to (.json.zst with zstandard) +``` + +`run.json` holds `key` (suite, dut), `inputs` (`dut_firmware` from `--benchpod-firmware`, `tests` +from git, `pod`, `wiring`, `tools`, `github`), `health`, and `results`: the run's `outcome` +(`passed`, `failed`, `error`, `invalid`), `counts` and one entry per test in `cases`. A case has its +`id` (the pytest node id), `outcome` (`passed`, `failed`, `error` for a failure in setup or +teardown, `skipped`, `xfailed`, `xpassed`), `duration_ms`, `message` (up to 16 KB), `skip_reason`, +`log` (captured output of failed tests, up to 64 KB), `trace` (the trace file, `""` when not kept), +`measurements`, and `calls`: per pod command the number of calls, the 50th and 95th percentile and +maximum time in ms, and the errors. `calls` is filled even when the trace file is not kept. + +A trace line looks like +`{"t": 12.304, "op": "command", "cmd": "can_write", "req": {...}, "ms": 3.1, "ok": true, "resp": {...}}`: +`t` is seconds since the run started, `op` one of `command`, `sequence`, `samples`, `stream`, +`raw` (a CMSIS-DAP or UART proxy session, logged when it closes, with its byte counts), `status`, +`power`. Requests and replies over 4 KB move to `att/`. Passwords, tokens, keys and other secrets +are replaced by `""` before anything is written. A run keeps at most 50 MB of traces and +attachments; past that, lines keep only `t`, `op`, `cmd`, `ms` and `ok`, and `health` says so. +Every `BenchPod` opened while the run records is traced, also one a test opens itself. + +`--embeddedci-keep-traces` picks which traces stay: `all` (default), `failed` (failed and error +tests only) or `none`. + +**Measurements.** The `measure` fixture records named numbers with optional limits: + +```python +def test_heater_response(benchpod, measure): + ... + measure("settle_ms", 840, unit="ms", max=1000) # fails the test above 1000 + measure("overshoot_c", 0.6, unit="C", drift="15%") # recorded, compared later +``` + +`value`, `min` and `max` must be finite numbers (`bool`, NaN and infinity raise an error), and a +name is used once per test. A value outside `min`/`max` fails the test after its body ran, listing +every failed limit; the limits are checked also when no run is being recorded. `drift` (`"20%"` or +`0.2`) is a tolerance for comparisons with a baseline and never fails the test. Numeric values a +test passes to pytest's `record_property` are recorded too (with source `record_property`; other +values are left out). A test keeps at most 200 measurements; the rest are dropped and counted in +`health.measurements_dropped`. + +**Inside `benchpod ci`.** The `benchpod ci` runner hands its run folder to the `pytest` steps it +starts through the environment; the plugin then writes `parts/.json` (the cases, health +counters, tool versions and pod) and its traces into that folder, and the runner merges them into +one `run.json`: + +| Variable | Value | +|---|---| +| `EMBEDDEDCI_RUN_DIR` | the run folder (alone, without `EMBEDDEDCI_RUN_STEP`: same as `--embeddedci-run-dir`) | +| `EMBEDDEDCI_RUN_STEP` | the step id: write `parts/.json` instead of `run.json` and `junit.xml` | +| `EMBEDDEDCI_RUN_ID` | the run id recorded in `run.json` | +| `EMBEDDEDCI_RUN_T0` | the run's start in unix seconds, so every step's trace shares one timeline | +| `EMBEDDEDCI_KEEP_TRACES` | `all`, `failed` or `none` (`--embeddedci-keep-traces` wins) | + ## Wiring profile The pod has no role-named pins, so the bench's wiring — which DUT signal is on which LA channel — diff --git a/packages/embeddedci/src/embeddedci/benchpod/client.py b/packages/embeddedci/src/embeddedci/benchpod/client.py index ee435a6..93f70ed 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/client.py +++ b/packages/embeddedci/src/embeddedci/benchpod/client.py @@ -385,6 +385,12 @@ def __init__( api_key=self._api_key, user_token=self._cloud_user_token, ) + # A test run being recorded (the pytest plugin's run folder): record this pod's calls too. + from .runs.active import active_run + + run = active_run() + if run is not None: + run.attach(self._transport) try: if lease and isinstance(self._transport, CloudTransport): self._lease = DeviceLease( @@ -416,6 +422,8 @@ def __init__( except BaseException: self.close() raise + if run is not None and run.active: + run.note_pod(self) def _auto_la_voltage(self) -> None: """On connect: when the pod has no I/O voltage yet and the wiring profile sets one, set it. diff --git a/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py b/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py index 5d40b30..7f7b00d 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py +++ b/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py @@ -26,6 +26,8 @@ from .wiring import Wiring from .constants import PULL_OHMS as _PULL_OHMS from .constants import PULLDOWN_CHANNELS +from .runs import pytest_runs as _runs +from .runs.pytest_runs import measure # noqa: F401 -- a fixture of this plugin #: The channels whose network pulls DOWN rather than up. _PULLDOWN_CHANNELS = frozenset(PULLDOWN_CHANNELS) @@ -218,6 +220,7 @@ def pytest_addoption(parser: "pytest.Parser") -> None: help="Sensor library version for the benchpod_sensors fixture: 'latest' or a release " "such as 0.2.0 (default: the version this SDK pins, or BENCHPOD_SENSOR_LIBRARY).", ) + _runs.add_options(parser) parser.addini( "benchpod_connection", help="Default BenchPod connection (host[:port], device path, 'usb', or " @@ -227,6 +230,10 @@ def pytest_addoption(parser: "pytest.Parser") -> None: def pytest_configure(config: "pytest.Config") -> None: + # Test run recording (--embeddedci-run-dir / benchpod ci) and the measure fixture's limits. + pm = getattr(config, "pluginmanager", None) + if pm is not None and not pm.has_plugin(_runs.PLUGIN_NAME): + pm.register(_runs.RunsPlugin(config), _runs.PLUGIN_NAME) config.addinivalue_line( "markers", "hardware: test needs a real BenchPod (and usually a wired DUT); " diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/__init__.py b/packages/embeddedci/src/embeddedci/benchpod/runs/__init__.py new file mode 100644 index 0000000..5bd89a2 --- /dev/null +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/__init__.py @@ -0,0 +1,60 @@ +"""Test run recording: the run folder (``run.json``, ``junit.xml``, per-case pod-call traces) the +pytest plugin writes with ``--embeddedci-run-dir`` or under ``benchpod ci``. + +- :mod:`.schema`: the file formats (schema version 1), redaction, caps, case slugs. +- :mod:`.recorder`: :class:`Recorder`, which records one transport instance's calls. +- :mod:`.active`: :class:`RunRecording` and the process's active run (:func:`active_run`), which + every :class:`~embeddedci.benchpod.BenchPod` opened while it is active attaches to. +- :mod:`.provenance`: git, tool, GitHub and pod details for ``inputs``. +""" + +from .active import ( + KEEP_TRACES_ENV, + RUN_DIR_ENV, + RUN_ID_ENV, + RUN_STEP_ENV, + RUN_T0_ENV, + RunRecording, + active_run, + clear_active_run, + default_run_dir, + new_run_id, + set_active_run, +) +from .recorder import Recorder, RecordingLink +from .schema import ( + SCHEMA_VERSION, + CallStats, + Case, + Measurement, + case_slug, + junit_xml, + parse_drift, + redact, + run_outcome, +) + +__all__ = [ + "KEEP_TRACES_ENV", + "RUN_DIR_ENV", + "RUN_ID_ENV", + "RUN_STEP_ENV", + "RUN_T0_ENV", + "SCHEMA_VERSION", + "CallStats", + "Case", + "Measurement", + "Recorder", + "RecordingLink", + "RunRecording", + "active_run", + "case_slug", + "clear_active_run", + "default_run_dir", + "junit_xml", + "new_run_id", + "parse_drift", + "redact", + "run_outcome", + "set_active_run", +] diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/active.py b/packages/embeddedci/src/embeddedci/benchpod/runs/active.py new file mode 100644 index 0000000..4488c3c --- /dev/null +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/active.py @@ -0,0 +1,449 @@ +"""The run being recorded in this process, and its trace files. + +There is at most one active :class:`RunRecording` per process (:func:`active_run`). The pytest +plugin starts it; :class:`~embeddedci.benchpod.BenchPod` asks for it after its transport is open and +attaches a :class:`~.recorder.Recorder`, so every pod opened while the run is active is recorded, +including ones a test opens itself. +""" + +from __future__ import annotations + +import os +import secrets +import threading +import time +import weakref +from dataclasses import dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import IO, Any, Dict, List, Mapping, Optional, Sequence, Tuple, Union + +from . import provenance +from .recorder import Recorder, suppressed +from .schema import ( + INLINE_JSON_MAX_BYTES, + KEEP_POLICIES, + MEASUREMENTS_PER_CASE, + TRACE_MAX_BYTES_PER_RUN, + CallStats, + Case, + Measurement, + build_part_record, + build_run_record, + case_slug, + compact_json, + empty_health, + junit_xml, + redact, + write_bytes_atomic, + write_json_atomic, +) + +try: # optional: compress attachments when the module is there + import zstandard as _zstd # type: ignore[import-not-found] +except ImportError: # pragma: no cover - depends on the environment + _zstd = None + +#: Environment handover from ``benchpod ci`` (spec section 4). +RUN_DIR_ENV = "EMBEDDEDCI_RUN_DIR" +RUN_ID_ENV = "EMBEDDEDCI_RUN_ID" +RUN_STEP_ENV = "EMBEDDEDCI_RUN_STEP" +RUN_T0_ENV = "EMBEDDEDCI_RUN_T0" +KEEP_TRACES_ENV = "EMBEDDEDCI_KEEP_TRACES" + +#: Case outcomes whose trace the ``failed`` policy keeps. +_KEPT_WHEN_FAILED = frozenset({"failed", "error"}) +#: Trace file for pod calls made outside any case. +SESSION_TRACE = "_session" + +_active: Optional["RunRecording"] = None +_active_lock = threading.Lock() + + +def active_run() -> Optional["RunRecording"]: + """The run being recorded in this process, or None.""" + run = _active + return run if run is not None and run.active else None + + +def set_active_run(run: Optional["RunRecording"]) -> None: + """Make ``run`` the process's active run (None clears it).""" + global _active + with _active_lock: + _active = run + + +def clear_active_run(run: Optional["RunRecording"] = None) -> None: + """Clear the active run (only when it is ``run``, if given).""" + global _active + with _active_lock: + if run is None or _active is run: + _active = None + + +def utc_stamp(ts: float) -> str: + """``2026-11-18T14:02:11.123Z``""" + dt = datetime.fromtimestamp(ts, tz=timezone.utc) + return dt.strftime("%Y-%m-%dT%H:%M:%S.") + f"{dt.microsecond // 1000:03d}Z" + + +def folder_stamp(ts: float) -> str: + """``20261118T140211Z``""" + return datetime.fromtimestamp(ts, tz=timezone.utc).strftime("%Y%m%dT%H%M%SZ") + + +def new_run_id(started: float) -> str: + """``local-20261118T140211Z-3f9a``: the id a run has until it is uploaded.""" + return f"local-{folder_stamp(started)}-{secrets.token_hex(2)}" + + +def default_run_dir(started: float, suite: str = "", base: Union[str, "os.PathLike[str]"] = ".") -> Path: + """``/embeddedci-runs/-``.""" + return Path(base) / "embeddedci-runs" / f"{folder_stamp(started)}-{case_slug_simple(suite or 'pytest')}" + + +def case_slug_simple(text: str) -> str: + return "".join(c if c.isalnum() or c in "._-" else "_" for c in text)[:80] or "pytest" + + +@dataclass +class _CaseState: + case_id: Optional[str] + calls: Dict[str, CallStats] = field(default_factory=dict) + measurements: List[Measurement] = field(default_factory=list) + file: Optional[IO[bytes]] = None + trace_rel: str = "" + bytes: int = 0 + att: List[Tuple[str, int]] = field(default_factory=list) + + +class RunRecording: + """One run being recorded: its folder, the trace of the current case, call statistics, the + measurements, and the pods it saw. + + ``keep`` is the trace policy (``all``, ``failed``, ``none``); ``t0`` the run's start in unix + seconds (trace ``t`` counts from it; ``benchpod ci`` passes its own so child processes share one + timeline). ``handover`` marks a child of ``benchpod ci``: it writes ``parts/.json`` instead + of ``run.json`` and ``junit.xml``. + """ + + def __init__(self, run_dir: Union[str, "os.PathLike[str]"], *, step: str = "pytest", + kind: str = "pytest", keep: str = "all", t0: Optional[float] = None, + handover: bool = False, run_id: str = "", + trace_cap: int = TRACE_MAX_BYTES_PER_RUN) -> None: + if keep not in KEEP_POLICIES: + raise ValueError(f"keep traces must be one of {', '.join(KEEP_POLICIES)}, got {keep!r}") + self.run_dir = Path(run_dir).resolve() + self.step = step + self.kind = kind + self.keep = keep + self.handover = handover + self.started = time.time() + self.t0 = float(t0) if t0 is not None else self.started + self.run_id = run_id or new_run_id(self.started) + self.trace_cap = trace_cap + self.health: Dict[str, Any] = empty_health() + self.active = True + self._lock = threading.RLock() + self._current: Optional[_CaseState] = None + self._session = _CaseState(None) + self._cases: Dict[str, _CaseState] = {} + self._bytes = 0 + self._att_n = 0 + self._used_traces: set = set() + self.recorders: List[Recorder] = [] + self._pods: List[Dict[str, Any]] = [] + self._bench: List["weakref.ReferenceType[Any]"] = [] + self.wiring: Dict[str, Any] = provenance.wiring_info(None) + self.run_dir.mkdir(parents=True, exist_ok=True) + + # -- time and state ------------------------------------------------------------------------ + + def now(self) -> float: + """Seconds since the run started (trace ``t``).""" + return round(time.time() - self.t0, 3) + + def wants_payloads(self) -> bool: + """Whether request and reply bodies are still written (not with ``keep=none``, not past the + run's trace cap).""" + return self.keep != "none" and not self.health["trace_truncated"] + + def add_health(self, key: str, n: int = 1) -> None: + with self._lock: + self.health[key] = int(self.health.get(key, 0)) + n + + def add_reason(self, reason: str, *, invalid: bool = True) -> None: + """A note on the run's health; ``invalid`` (default) marks the run as not a valid measurement.""" + with self._lock: + self.health["reasons"].append(reason) + if invalid: + self.health["valid"] = False + + # -- cases --------------------------------------------------------------------------------- + + def begin_case(self, case_id: str) -> None: + """Pod calls from now on belong to ``case_id`` (until :meth:`end_case`).""" + with self._lock: + self._close_file(self._current) + state = self._cases.get(case_id) + if state is None: + state = _CaseState(case_id) + self._cases[case_id] = state + self._current = state + + def end_case(self, case_id: str, outcome: str) -> Tuple[str, Dict[str, CallStats]]: + """Close ``case_id``'s trace and apply the keep policy for ``outcome``: the trace path + relative to the run folder ("" when none was kept) and its per-command call statistics.""" + with self._lock: + state = self._cases.get(case_id) or _CaseState(case_id) + self._close_file(state) + if self._current is state: + self._current = None + if state.trace_rel and self.keep == "failed" and outcome not in _KEPT_WHEN_FAILED: + self._drop_files(state) + self._snapshot_wiring() + return state.trace_rel, dict(state.calls) + + def current_case(self) -> Optional[str]: + state = self._current + return state.case_id if state is not None else None + + def add_measurement(self, case_id: str, m: Measurement) -> bool: + """Store a measurement for ``case_id``; False (and counted in ``health``) past the cap.""" + with self._lock: + state = self._cases.get(case_id) + if state is None: + state = _CaseState(case_id) + self._cases[case_id] = state + if len(state.measurements) >= MEASUREMENTS_PER_CASE: + self.health["measurements_dropped"] += 1 + return False + state.measurements.append(m) + return True + + def measurements(self, case_id: str) -> List[Measurement]: + state = self._cases.get(case_id) + return list(state.measurements) if state is not None else [] + + def session_calls(self) -> Dict[str, CallStats]: + """Call statistics of pod calls made outside every case.""" + return dict(self._session.calls) + + # -- pods ---------------------------------------------------------------------------------- + + def attach(self, transport: Any) -> Recorder: + """Record ``transport`` (idempotent). A transport recorded for an earlier run moves to this one.""" + with self._lock: + existing = getattr(transport, "_embeddedci_recorder", None) + if isinstance(existing, Recorder): + if existing.run is self: + return existing + existing.uninstall() + rec = Recorder(self, transport).install() + self.recorders.append(rec) + return rec + + def note_pod(self, bench: Any) -> None: + """Remember a connected :class:`~embeddedci.benchpod.BenchPod` for ``inputs.pod`` and + ``inputs.wiring``. Reads the status (when no call has yet) and, on a pod that keeps one, the + stored calibration; these reads are not part of any case's trace. Never raises.""" + try: + transport = bench.transport + rec = getattr(transport, "_embeddedci_recorder", None) + with suppressed(): + status = rec.last_status if rec is not None and rec.last_status else None + if status is None: + try: + status = bench.status() + except Exception: # noqa: BLE001 + status = {} + calibration = None + try: + if bench.capabilities.calibrate: + reply = bench.command({"cmd": "calibrate"}) + if isinstance(reply, dict) and reply.get("calibrated") is True: + calibration = reply + except Exception: # noqa: BLE001 - provenance never fails a run + calibration = None + block = provenance.pod_info(status or {}, transport=transport, name=_pod_name(bench, transport), + calibration=calibration) + except Exception: # noqa: BLE001 + return + with self._lock: + self._pods.append(block) + self._bench.append(weakref.ref(bench)) + self._snapshot_wiring() + + def pod(self) -> Dict[str, Any]: + """The first pod this run connected to ({} when none).""" + return dict(self._pods[0]) if self._pods else {} + + def pods(self) -> List[Dict[str, Any]]: + return [dict(p) for p in self._pods] + + def _snapshot_wiring(self) -> None: + for ref in self._bench: + bench = ref() + wiring = getattr(bench, "_wiring", None) if bench is not None else None + if wiring is not None: + self.wiring = provenance.wiring_info(wiring) + return + + # -- trace --------------------------------------------------------------------------------- + + def record(self, *, t: float, op: str, cmd: str, req: Any, ms: float, ok: bool, resp: Any = None, + error: str = "", extra: Optional[Mapping[str, Any]] = None, recorder: Any = None) -> None: + """One trace line for the current case (spec section 3) and its call statistics.""" + with self._lock: + state = self._current or self._session + state.calls.setdefault(cmd or op, CallStats()).add(ms, ok) + if self.keep == "none": + return + ms_r = round(ms, 3) + if self.health["trace_truncated"] or self._bytes >= self.trace_cap: + self.health["trace_truncated"] = True + line: Dict[str, Any] = {"t": t, "op": op, "cmd": cmd, "ms": ms_r, "ok": ok} + else: + line = {"t": t, "op": op} + if cmd: + line["cmd"] = cmd + big: Dict[str, Any] = {} + req_r = redact(req) if req is not None else {} + req_text = compact_json(req_r) + if len(req_text.encode("utf-8")) <= INLINE_JSON_MAX_BYTES: + line["req"] = req_r + else: + big["req"] = req_r + line["ms"] = ms_r + line["ok"] = ok + for k, v in (extra or {}).items(): + line[k] = v + if ok and resp is not None: + resp_r = redact(resp) + resp_text = compact_json(resp_r) + if len(resp_text.encode("utf-8")) <= INLINE_JSON_MAX_BYTES: + line["resp"] = resp_r + else: + big["resp"] = resp_r + if error: + line["error"] = error + if big: + line["att"] = self._write_att(state, cmd or op, big) + self._write_line(state, line) + + def _write_att(self, state: _CaseState, what: str, payload: Mapping[str, Any]) -> str: + data = compact_json(payload).encode("utf-8") + ext = ".json" + if _zstd is not None: + data = _zstd.ZstdCompressor().compress(data) + ext = ".json.zst" + att_dir = self.run_dir / "att" + att_dir.mkdir(parents=True, exist_ok=True) + safe = case_slug_simple(what) + while True: + self._att_n += 1 + rel = f"att/{self._att_n:04d}-{safe}{ext}" + try: + with open(self.run_dir / rel, "xb") as f: # several processes share att/ + f.write(data) + break + except FileExistsError: + continue + self._bytes += len(data) + state.att.append((rel, len(data))) + return rel + + def _write_line(self, state: _CaseState, line: Mapping[str, Any]) -> None: + if state.file is None: + if not self._open_trace(state): + return + assert state.file is not None + data = (compact_json(line) + "\n").encode("utf-8") + state.file.write(data) + state.file.flush() + state.bytes += len(data) + self._bytes += len(data) + + def _open_trace(self, state: _CaseState) -> bool: + trace_dir = self.run_dir / "trace" + trace_dir.mkdir(parents=True, exist_ok=True) + if state.trace_rel: # reopened (a case whose trace was closed between phases) + state.file = open(self.run_dir / state.trace_rel, "ab") + return True + stem = case_slug(state.case_id) if state.case_id is not None else SESSION_TRACE + n = 1 + while True: + name = stem if n == 1 else f"{stem}-{n}" + rel = f"trace/{name}.ndjson" + if rel not in self._used_traces: + try: + state.file = open(self.run_dir / rel, "xb") # another step may own the name + break + except FileExistsError: + pass + n += 1 + self._used_traces.add(rel) + state.trace_rel = rel + return True + + @staticmethod + def _close_file(state: Optional[_CaseState]) -> None: + if state is not None and state.file is not None: + try: + state.file.close() + finally: + state.file = None + + def _drop_files(self, state: _CaseState) -> None: + for rel, size in [(state.trace_rel, state.bytes)] + state.att: + try: + os.unlink(self.run_dir / rel) + self._bytes -= size + except OSError: + pass + state.trace_rel = "" + state.bytes = 0 + state.att = [] + + # -- end ----------------------------------------------------------------------------------- + + def finish(self) -> None: + """Stop recording: later calls of recorded transports go straight through.""" + with self._lock: + self.active = False + self._close_file(self._current) + self._close_file(self._session) + self._current = None + self._snapshot_wiring() + clear_active_run(self) + + def write_run(self, *, cases: Sequence[Case], key: Mapping[str, Any], inputs: Mapping[str, Any], + suite: str, finished: Optional[float] = None) -> Dict[str, Any]: + """Write ``junit.xml`` and then ``run.json`` (atomically, last). Returns the record.""" + end = finished if finished is not None else time.time() + duration_ms = int(round((end - self.started) * 1000)) + record = build_run_record(run_id=self.run_id, key=key, trigger=provenance.trigger(), + started=utc_stamp(self.started), finished=utc_stamp(end), + inputs=inputs, health=self.health, cases=cases, + duration_ms=duration_ms, keep=self.keep) + write_bytes_atomic(self.run_dir / "junit.xml", + junit_xml(suite, cases, timestamp=utc_stamp(self.started), duration_ms=duration_ms)) + write_json_atomic(self.run_dir / "run.json", record) + return record + + def write_part(self, *, cases: Sequence[Case], tools: Mapping[str, Any]) -> Dict[str, Any]: + """Write ``parts/.json`` for ``benchpod ci`` to merge. Returns the record.""" + record = build_part_record(step=self.step, cases=cases, health=self.health, tools=tools, + pod=self.pod() or None) + write_json_atomic(self.run_dir / "parts" / f"{case_slug_simple(self.step)}.json", record) + return record + + +def _pod_name(bench: Any, transport: Any) -> str: + name = str(getattr(bench, "_device_name", "") or "") + if name: + return name + addr = str(getattr(transport, "addr", "") or "") + host = addr.rsplit(":", 1)[0] if addr.count(":") == 1 else addr + host = host.removesuffix(".local") + return host if host.startswith("benchpod-") else "" diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/provenance.py b/packages/embeddedci/src/embeddedci/benchpod/runs/provenance.py new file mode 100644 index 0000000..ffff443 --- /dev/null +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/provenance.py @@ -0,0 +1,175 @@ +"""What a run records around its results: git state of the tests, tool versions, GitHub run, the +DUT firmware file and the pod (``inputs`` in ``run.json``). Every helper returns empty values +instead of raising: provenance never fails a run.""" + +from __future__ import annotations + +import hashlib +import os +import platform +import re +import subprocess +import sys +from pathlib import Path +from typing import Any, Dict, Mapping, Optional + +from .schema import empty_pod, sha256_of_json + +_GIT_TIMEOUT = 5.0 + + +def _git(cwd: str, *args: str) -> Optional[str]: + try: + out = subprocess.run(["git", *args], cwd=cwd, capture_output=True, text=True, + timeout=_GIT_TIMEOUT, check=False) + except (OSError, subprocess.SubprocessError): + return None + if out.returncode != 0: + return None + return out.stdout.strip() + + +def repo_slug(remote_url: str) -> str: + """``owner/repo`` from a git remote URL (https or ssh); "" when it does not look like one (the + URL itself is never recorded: it can carry credentials).""" + m = re.search(r"[:/]([^/:@]+/[^/:@]+?)(?:\.git)?/?$", (remote_url or "").strip()) + return m.group(1) if m else "" + + +def git_info(cwd: str) -> Dict[str, Any]: + """``{"repo", "ref", "sha", "dirty"}`` of the git checkout at ``cwd`` (empty when not one).""" + info: Dict[str, Any] = {"repo": "", "ref": "", "sha": "", "dirty": False} + sha = _git(cwd, "rev-parse", "HEAD") + if not sha: + return info + info["sha"] = sha + ref = _git(cwd, "rev-parse", "--abbrev-ref", "HEAD") or "" + info["ref"] = "" if ref == "HEAD" else ref + if not info["ref"]: + info["ref"] = os.environ.get("GITHUB_HEAD_REF") or os.environ.get("GITHUB_REF_NAME", "") + info["repo"] = repo_slug(_git(cwd, "config", "--get", "remote.origin.url") or "") + if not info["repo"]: + info["repo"] = os.environ.get("GITHUB_REPOSITORY", "") + status = _git(cwd, "status", "--porcelain", "--untracked-files=no") + info["dirty"] = bool(status) + return info + + +def tools_info(pytest_version: str = "") -> Dict[str, str]: + """Tool versions: ``cli`` (from ``EMBEDDEDCI_CLI_VERSION`` when the CLI sets it), ``sdk``, + ``python``, ``pytest`` and ``os`` (``darwin-arm64``, ``linux-x86_64``).""" + from ... import __version__ as sdk_version + + machine = platform.machine().lower() or "unknown" + return { + "cli": os.environ.get("EMBEDDEDCI_CLI_VERSION", ""), + "sdk": sdk_version, + "python": platform.python_version(), + "pytest": pytest_version, + "os": f"{sys.platform}-{machine}", + } + + +def github_info() -> Dict[str, str]: + """The GitHub Actions run, from the standard ``GITHUB_*`` variables (empty elsewhere).""" + server = os.environ.get("GITHUB_SERVER_URL", "https://github.com") + repo = os.environ.get("GITHUB_REPOSITORY", "") + run_id = os.environ.get("GITHUB_RUN_ID", "") + return { + "run_id": run_id, + "run_attempt": os.environ.get("GITHUB_RUN_ATTEMPT", ""), + "run_url": f"{server}/{repo}/actions/runs/{run_id}" if repo and run_id else "", + "workflow": os.environ.get("GITHUB_WORKFLOW", ""), + "actor": os.environ.get("GITHUB_ACTOR", ""), + } + + +def trigger() -> str: + """``github`` inside GitHub Actions, ``other`` in another CI, else ``manual``.""" + if os.environ.get("GITHUB_ACTIONS") == "true": + return "github" + if os.environ.get("CI", "").lower() in ("1", "true", "yes"): + return "other" + return "manual" + + +def firmware_info(path: Optional[str]) -> Dict[str, Any]: + """``{"file", "sha256", "size", "build_id", "version"}`` of a DUT firmware file ({} when none).""" + if not path: + return {} + p = Path(path) + out: Dict[str, Any] = {"file": p.name, "sha256": "", "size": 0, "build_id": "", "version": ""} + try: + h = hashlib.sha256() + with p.open("rb") as f: + for block in iter(lambda: f.read(1 << 20), b""): + h.update(block) + out["sha256"] = h.hexdigest() + out["size"] = p.stat().st_size + except OSError: + pass + return out + + +def connection_kind(transport: Any) -> str: + """``lan`` | ``usb_net`` | ``cloud`` | ``serial`` for a transport instance.""" + from ..transport.cloud import CloudTransport + from ..transport.serial import SerialTransport + + if isinstance(transport, CloudTransport): + return "cloud" + if isinstance(transport, SerialTransport): + return "serial" + if getattr(transport, "usb_net", False): + return "usb_net" + return "lan" + + +def board_label(status: Mapping[str, Any]) -> str: + """The board as its silkscreen tag names it (``analog-v3``, ``digital-v3``), from the status + report: ``board_tag`` when the firmware sends one, else the front end plus ``board_rev``, else + the firmware's ``board``.""" + tag = str(status.get("board_tag") or "").strip() + if tag: + return tag + rev = str(status.get("board_rev") or "").strip() + if rev and rev != "unknown": + kind = "digital" if status.get("analog") is False else "analog" + return f"{kind}-{rev}" + return str(status.get("board") or "") + + +def _int(value: Any) -> int: + try: + return int(value) + except (TypeError, ValueError): + return 0 + + +def pod_info(status: Mapping[str, Any], *, transport: Any = None, name: str = "", + calibration: Optional[Mapping[str, Any]] = None) -> Dict[str, Any]: + """The ``inputs.pod`` block from a status report: name, board, firmware, gateware, + features_rev, calibration_sha256 (of ``calibration``, "" without one) and connection kind.""" + pod = empty_pod() + pod["name"] = (name or str(status.get("name") or status.get("hostname") or "") + or str(getattr(transport, "device_name", "") or "")) + pod["board"] = board_label(status) + pod["firmware"] = str(status.get("version", status.get("firmware_version", "")) or "") + pod["gateware"] = _int(status.get("gateware")) + pod["features_rev"] = _int(status.get("features_rev")) + if calibration: + pod["calibration_sha256"] = sha256_of_json(dict(calibration)) + pod["connection"] = connection_kind(transport) if transport is not None else "" + return pod + + +def wiring_info(wiring: Any) -> Dict[str, Any]: + """``{"name", "sha256", "snapshot"}`` of a :class:`~embeddedci.benchpod.wiring.Wiring`.""" + if wiring is None: + return {"name": "", "sha256": "", "snapshot": {}} + try: + snapshot = wiring.to_dict() + except Exception: # noqa: BLE001 + return {"name": "", "sha256": "", "snapshot": {}} + return {"name": str(getattr(wiring, "profile_name", "") or ""), "sha256": sha256_of_json(snapshot), + "snapshot": snapshot} diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/pytest_runs.py b/packages/embeddedci/src/embeddedci/benchpod/runs/pytest_runs.py new file mode 100644 index 0000000..ac56de8 --- /dev/null +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/pytest_runs.py @@ -0,0 +1,340 @@ +"""The pytest side of run recording: the ``--embeddedci-*`` options, the ``measure`` fixture and the +hooks that turn pytest's reports into the run folder's cases. + +Off unless asked for: ``--embeddedci-run-dir DIR`` or ``EMBEDDEDCI_RUN_DIR`` turns it on. With +``EMBEDDEDCI_RUN_DIR`` and ``EMBEDDEDCI_RUN_STEP`` (set by ``benchpod ci``) the plugin writes +``parts/.json`` and its traces into that folder instead of a whole run. ``measure`` checks its +limits whether a run is being recorded or not. +""" + +from __future__ import annotations + +import os +from typing import Any, Dict, Iterator, List, Optional, Tuple + +import pytest + +from . import provenance +from .active import ( + KEEP_TRACES_ENV, + RUN_DIR_ENV, + RUN_ID_ENV, + RUN_STEP_ENV, + RUN_T0_ENV, + RunRecording, + active_run, + default_run_dir, + set_active_run, +) +from .schema import ( + KEEP_POLICIES, + LOG_MAX_BYTES, + MEASUREMENTS_PER_CASE, + Case, + Measurement, + cap_text, + empty_pod, + is_plain_number, +) + +PLUGIN_NAME = "embeddedci-runs" +#: ``--embeddedci-run-dir auto``: the default folder, ``./embeddedci-runs/-/``. +AUTO_RUN_DIR = "auto" + + +def add_options(parser: "pytest.Parser") -> None: + group = parser.getgroup("embeddedci", "EmbeddedCI test run recording") + group.addoption( + "--embeddedci-run-dir", action="store", default=None, dest="embeddedci_run_dir", + metavar="DIR", + help="Record this run into the folder DIR (run.json, junit.xml, a pod-call trace per test); " + "'auto' picks ./embeddedci-runs/-/. Off by default; EMBEDDEDCI_RUN_DIR " + "turns it on too.", + ) + group.addoption( + "--embeddedci-suite", action="store", default=None, dest="embeddedci_suite", + help="Suite name recorded in run.json (key.suite; default 'pytest').", + ) + group.addoption( + "--embeddedci-dut", action="store", default=None, dest="embeddedci_dut", + help="Name of the board under test recorded in run.json (key.dut; default '').", + ) + group.addoption( + "--embeddedci-keep-traces", action="store", default=None, dest="embeddedci_keep_traces", + choices=list(KEEP_POLICIES), + help="Which pod-call traces to keep: all (default), failed (failed and error tests only) or " + "none. Falls back to EMBEDDEDCI_KEEP_TRACES.", + ) + + +# -- measure ------------------------------------------------------------------------------------- + +class Measure: + """The ``measure`` fixture: ``measure(name, value, unit="", min=None, max=None, drift=None)``. + + Records a named number for the test. ``min``/``max`` are limits: a value outside them fails the + test (all failures listed, after the test body). ``drift`` (``"20%"`` or ``0.2``) is the + tolerance a later comparison against a baseline uses; it never fails the test. Numbers only: + ``bool``, NaN and infinity are refused. Stored in the run when one is being recorded (at most + 200 per test); the limits are checked either way.""" + + def __init__(self, case_id: str, run: Optional[RunRecording]) -> None: + self.case_id = case_id + self._run = run + self.items: List[Measurement] = [] + self._names: set = set() + self._reported = 0 + + def __call__(self, name: str, value: Any, unit: str = "", min: Any = None, max: Any = None, + drift: Any = None) -> Measurement: + m = Measurement.make(name, value, unit=unit, min=min, max=max, drift=drift, source="measure") + if name in self._names: + raise ValueError(f"measurement {name!r} was already recorded in this test: names are " + f"unique per test") + self._names.add(name) + self.items.append(m) + run = self._run + if run is not None and run.active: + run.add_measurement(self.case_id, m) + return m + + @property + def failures(self) -> List[Measurement]: + """Every measurement outside its limits so far.""" + return [m for m in self.items if m.outcome == "failed"] + + def take_unreported(self) -> List[Measurement]: + """The failures not yet raised (each failure fails the test once).""" + failed = self.failures + new = failed[self._reported:] + self._reported = len(failed) + return new + + +def _limit_text(failures: List[Measurement]) -> str: + lines = [f"{len(failures)} measurement(s) outside their limits:"] + lines += [f" {m.describe_failure()}" for m in failures] + return "\n".join(lines) + + +@pytest.fixture +def measure(request: "pytest.FixtureRequest") -> Iterator[Measure]: + """Record named numeric values with optional limits: + ``measure("settle_ms", 840, unit="ms", max=1000)``. A value outside ``min``/``max`` fails the + test (every failure listed); ``drift`` (``"20%"`` or ``0.2``) is kept for comparisons with a + baseline. Stored in the run folder when one is recorded.""" + m = Measure(request.node.nodeid, active_run()) + request.node._embeddedci_measure = m + yield m + pending = m.take_unreported() + if pending: + rep_call = getattr(request.node, "rep_call", None) + if rep_call is not None and rep_call.failed: + return # the test failed already; the measurements are still recorded + pytest.fail(_limit_text(pending), pytrace=False) + + +# -- outcomes -------------------------------------------------------------------------------------- + +def _skip_reason(rep: Any) -> str: + longrepr = getattr(rep, "longrepr", None) + if isinstance(longrepr, tuple) and len(longrepr) == 3: + text = str(longrepr[2]) + else: + text = str(longrepr or "") + for prefix in ("Skipped: ", "Skipped:"): + if text.startswith(prefix): + return text[len(prefix):].strip() + return text.strip() + + +def case_outcome(reports: Dict[str, Any]) -> Tuple[str, str, str]: + """``(outcome, message, skip_reason)`` of a test from its setup/call/teardown reports. + + A failure in setup or teardown is ``error``, one in the test body ``failed``; ``xfailed`` and + ``xpassed`` come from pytest's xfail marker (a strict xpass is ``failed``, as pytest reports it).""" + outcome = "" + message = "" + reason = "" + for when in ("setup", "call", "teardown"): + rep = reports.get(when) + if rep is None: + continue + wasxfail = getattr(rep, "wasxfail", None) + if wasxfail is not None and rep.skipped: + outcome, reason = "xfailed", str(wasxfail) + message = rep.longreprtext or "" + elif wasxfail is not None and rep.passed: + outcome, reason = "xpassed", str(wasxfail) + elif rep.failed: + if when == "call": + outcome, message = "failed", rep.longreprtext or "" + elif when == "teardown" and outcome == "failed": + message = (message + "\n\n" + (rep.longreprtext or "")).strip() + else: + outcome, message = "error", rep.longreprtext or "" + elif rep.skipped: + outcome, reason = "skipped", _skip_reason(rep) + elif rep.passed and when == "call": + outcome = "passed" + elif rep.passed and when == "setup" and not outcome: + outcome = "passed" + return outcome or "passed", message, reason + + +def _captured_output(reports: Dict[str, Any]) -> str: + """The captured stdout/stderr/log of a test (pytest's report sections), as one text.""" + best: List[Tuple[str, str]] = [] + for when in ("setup", "call", "teardown"): + rep = reports.get(when) + sections = list(getattr(rep, "sections", []) or []) if rep is not None else [] + if len(sections) >= len(best): + best = sections + parts = [f"----- {title} -----\n{content.rstrip()}" for title, content in best if content.strip()] + return cap_text("\n".join(parts), LOG_MAX_BYTES, keep="tail") + + +# -- the plugin ------------------------------------------------------------------------------------ + +class RunsPlugin: + """Registered for every session: checks ``measure`` limits always, and records the run when + ``--embeddedci-run-dir`` or ``EMBEDDEDCI_RUN_DIR`` asks for it.""" + + def __init__(self, config: "pytest.Config") -> None: + self.config = config + self.run: Optional[RunRecording] = None + self.cases: List[Case] = [] + self.suite = "pytest" + self.dut = "" + self.record: Optional[Dict[str, Any]] = None + self._reports: Dict[str, Dict[str, Any]] = {} + + # -- start -------------------------------------------------------------------------------- + + def _settings(self) -> Optional[Dict[str, Any]]: + config = self.config + if hasattr(config, "workerinput"): # pytest-xdist worker: the controller records the cases + return None + env_dir = os.environ.get(RUN_DIR_ENV, "").strip() + env_step = os.environ.get(RUN_STEP_ENV, "").strip() + opt_dir = config.getoption("embeddedci_run_dir", None) + self.suite = config.getoption("embeddedci_suite", None) or "pytest" + self.dut = config.getoption("embeddedci_dut", None) or "" + keep = (config.getoption("embeddedci_keep_traces", None) + or os.environ.get(KEEP_TRACES_ENV, "").strip().lower() or "all") + if keep not in KEEP_POLICIES: + raise pytest.UsageError(f"{KEEP_TRACES_ENV} must be one of {', '.join(KEEP_POLICIES)}, got {keep!r}") + t0_text = os.environ.get(RUN_T0_ENV, "").strip() + try: + t0 = float(t0_text) if t0_text else None + except ValueError: + raise pytest.UsageError(f"{RUN_T0_ENV} must be unix seconds, got {t0_text!r}") from None + if env_dir and env_step: + return {"run_dir": env_dir, "step": env_step, "handover": True, "keep": keep, "t0": t0, + "run_id": os.environ.get(RUN_ID_ENV, "").strip()} + run_dir = opt_dir or env_dir + if not run_dir: + return None + return {"run_dir": run_dir, "step": "pytest", "handover": False, "keep": keep, "t0": t0, + "run_id": os.environ.get(RUN_ID_ENV, "").strip()} + + @pytest.hookimpl(tryfirst=True) + def pytest_sessionstart(self, session: "pytest.Session") -> None: + settings = self._settings() + if settings is None: + return + run_dir = settings["run_dir"] + if run_dir == AUTO_RUN_DIR: + import time + + run_dir = default_run_dir(time.time(), self.suite, base=str(self.config.invocation_params.dir)) + self.run = RunRecording(run_dir, step=settings["step"], keep=settings["keep"], + t0=settings["t0"], handover=settings["handover"], + run_id=settings["run_id"]) + set_active_run(self.run) + + # -- per test ----------------------------------------------------------------------------- + + def pytest_runtest_logstart(self, nodeid: str, location: Any) -> None: + if self.run is not None: + self.run.begin_case(nodeid) + + def pytest_runtest_logreport(self, report: Any) -> None: + if self.run is not None: + self._reports.setdefault(report.nodeid, {})[report.when] = report + + def pytest_runtest_logfinish(self, nodeid: str, location: Any) -> None: + run = self.run + if run is None: + return + reports = self._reports.pop(nodeid, {}) + outcome, message, reason = case_outcome(reports) + duration_ms = int(round(sum(float(getattr(r, "duration", 0.0) or 0.0) for r in reports.values()) * 1000)) + last = reports.get("teardown") or reports.get("call") or reports.get("setup") + for key, value in list(getattr(last, "user_properties", []) or []): + if is_plain_number(value): + try: + m = Measurement.make(str(key), value, source="record_property") + except (TypeError, ValueError): + continue + run.add_measurement(nodeid, m) + trace, calls = run.end_case(nodeid, outcome) + self.cases.append(Case( + id=nodeid, step=run.step, kind="pytest", outcome=outcome, duration_ms=duration_ms, + message=message if outcome in ("failed", "error", "xfailed") else "", + skip_reason=reason, attempt=1, trace=trace, + log=_captured_output(reports) if outcome in ("failed", "error") else "", + measurements=run.measurements(nodeid)[:MEASUREMENTS_PER_CASE], calls=calls)) + + def pytest_collectreport(self, report: Any) -> None: + if self.run is None or not report.failed: + return + self.cases.append(Case(id=report.nodeid or "", step=self.run.step, kind="pytest", + outcome="error", message=report.longreprtext or "collection failed")) + + @pytest.hookimpl(trylast=True) + def pytest_runtest_call(self, item: "pytest.Item") -> None: + """After the test body passed: fail it when a ``measure`` value is outside its limits.""" + m = getattr(item, "_embeddedci_measure", None) + if m is None: + return + pending = m.take_unreported() + if pending: + pytest.fail(_limit_text(pending), pytrace=False) + + # -- end ---------------------------------------------------------------------------------- + + @pytest.hookimpl(trylast=True) + def pytest_sessionfinish(self, session: "pytest.Session", exitstatus: int) -> None: + run = self.run + if run is None: + return + run.finish() + tools = provenance.tools_info(pytest.__version__) + if run.handover: + self.record = run.write_part(cases=self.cases, tools=tools) + return + inputs = { + "dut_firmware": provenance.firmware_info(self.config.getoption("benchpod_firmware", None)), + "tests": provenance.git_info(str(self.config.rootpath)), + "pod": run.pod() or empty_pod(), + "wiring": run.wiring, + "tools": tools, + "github": provenance.github_info(), + } + key = {"suite": self.suite, "dut": self.dut, "definition_sha256": ""} + self.record = run.write_run(cases=self.cases, key=key, inputs=inputs, suite=self.suite) + + def pytest_terminal_summary(self, terminalreporter: Any) -> None: + run = self.run + if run is None: + return + if run.handover: + terminalreporter.write_line(f"embeddedci: step {run.step} recorded in {run.run_dir}") + elif self.record is not None: + outcome = self.record["results"]["outcome"] + terminalreporter.write_line(f"embeddedci: run {outcome}, recorded in {run.run_dir}") + + def pytest_unconfigure(self, config: "pytest.Config") -> None: + if self.run is not None and self.run.active: + self.run.finish() diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/recorder.py b/packages/embeddedci/src/embeddedci/benchpod/runs/recorder.py new file mode 100644 index 0000000..aa4258d --- /dev/null +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/recorder.py @@ -0,0 +1,347 @@ +"""The transport recorder: every pod call of a test, timed, into the run's trace. + +:class:`Recorder` wraps one transport *instance*: it binds wrappers as instance attributes for +``command``, ``command_sequence``, ``samples``, ``stream_chunks``, ``status``, +``status_without_caps``, ``target_power``, ``dap_start`` and ``uart_proxy_start``, so the +transport's class (and ``isinstance``) is unchanged and no transport class needs to know about it. +A call made from inside another recorded call (``status`` sending ``command``) is recorded once, +as the outer call. + +When the run it belongs to has ended, each wrapper is one attribute check and a direct call. +""" + +from __future__ import annotations + +import functools +import threading +import time +from typing import TYPE_CHECKING, Any, Callable, Dict, Iterator, List, Optional + +if TYPE_CHECKING: # pragma: no cover + from .active import RunRecording + +#: method name -> trace ``op`` +WRAPPED = { + "command": "command", + "command_sequence": "sequence", + "samples": "samples", + "stream_chunks": "stream", + "status": "status", + "status_without_caps": "status", + "target_power": "power", + "dap_start": "raw", + "uart_proxy_start": "raw", +} + +_tls = threading.local() + + +def _depth() -> int: + return getattr(_tls, "depth", 0) + + +class _Nested: + """Marks a recorded call in progress on this thread, so calls it makes are not recorded again.""" + + def __enter__(self) -> None: + _tls.depth = _depth() + 1 + + def __exit__(self, *exc: object) -> None: + _tls.depth = _depth() - 1 + + +def suppressed() -> _Nested: + """A block whose pod calls are not recorded (the run's own provenance reads).""" + return _Nested() + + +def _cmd_of(req: Any) -> str: + if isinstance(req, dict): + return str(req.get("cmd") or "") + return "" + + +class RecordingLink: + """A raw link (CMSIS-DAP or UART proxy) that counts bytes and logs one ``raw`` trace line when + it closes. Every other attribute is the wrapped link's.""" + + def __init__(self, link: Any, recorder: "Recorder", mode: str, cmd: str, req: Dict[str, Any], + started: float, t: float) -> None: + self._link = link + self._recorder = recorder + self._mode = mode + self._cmd = cmd + self._req = req + self._started = started + self._t = t + self.bytes_in = 0 + self.bytes_out = 0 + self._logged = False + + def read(self, n: int) -> bytes: + data = self._link.read(n) + self.bytes_in += len(data) + return data + + def write(self, data: bytes) -> int: + n = self._link.write(data) + self.bytes_out += int(n if isinstance(n, int) else len(data)) + return n + + def close(self) -> None: + try: + self._link.close() + finally: + self._log() + + def _log(self) -> None: + if self._logged: + return + self._logged = True + err = getattr(self._link, "error", None) + ms = (time.perf_counter() - self._started) * 1000.0 + self._recorder.run.record( + t=self._t, op="raw", cmd=self._cmd, req=self._req, ms=ms, ok=err is None, + error=str(err) if err is not None else "", + extra={"mode": self._mode, "bytes_in": self.bytes_in, "bytes_out": self.bytes_out}, + recorder=self._recorder) + + def __getattr__(self, name: str) -> Any: + return getattr(self._link, name) + + def __enter__(self) -> "RecordingLink": + return self + + def __exit__(self, *exc: object) -> None: + self.close() + + +class Recorder: + """Records one transport instance's calls into a :class:`~.active.RunRecording`.""" + + def __init__(self, run: "RunRecording", transport: Any) -> None: + self.run = run + self.transport = transport + #: The last status report seen (for the run's ``inputs.pod``). + self.last_status: Optional[Dict[str, Any]] = None + self._busy_seen = int(getattr(transport, "busy_retries", 0) or 0) + self._reconnects_seen = int(getattr(transport, "reconnects", 0) or 0) + self.originals: Dict[str, Callable[..., Any]] = {} + #: Names whose original was an instance attribute (put back on uninstall). + self._own: Dict[str, Callable[..., Any]] = {} + + # -- install ------------------------------------------------------------------------------- + + def install(self) -> "Recorder": + for name, op in WRAPPED.items(): + fn = getattr(self.transport, name, None) + if fn is None or not callable(fn): + continue + self.originals[name] = fn + if name in getattr(self.transport, "__dict__", {}): + self._own[name] = fn + if name == "stream_chunks": + wrapper = self._wrap_stream(fn) + elif op == "raw": + wrapper = self._wrap_raw(name, fn) + else: + wrapper = self._wrap_call(name, op, fn) + setattr(self.transport, name, wrapper) + self.transport._embeddedci_recorder = self + return self + + def uninstall(self) -> None: + for name in self.originals: + if name in self._own: + setattr(self.transport, name, self._own[name]) + continue + try: + delattr(self.transport, name) + except AttributeError: + pass + self.originals.clear() + self._own.clear() + if getattr(self.transport, "_embeddedci_recorder", None) is self: + try: + del self.transport._embeddedci_recorder + except AttributeError: + pass + + # -- counters ------------------------------------------------------------------------------ + + def _counters(self) -> None: + """Move the transport's busy-retry and reconnect counters into the run's health.""" + busy = int(getattr(self.transport, "busy_retries", 0) or 0) + if busy > self._busy_seen: + self.run.add_health("busy_retries", busy - self._busy_seen) + self._busy_seen = busy + rec = int(getattr(self.transport, "reconnects", 0) or 0) + if rec > self._reconnects_seen: + self.run.add_health("reconnects", rec - self._reconnects_seen) + self._reconnects_seen = rec + + # -- wrappers ------------------------------------------------------------------------------ + + @staticmethod + def _request(name: str, args: tuple, kwargs: dict) -> Any: + if name in ("command", "samples", "stream_chunks"): + return args[0] if args else kwargs.get("req") + if name == "command_sequence": + return list(args[0] if args else kwargs.get("reqs") or []) + if name == "status": + return {"cmd": "status"} + if name == "status_without_caps": + return {"cmd": "status", "caps": False} + if name == "target_power": + keys = ("efuse", "on", "delay_ms") + req = dict(zip(keys, args)) + req.update({k: v for k, v in kwargs.items() if k in keys}) + return {"cmd": "target_power", **req} + return {} + + def _wrap_call(self, name: str, op: str, fn: Callable[..., Any]) -> Callable[..., Any]: + run = self.run + rec = self + + @functools.wraps(fn) + def wrapper(*args: Any, **kwargs: Any) -> Any: + if not run.active or _depth(): + return fn(*args, **kwargs) + req = rec._request(name, args, kwargs) + if name == "command_sequence": + cmd = _cmd_of(req[0]) if req else "" + else: + cmd = _cmd_of(req) + t = run.now() + start = time.perf_counter() + try: + with _Nested(): + result = fn(*args, **kwargs) + except BaseException as exc: + ms = (time.perf_counter() - start) * 1000.0 + rec._counters() + run.record(t=t, op=op, cmd=cmd, req=req, ms=ms, ok=False, error=_error_text(exc), + recorder=rec) + raise + ms = (time.perf_counter() - start) * 1000.0 + rec._counters() + extra: Dict[str, Any] = {} + resp: Any = result + if op == "samples": + extra["n"] = len(result) if hasattr(result, "__len__") else 0 + if op == "status" and isinstance(result, dict): + rec.last_status = result + if op == "power": + resp = None + run.record(t=t, op=op, cmd=cmd, req=req, ms=ms, ok=True, resp=resp, extra=extra, + recorder=rec) + return result + + return wrapper + + def _wrap_stream(self, fn: Callable[..., Iterator[Dict[str, Any]]]) -> Callable[..., Any]: + run = self.run + rec = self + + @functools.wraps(fn) + def wrapper(*args: Any, **kwargs: Any) -> Any: + if not run.active or _depth(): + return fn(*args, **kwargs) + return rec._stream(fn, args, kwargs) + + return wrapper + + def _stream(self, fn: Callable[..., Iterator[Dict[str, Any]]], args: tuple, + kwargs: dict) -> Iterator[Dict[str, Any]]: + run = self.run + req = self._request("stream_chunks", args, kwargs) + cmd = _cmd_of(req) + t = run.now() + start = time.perf_counter() + keep = run.wants_payloads() + chunks: List[Dict[str, Any]] = [] + n = 0 + count = 0 + ok = False + error = "" + try: + with _Nested(): + it = iter(fn(*args, **kwargs)) + while True: + with _Nested(): + try: + chunk = next(it) + except StopIteration: + break + count += 1 + data = chunk.get("data") if isinstance(chunk, dict) else None + if isinstance(data, list): + n += len(data) + if keep: + chunks.append(chunk) + yield chunk + ok = True + except GeneratorExit: + ok = True # the caller stopped reading early: not a pod error + raise + except BaseException as exc: + error = _error_text(exc) + raise + finally: + ms = (time.perf_counter() - start) * 1000.0 + self._counters() + run.record(t=t, op="stream", cmd=cmd, req=req, ms=ms, ok=ok, error=error, + resp=chunks if keep else None, extra={"n": n, "chunks": count}, recorder=self) + + def _wrap_raw(self, name: str, fn: Callable[..., Any]) -> Callable[..., Any]: + run = self.run + rec = self + mode = "dap" if name == "dap_start" else "uart" + + @functools.wraps(fn) + def wrapper(*args: Any, **kwargs: Any) -> Any: + if not run.active or _depth(): + return fn(*args, **kwargs) + req = _raw_request(name, fn, args, kwargs) + t = run.now() + start = time.perf_counter() + try: + with _Nested(): + link = fn(*args, **kwargs) + except BaseException as exc: + ms = (time.perf_counter() - start) * 1000.0 + run.record(t=t, op="raw", cmd=name, req=req, ms=ms, ok=False, error=_error_text(exc), + extra={"mode": mode, "bytes_in": 0, "bytes_out": 0}, recorder=rec) + raise + return RecordingLink(link, rec, mode, name, req, start, t) + + return wrapper + + +def _raw_request(name: str, fn: Callable[..., Any], args: tuple, kwargs: dict) -> Dict[str, Any]: + import inspect + + out: Dict[str, Any] = {"cmd": name} + try: + bound = inspect.signature(fn).bind_partial(*args, **kwargs) + except (TypeError, ValueError): + return out + for k, v in bound.arguments.items(): + if v is None or isinstance(v, (bool, int, float, str)): + out[k] = v + elif hasattr(v, "to_request"): + try: + out[k] = v.to_request() + except Exception: # noqa: BLE001 + out[k] = str(v) + else: + out[k] = str(v) + return out + + +def _error_text(exc: BaseException) -> str: + text = str(exc) or type(exc).__name__ + code = getattr(exc, "code", "") + if code and isinstance(code, str) and code not in text: + text = f"{code} {text}" + return text diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/schema.py b/packages/embeddedci/src/embeddedci/benchpod/runs/schema.py new file mode 100644 index 0000000..7b21445 --- /dev/null +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/schema.py @@ -0,0 +1,451 @@ +"""The run folder's file formats (schema version 1): ``run.json``, ``junit.xml``, ``parts/.json``. + +The records here are plain data. :mod:`~embeddedci.benchpod.runs.active` fills them while tests run +and writes them at the end. Field names are part of the contract with ``benchpod ci`` and the server: +add fields, never rename them. +""" + +from __future__ import annotations + +import hashlib +import json +import math +import numbers +import os +import re +import tempfile +import xml.etree.ElementTree as ET +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Union + +SCHEMA_VERSION = 1 + +#: Case outcomes, and the order counts are listed in. +CASE_OUTCOMES = ("passed", "failed", "error", "skipped", "xfailed", "xpassed") +#: Run outcomes. +RUN_OUTCOMES = ("passed", "failed", "error", "invalid") +#: Trace keep policies. +KEEP_POLICIES = ("all", "failed", "none") +#: Measurement sources. +MEASUREMENT_SOURCES = ("measure", "native", "record_property", "junit", "openhtf") + +#: Caps (section 2 and 3 of the spec). +MESSAGE_MAX_BYTES = 16 * 1024 +LOG_MAX_BYTES = 64 * 1024 +MEASUREMENTS_PER_CASE = 200 +INLINE_JSON_MAX_BYTES = 4 * 1024 +TRACE_MAX_BYTES_PER_RUN = 50 * 1024 * 1024 +SLUG_MAX_CHARS = 120 + +REDACTED = "" +_REDACT_KEYS = frozenset({"password", "passwd", "psk", "token", "api_key", "apikey", "secret", + "authorization", "wifi_pass"}) +#: ``key`` is redacted only when its value is a string longer than this (a key id stays readable). +_KEY_MIN_SECRET_LEN = 16 + +_SLUG_BAD = re.compile(r"[^A-Za-z0-9._-]") + + +# -- small helpers --------------------------------------------------------------------------------- + +def case_slug(case_id: str) -> str: + """The file name stem for a case id: every character outside ``[A-Za-z0-9._-]`` replaced by + ``_``, at most 120 characters, plus ``-`` when it was cut or changed.""" + slug = _SLUG_BAD.sub("_", case_id) + changed = slug != case_id + if len(slug) > SLUG_MAX_CHARS: + slug = slug[:SLUG_MAX_CHARS] + changed = True + if not slug: + changed = True + if changed: + slug += "-" + hashlib.sha1(case_id.encode("utf-8")).hexdigest()[:8] + return slug + + +def redact(value: Any) -> Any: + """A copy of ``value`` with secrets replaced by ``""``, at any depth. + + Matched keys (case-insensitive): ``password``, ``passwd``, ``psk``, ``token``, ``api_key``, + ``apikey``, ``secret``, ``authorization``, ``wifi_pass``, and ``key`` when its value is a string + longer than 16 characters.""" + if isinstance(value, Mapping): + out: Dict[Any, Any] = {} + for k, v in value.items(): + lk = str(k).lower() + if lk in _REDACT_KEYS or (lk == "key" and isinstance(v, str) and len(v) > _KEY_MIN_SECRET_LEN): + out[k] = REDACTED + else: + out[k] = redact(v) + return out + if isinstance(value, (list, tuple)): + return [redact(v) for v in value] + return value + + +def cap_text(text: str, max_bytes: int, *, keep: str = "head") -> str: + """``text`` cut to at most ``max_bytes`` UTF-8 bytes. ``keep="head"`` keeps the start, + ``"tail"`` the end; a cut text says how much was left out.""" + if not text: + return "" + raw = text.encode("utf-8") + if len(raw) <= max_bytes: + return text + cut = len(raw) - max_bytes + note = f"\n[... {cut} bytes cut ...]\n" + room = max(0, max_bytes - len(note.encode("utf-8"))) + if keep == "tail": + body = raw[len(raw) - room:].decode("utf-8", "ignore") + return note.lstrip("\n") + body + body = raw[:room].decode("utf-8", "ignore") + return body + note.rstrip("\n") + + +def canonical_json(value: Any) -> bytes: + """Sorted keys, no spaces: the bytes the ``sha256`` fields hash.""" + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, + default=str).encode("utf-8") + + +def sha256_of_json(value: Any) -> str: + return hashlib.sha256(canonical_json(value)).hexdigest() + + +def compact_json(value: Any) -> str: + """One-line JSON for trace lines (anything not JSON becomes its ``str``).""" + return json.dumps(value, separators=(",", ":"), ensure_ascii=False, default=_json_default) + + +def _json_default(obj: Any) -> Any: + if isinstance(obj, (bytes, bytearray, memoryview)): + return {"bytes": len(obj)} + if hasattr(obj, "to_dict"): + try: + return obj.to_dict() + except Exception: # noqa: BLE001 - fall back to str + pass + return str(obj) + + +def write_json_atomic(path: Union[str, "os.PathLike[str]"], data: Any) -> None: + """Write ``data`` as JSON to ``path`` through a temporary file in the same folder and a rename, + so a reader never sees half a file.""" + write_bytes_atomic(path, (json.dumps(data, indent=2, ensure_ascii=False, default=_json_default) + + "\n").encode("utf-8")) + + +def write_bytes_atomic(path: Union[str, "os.PathLike[str]"], data: bytes) -> None: + p = Path(path) + p.parent.mkdir(parents=True, exist_ok=True) + fd, tmp = tempfile.mkstemp(prefix=f".{p.name}.", suffix=".tmp", dir=str(p.parent)) + try: + with os.fdopen(fd, "wb") as f: + f.write(data) + os.replace(tmp, p) + except BaseException: + try: + os.unlink(tmp) + except OSError: + pass + raise + + +# -- measurements ---------------------------------------------------------------------------------- + +def parse_drift(drift: Union[str, float, int, None]) -> Optional[float]: + """A drift tolerance as a fraction: ``"20%"`` and ``0.2`` both give ``0.2``; ``None`` stays None.""" + if drift is None: + return None + if isinstance(drift, bool): + raise TypeError("drift must be a fraction (0.2) or a percentage string ('20%'), not a bool") + if isinstance(drift, str): + text = drift.strip() + if not text.endswith("%"): + raise ValueError(f"drift {drift!r}: write a percentage ('20%') or a fraction (0.2)") + try: + value = float(text[:-1].strip()) / 100.0 + except ValueError: + raise ValueError(f"drift {drift!r}: write a percentage ('20%') or a fraction (0.2)") from None + elif isinstance(drift, numbers.Real): + value = float(drift) + else: + raise TypeError(f"drift must be a fraction (0.2) or a percentage string ('20%'), got {drift!r}") + if not math.isfinite(value) or value < 0: + raise ValueError(f"drift {drift!r}: must be a finite, non-negative tolerance") + return value + + +def check_number(value: Any, what: str) -> Union[int, float]: + """``value`` when it is a finite int or float (bool, NaN and infinity are refused).""" + if isinstance(value, bool) or not isinstance(value, numbers.Real): + raise TypeError(f"{what} must be a number (int or float), got {type(value).__name__} {value!r}") + if isinstance(value, numbers.Integral): + return int(value) + f = float(value) + if not math.isfinite(f): + raise ValueError(f"{what} must be a finite number, got {value!r}") + return f + + +def is_plain_number(value: Any) -> bool: + """True for a finite int or float that is not a bool (what ``record_property`` values count).""" + if isinstance(value, bool) or not isinstance(value, numbers.Real): + return False + return math.isfinite(float(value)) + + +@dataclass +class Measurement: + """One named value of a case (spec section 2.2).""" + + name: str + value: Union[int, float] + unit: str = "" + min: Optional[Union[int, float]] = None + max: Optional[Union[int, float]] = None + drift: Optional[float] = None + outcome: str = "recorded" + source: str = "measure" + + @classmethod + def make(cls, name: str, value: Any, unit: str = "", min: Any = None, max: Any = None, + drift: Any = None, source: str = "measure") -> "Measurement": + """Validate and build a measurement; ``outcome`` follows from the limits.""" + if not isinstance(name, str) or not name.strip(): + raise ValueError(f"measurement name must be a non-empty string, got {name!r}") + if source not in MEASUREMENT_SOURCES: + raise ValueError(f"measurement source {source!r} is not one of {', '.join(MEASUREMENT_SOURCES)}") + v = check_number(value, f"measurement {name!r} value") + lo = check_number(min, f"measurement {name!r} min") if min is not None else None + hi = check_number(max, f"measurement {name!r} max") if max is not None else None + if lo is not None and hi is not None and lo > hi: + raise ValueError(f"measurement {name!r}: min {lo} is above max {hi}") + if not isinstance(unit, str): + raise TypeError(f"measurement {name!r} unit must be a string, got {unit!r}") + if lo is None and hi is None: + outcome = "recorded" + elif (lo is not None and v < lo) or (hi is not None and v > hi): + outcome = "failed" + else: + outcome = "passed" + return cls(name=name, value=v, unit=unit, min=lo, max=hi, drift=parse_drift(drift), + outcome=outcome, source=source) + + def describe_failure(self) -> str: + unit = f" {self.unit}" if self.unit else "" + limits = [] + if self.min is not None: + limits.append(f"min {self.min}{unit}") + if self.max is not None: + limits.append(f"max {self.max}{unit}") + return f"{self.name} = {self.value}{unit} is outside the limits ({', '.join(limits)})" + + def to_dict(self) -> Dict[str, Any]: + return {"name": self.name, "value": self.value, "unit": self.unit, "min": self.min, + "max": self.max, "drift": self.drift, "outcome": self.outcome, "source": self.source} + + +# -- calls ----------------------------------------------------------------------------------------- + +@dataclass +class CallStats: + """Per-command timings of one case: ``n``, ``p50_ms``, ``p95_ms``, ``max_ms``, ``errors``.""" + + ms: List[float] = field(default_factory=list) + errors: int = 0 + + def add(self, ms: float, ok: bool) -> None: + self.ms.append(ms) + if not ok: + self.errors += 1 + + def to_dict(self) -> Dict[str, Any]: + data = sorted(self.ms) + return {"n": len(data), "p50_ms": _round_ms(percentile(data, 50)), + "p95_ms": _round_ms(percentile(data, 95)), + "max_ms": _round_ms(data[-1] if data else 0.0), "errors": self.errors} + + +def percentile(sorted_values: Sequence[float], pct: float) -> float: + """Nearest-rank percentile of already sorted values (0 for none).""" + if not sorted_values: + return 0.0 + rank = max(1, math.ceil(pct / 100.0 * len(sorted_values))) + return float(sorted_values[min(rank, len(sorted_values)) - 1]) + + +def _round_ms(ms: float) -> float: + return round(float(ms), 3) + + +# -- cases ----------------------------------------------------------------------------------------- + +@dataclass +class Case: + """One test case (spec section 2.1).""" + + id: str + step: str = "pytest" + kind: str = "pytest" + outcome: str = "passed" + duration_ms: Union[int, float] = 0 + message: str = "" + skip_reason: str = "" + attempt: int = 1 + trace: str = "" + log: str = "" + measurements: List[Measurement] = field(default_factory=list) + calls: Dict[str, CallStats] = field(default_factory=dict) + + def to_dict(self) -> Dict[str, Any]: + return { + "id": self.id, + "step": self.step, + "kind": self.kind, + "outcome": self.outcome, + "duration_ms": self.duration_ms, + "message": cap_text(self.message, MESSAGE_MAX_BYTES), + "skip_reason": self.skip_reason, + "attempt": self.attempt, + "trace": self.trace, + "log": cap_text(self.log, LOG_MAX_BYTES, keep="tail"), + "measurements": [m.to_dict() for m in self.measurements], + "calls": {cmd: s.to_dict() for cmd, s in sorted(self.calls.items())}, + } + + +def count_outcomes(cases: Iterable[Case]) -> Dict[str, int]: + counts = {k: 0 for k in CASE_OUTCOMES} + for c in cases: + if c.outcome in counts: + counts[c.outcome] += 1 + return counts + + +def run_outcome(cases: Iterable[Case], valid: bool = True, step_failed: bool = False) -> str: + """``invalid`` when the run is not valid; else ``error`` when any case is ``error``; else + ``failed`` when any case failed (or a required step failed); else ``passed``.""" + if not valid: + return "invalid" + outcomes = {c.outcome for c in cases} + if "error" in outcomes: + return "error" + if "failed" in outcomes or step_failed: + return "failed" + return "passed" + + +def empty_health() -> Dict[str, Any]: + return {"valid": True, "reasons": [], "lost_samples": 0, "rate_misses": 0, "busy_retries": 0, + "reconnects": 0, "lease_lost": False, "measurements_dropped": 0, "trace_truncated": False} + + +def empty_pod() -> Dict[str, Any]: + return {"name": "", "board": "", "firmware": "", "gateware": 0, "features_rev": 0, + "calibration_sha256": "", "connection": ""} + + +# -- writers --------------------------------------------------------------------------------------- + +def build_run_record(*, run_id: str, key: Mapping[str, Any], trigger: str, started: str, + finished: str, inputs: Mapping[str, Any], health: Mapping[str, Any], + cases: Sequence[Case], duration_ms: Union[int, float], keep: str) -> Dict[str, Any]: + """The ``run.json`` object (spec section 2).""" + return { + "schema_version": SCHEMA_VERSION, + "run_id": run_id, + "key": dict(key), + "trigger": trigger, + "started": started, + "finished": finished, + "inputs": dict(inputs), + "health": dict(health), + "results": { + "outcome": run_outcome(cases, valid=bool(health.get("valid", True))), + "duration_ms": duration_ms, + "counts": count_outcomes(cases), + "cases": [c.to_dict() for c in cases], + }, + "keep": {"traces": keep}, + } + + +def build_part_record(*, step: str, cases: Sequence[Case], health: Mapping[str, Any], + tools: Mapping[str, Any], pod: Optional[Mapping[str, Any]]) -> Dict[str, Any]: + """The ``parts/.json`` object a child process hands to ``benchpod ci`` (spec section 4).""" + part_health = {k: health.get(k, empty_health()[k]) for k in + ("lost_samples", "rate_misses", "busy_retries", "reconnects", + "measurements_dropped", "trace_truncated", "reasons")} + out: Dict[str, Any] = { + "schema_version": SCHEMA_VERSION, + "step": step, + "kind": "pytest", + "cases": [c.to_dict() for c in cases], + "health": part_health, + "tools": {k: tools.get(k, "") for k in ("sdk", "python", "pytest")}, + } + if pod: + out["pod"] = dict(pod) + return out + + +def _split_case_id(case_id: str) -> "tuple[str, str]": + """JUnit ``classname`` and ``name`` for a pytest node id, the way pytest's own junitxml does: + ``tests/test_a.py::TestX::test_y[1]`` -> ``tests.test_a.TestX``, ``test_y[1]``.""" + parts = case_id.split("::") + if len(parts) == 1: + return "", case_id + names = parts[:-1] + path = names[0] + if path.endswith(".py"): + path = path[:-3] + path = path.replace("/", ".").replace("\\", ".") + return ".".join([path] + names[1:]), parts[-1] + + +def _xml_safe(text: str) -> str: + return re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f]", "?", text or "") + + +def junit_xml(suite: str, cases: Sequence[Case], *, timestamp: str = "", + duration_ms: Union[int, float] = 0) -> bytes: + """The cases as JUnit XML: one ```` inside ````.""" + counts = count_outcomes(cases) + root = ET.Element("testsuites") + ts = ET.SubElement(root, "testsuite", { + "name": suite, + "tests": str(len(cases)), + "failures": str(counts["failed"]), + "errors": str(counts["error"]), + "skipped": str(counts["skipped"] + counts["xfailed"]), + "time": f"{float(duration_ms) / 1000.0:.3f}", + }) + if timestamp: + ts.set("timestamp", timestamp) + for case in cases: + classname, name = _split_case_id(case.id) + tc = ET.SubElement(ts, "testcase", {"classname": classname, "name": name, + "time": f"{float(case.duration_ms) / 1000.0:.3f}"}) + if case.measurements: + props = ET.SubElement(tc, "properties") + for m in case.measurements: + ET.SubElement(props, "property", {"name": m.name, "value": str(m.value)}) + message = _xml_safe(cap_text(case.message, MESSAGE_MAX_BYTES)) + first_line = message.strip().splitlines()[-1] if message.strip() else "" + if case.outcome == "failed": + el = ET.SubElement(tc, "failure", {"message": first_line[:500]}) + el.text = message + elif case.outcome == "error": + el = ET.SubElement(tc, "error", {"message": first_line[:500]}) + el.text = message + elif case.outcome == "skipped": + ET.SubElement(tc, "skipped", {"type": "pytest.skip", "message": _xml_safe(case.skip_reason)}) + elif case.outcome == "xfailed": + ET.SubElement(tc, "skipped", {"type": "pytest.xfail", "message": _xml_safe(case.skip_reason)}) + if case.log: + out = ET.SubElement(tc, "system-out") + out.text = _xml_safe(cap_text(case.log, LOG_MAX_BYTES, keep="tail")) + ET.indent(root) + return (b'\n' + + ET.tostring(root, encoding="utf-8", xml_declaration=False) + b"\n") diff --git a/packages/embeddedci/src/embeddedci/benchpod/transport/cloud.py b/packages/embeddedci/src/embeddedci/benchpod/transport/cloud.py index 930a602..c5affd0 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/transport/cloud.py +++ b/packages/embeddedci/src/embeddedci/benchpod/transport/cloud.py @@ -330,6 +330,9 @@ def __init__( # Set by BenchPod once it holds a device lease; sent on tunnel/command requests so the server # confirms this client is the lease holder (and lets concurrent runs serialize). None = none. self.lease_id: "str | None" = None + #: How many times a command was sent again because the pod answered "device busy" (read by + #: the test-run recorder for the run's health). + self.busy_retries = 0 def _session_token(self) -> str: """The cloud session token, minted on first use and renewed shortly before it expires, so @@ -429,6 +432,7 @@ def command(self, req: dict) -> Any: # type: ignore[override] if (payload.get("status") == "error" and str(payload.get("error") or "").strip().lower() == _DEVICE_BUSY and time.monotonic() + _BUSY_RETRY_WAIT < busy_until): + self.busy_retries = getattr(self, "busy_retries", 0) + 1 time.sleep(_BUSY_RETRY_WAIT) continue break diff --git a/packages/embeddedci/tests/runs_fake.py b/packages/embeddedci/tests/runs_fake.py new file mode 100644 index 0000000..b47563a --- /dev/null +++ b/packages/embeddedci/tests/runs_fake.py @@ -0,0 +1,82 @@ +"""A fake LAN pod for the run-recording tests (test_runs.py copies it into pytester suites).""" + +from __future__ import annotations + +from typing import Any, Dict, Iterator, List + +from embeddedci.benchpod.transport.base import Transport + +STATUS = {"version": "3.10.11", "board": "stm32h563", "board_rev": "v3", "analog": True, + "gateware": 58, "features_rev": 0, "caps": ["calibrate", "la_pins"]} +CALIBRATION = {"source": "current_in", "calibrated": True, "offset": 0.0012} + + +class _Link: + def __init__(self) -> None: + self.error = None + self.closed = False + + def read(self, n: int) -> bytes: + return b"x" * min(n, 10) + + def write(self, data: bytes) -> int: + return len(data) + + def close(self) -> None: + self.closed = True + + +class FakePod(Transport): + """A LAN-like pod: status (via command, like the TCP transport), samples, streams, raw links.""" + + def __init__(self, status: Dict[str, Any] = STATUS) -> None: + self.timeout = 30.0 + self.addr = "benchpod-abc123.local:8080" + self.status_reply = dict(status) + self.requests: List[dict] = [] + self.busy_retries = 0 + + def command(self, req: dict) -> Any: + self.requests.append(req) + cmd = req.get("cmd") + if cmd == "status": + return dict(self.status_reply) + if cmd == "calibrate": + return dict(CALIBRATION) + if cmd == "boom": + raise RuntimeError("BP0412 the pod refused") + if cmd == "big": + return {"blob": "y" * 6000} + if cmd == "busy_twice": + self.busy_retries += 2 + return {} + return {"echo": req} + + def command_sequence(self, reqs: List[dict]) -> List[Any]: + return [self.command(r) for r in reqs] + + def samples(self, req: dict) -> List[int]: + return list(range(int(req.get("n", 10)))) + + def stream_chunks(self, req: dict) -> Iterator[Dict[str, Any]]: + yield {"data": [1, 2, 3], "more": True} + yield {"data": [4, 5], "more": False} + + def status(self) -> Any: + return self.command({"cmd": "status"}) + + def status_without_caps(self) -> Any: + return self.command({"cmd": "status", "caps": False}) + + def target_power(self, efuse: int, on: bool, delay_ms: int = 0) -> None: + self.requests.append({"cmd": "target_power", "efuse": efuse, "on": on}) + + def dap_start(self, swclk, swdio, packet_size=None, packet_count=None, wait_ms=None): + return _Link() + + def uart_proxy_start(self, rx, tx, baud, parity="none", stop_bits=1, half_duplex=False, + entry=None, de=None, de_active_low=False): + return _Link() + + def close(self) -> None: + pass diff --git a/packages/embeddedci/tests/test_runs.py b/packages/embeddedci/tests/test_runs.py new file mode 100644 index 0000000..edf2c91 --- /dev/null +++ b/packages/embeddedci/tests/test_runs.py @@ -0,0 +1,666 @@ +"""Test run recording: the run folder formats, the transport recorder, the pytest plugin's run +folder and handover, and the ``measure`` fixture. No hardware: every pod is a fake transport.""" + +from __future__ import annotations + +import json +import xml.etree.ElementTree as ET +from pathlib import Path +from typing import Any, Dict, List + +import pytest + +from embeddedci.benchpod import BenchPod +from embeddedci.benchpod.runs import ( + Case, + Measurement, + RunRecording, + active_run, + case_slug, + junit_xml, + parse_drift, + redact, + run_outcome, + set_active_run, +) +from embeddedci.benchpod.runs import active as active_mod +from embeddedci.benchpod.runs import provenance +from embeddedci.benchpod.runs.schema import CallStats, cap_text, sha256_of_json +from embeddedci.benchpod.transport.base import Transport +from runs_fake import CALIBRATION, FakePod + +RUN_ENV = ("EMBEDDEDCI_RUN_DIR", "EMBEDDEDCI_RUN_STEP", "EMBEDDEDCI_RUN_ID", "EMBEDDEDCI_RUN_T0", + "EMBEDDEDCI_KEEP_TRACES") + + +@pytest.fixture(autouse=True) +def _no_run_env(monkeypatch): + for name in RUN_ENV: + monkeypatch.delenv(name, raising=False) + yield + set_active_run(None) + + +def _lines(path: Path) -> List[Dict[str, Any]]: + return [json.loads(line) for line in path.read_text().splitlines() if line.strip()] + + +# -- schema ---------------------------------------------------------------------------------------- + +def test_case_slug_unchanged_ids_keep_their_name(): + assert case_slug("boot_ms") == "boot_ms" + assert case_slug("a-b.c_d") == "a-b.c_d" + + +def test_case_slug_changed_or_cut_ids_get_a_hash(): + import hashlib + + cid = "tests/test_can_hil.py::test_ack" + h = hashlib.sha1(cid.encode()).hexdigest()[:8] + assert case_slug(cid) == f"tests_test_can_hil.py__test_ack-{h}" + long_id = "a" * 300 + slug = case_slug(long_id) + assert slug.startswith("a" * 120 + "-") and len(slug) == 129 + + +def test_redact_any_depth_case_insensitive(): + data = {"cmd": "wifi_set", "SSID": "lab", "Password": "hunter2", + "nested": [{"api_key": "eci_abc"}, {"Authorization": "Bearer x"}], + "cfg": {"psk": "s", "wifi_pass": "p", "Token": "t", "secret": "z", "passwd": "q", + "apikey": "k"}, + "key": "short", "other": {"key": "0123456789abcdefXYZ"}} + out = redact(data) + assert out["SSID"] == "lab" and out["Password"] == "" + assert out["nested"] == [{"api_key": ""}, {"Authorization": ""}] + assert set(out["cfg"].values()) == {""} + assert out["key"] == "short" # 16 chars or fewer: a key id, kept + assert out["other"]["key"] == "" + assert data["Password"] == "hunter2" # the input is not changed + + +def test_cap_text_head_and_tail(): + text = "a" * 100 + "b" * 100 + head = cap_text(text, 120) + tail = cap_text(text, 120, keep="tail") + assert len(head.encode()) <= 120 and head.startswith("aaa") and "bytes cut" in head + assert len(tail.encode()) <= 120 and tail.endswith("bbb") and "bytes cut" in tail + assert cap_text("short", 120) == "short" + + +@pytest.mark.parametrize("given,want", [("20%", 0.2), (" 15 % ", 0.15), (0.2, 0.2), (0, 0.0), + (None, None)]) +def test_parse_drift(given, want): + got = parse_drift(given) + assert got == pytest.approx(want) if want is not None else got is None + + +@pytest.mark.parametrize("bad", ["20", "abc%", True, -0.1, float("nan"), [1]]) +def test_parse_drift_rejects(bad): + with pytest.raises((TypeError, ValueError)): + parse_drift(bad) + + +def test_measurement_outcomes_and_numbers(): + assert Measurement.make("a", 5).outcome == "recorded" + assert Measurement.make("a", 5, drift="10%").outcome == "recorded" + assert Measurement.make("a", 5, min=1, max=10).outcome == "passed" + assert Measurement.make("a", 11, max=10).outcome == "failed" + assert Measurement.make("a", 0, min=1).outcome == "failed" + m = Measurement.make("settle_ms", 840, unit="ms", max=1000, drift="20%") + assert m.to_dict() == {"name": "settle_ms", "value": 840, "unit": "ms", "min": None, "max": 1000, + "drift": 0.2, "outcome": "passed", "source": "measure"} + for bad in (True, float("nan"), float("inf"), "1", None): + with pytest.raises((TypeError, ValueError)): + Measurement.make("a", bad) + with pytest.raises(ValueError): + Measurement.make("a", 1, min=5, max=1) + with pytest.raises(ValueError): + Measurement.make("", 1) + + +def test_run_outcome_rules(): + def cases(*outcomes): + return [Case(id=str(i), outcome=o) for i, o in enumerate(outcomes)] + + assert run_outcome(cases("passed", "skipped", "xfailed", "xpassed")) == "passed" + assert run_outcome(cases("passed", "failed")) == "failed" + assert run_outcome(cases("failed", "error")) == "error" + assert run_outcome(cases("passed"), valid=False) == "invalid" + assert run_outcome(cases("passed"), step_failed=True) == "failed" + assert run_outcome([]) == "passed" + + +def test_call_stats_percentiles(): + s = CallStats() + for ms in range(1, 101): + s.add(float(ms), ok=ms != 50) + d = s.to_dict() + assert d == {"n": 100, "p50_ms": 50.0, "p95_ms": 95.0, "max_ms": 100.0, "errors": 1} + assert CallStats().to_dict() == {"n": 0, "p50_ms": 0.0, "p95_ms": 0.0, "max_ms": 0.0, "errors": 0} + + +def test_junit_xml(): + cases = [ + Case(id="tests/test_a.py::test_ok", duration_ms=12, + measurements=[Measurement.make("v", 1.5, unit="V")]), + Case(id="tests/test_a.py::TestX::test_bad[1]", outcome="failed", message="assert 1 == 2"), + Case(id="tests/test_a.py::test_err", outcome="error", message="fixture broke"), + Case(id="tests/test_a.py::test_skip", outcome="skipped", skip_reason="no pod"), + Case(id="tests/test_a.py::test_xf", outcome="xfailed", skip_reason="known"), + Case(id="tests/test_a.py::test_log", outcome="failed", message="x", log="printed\x01"), + ] + root = ET.fromstring(junit_xml("hil", cases, duration_ms=1500)) + suite = root.find("testsuite") + assert suite.get("name") == "hil" and suite.get("tests") == "6" + assert (suite.get("failures"), suite.get("errors"), suite.get("skipped")) == ("2", "1", "2") + tcs = suite.findall("testcase") + assert tcs[0].get("classname") == "tests.test_a" and tcs[0].get("name") == "test_ok" + assert tcs[0].find("properties/property").attrib == {"name": "v", "value": "1.5"} + assert tcs[1].get("classname") == "tests.test_a.TestX" and tcs[1].get("name") == "test_bad[1]" + assert tcs[1].find("failure").text == "assert 1 == 2" + assert tcs[2].find("error") is not None + assert tcs[3].find("skipped").get("message") == "no pod" + assert tcs[5].find("system-out").text == "printed?" + + +def test_atomic_write_leaves_no_temp_files(tmp_path): + from embeddedci.benchpod.runs.schema import write_json_atomic + + write_json_atomic(tmp_path / "run.json", {"a": 1}) + write_json_atomic(tmp_path / "run.json", {"a": 2}) + assert json.loads((tmp_path / "run.json").read_text()) == {"a": 2} + assert [p.name for p in tmp_path.iterdir()] == ["run.json"] + + +def test_provenance_helpers(tmp_path): + assert provenance.repo_slug("git@github.com:embeddedci-com/examples.git") == "embeddedci-com/examples" + assert provenance.repo_slug("https://user:tok@github.com/o/r.git") == "o/r" + assert provenance.repo_slug("nonsense") == "" + assert provenance.board_label({"board": "stm32h563", "board_rev": "v3", "analog": False}) == "digital-v3" + assert provenance.board_label({"board": "stm32h563", "board_rev": "unknown"}) == "stm32h563" + assert provenance.board_label({"board_tag": "analog-v3r2"}) == "analog-v3r2" + fw = tmp_path / "app.elf" + fw.write_bytes(b"\x7fELF" + b"\0" * 12) + info = provenance.firmware_info(str(fw)) + assert info["file"] == "app.elf" and info["size"] == 16 and len(info["sha256"]) == 64 + assert provenance.firmware_info(None) == {} + tools = provenance.tools_info("8.0") + assert set(tools) == {"cli", "sdk", "python", "pytest", "os"} and tools["pytest"] == "8.0" + assert provenance.git_info(str(tmp_path)) == {"repo": "", "ref": "", "sha": "", "dirty": False} + + +# -- recorder -------------------------------------------------------------------------------------- + +def test_recorder_traces_calls_per_case(tmp_path): + run = RunRecording(tmp_path / "run") + pod = FakePod() + run.attach(pod) + assert isinstance(pod, FakePod) and isinstance(pod, Transport) + run.begin_case("tests/t.py::test_a") + pod.command({"cmd": "can_write", "id": 1, "password": "hunter2"}) + pod.command({"cmd": "can_write", "id": 2}) + with pytest.raises(RuntimeError): + pod.command({"cmd": "boom"}) + pod.status() # status sends command inside: recorded once + assert pod.samples({"cmd": "la_capture", "n": 5}) == [0, 1, 2, 3, 4] + assert [c["data"] for c in pod.stream_chunks({"cmd": "capture_dual"})] == [[1, 2, 3], [4, 5]] + pod.target_power(1, True) + pod.command_sequence([{"cmd": "ota_begin"}, {"cmd": "ota_end"}]) + link = pod.dap_start(11, 12) + link.write(b"abcd") + assert link.read(100) == b"x" * 10 + link.close() + trace, calls = run.end_case("tests/t.py::test_a", "failed") + run.finish() + + assert trace.startswith("trace/tests_t.py__test_a-") and trace.endswith(".ndjson") + lines = _lines(tmp_path / "run" / trace) + ops = [(ln["op"], ln.get("cmd")) for ln in lines] + assert ops == [("command", "can_write"), ("command", "can_write"), ("command", "boom"), + ("status", "status"), ("samples", "la_capture"), ("stream", "capture_dual"), + ("power", "target_power"), ("sequence", "ota_begin"), ("raw", "dap_start")] + first = lines[0] + assert first["req"]["password"] == "" and first["ok"] is True + assert first["resp"]["echo"]["password"] == "" + assert isinstance(first["t"], float) and first["ms"] >= 0 + assert lines[2]["ok"] is False and "BP0412" in lines[2]["error"] + assert lines[4]["n"] == 5 and lines[4]["resp"] == [0, 1, 2, 3, 4] + assert lines[5]["n"] == 5 and lines[5]["chunks"] == 2 + raw = lines[8] + assert raw["mode"] == "dap" and raw["bytes_out"] == 4 and raw["bytes_in"] == 10 and raw["ok"] is True + assert calls["can_write"].to_dict()["n"] == 2 + assert calls["boom"].to_dict()["errors"] == 1 + assert calls["status"].to_dict()["n"] == 1 + assert "command" not in calls + + +def test_recorder_offloads_big_payloads(tmp_path): + run = RunRecording(tmp_path / "run") + pod = FakePod() + run.attach(pod) + run.begin_case("c") + pod.command({"cmd": "big"}) + pod.samples({"cmd": "la_capture", "n": 5000}) + trace, _ = run.end_case("c", "passed") + lines = _lines(tmp_path / "run" / trace) + assert "resp" not in lines[0] and lines[0]["att"].startswith("att/0001-big.json") + assert lines[1]["att"].startswith("att/0002-la_capture.json") and lines[1]["n"] == 5000 + att = tmp_path / "run" / lines[0]["att"] + data = att.read_bytes() + if att.name.endswith(".zst"): + import zstandard + + data = zstandard.ZstdDecompressor().decompress(data) + assert json.loads(data)["resp"]["blob"] == "y" * 6000 + + +def test_recorder_zstd_is_optional(tmp_path, monkeypatch): + monkeypatch.setattr(active_mod, "_zstd", None) + run = RunRecording(tmp_path / "run") + pod = FakePod() + run.attach(pod) + run.begin_case("c") + pod.command({"cmd": "big"}) + trace, _ = run.end_case("c", "passed") + att = _lines(tmp_path / "run" / trace)[0]["att"] + assert att == "att/0001-big.json" + assert json.loads((tmp_path / "run" / att).read_text())["resp"]["blob"] == "y" * 6000 + + +def test_keep_failed_drops_passed_traces_but_keeps_calls(tmp_path): + run = RunRecording(tmp_path / "run", keep="failed") + pod = FakePod() + run.attach(pod) + run.begin_case("ok") + pod.command({"cmd": "big"}) + trace_ok, calls_ok = run.end_case("ok", "passed") + run.begin_case("bad") + pod.command({"cmd": "x"}) + trace_bad, _ = run.end_case("bad", "failed") + assert trace_ok == "" and calls_ok["big"].to_dict()["n"] == 1 + assert trace_bad and (tmp_path / "run" / trace_bad).exists() + assert not list((tmp_path / "run" / "att").iterdir()) # the passed case's att went too + + +def test_keep_none_writes_nothing(tmp_path): + run = RunRecording(tmp_path / "run", keep="none") + pod = FakePod() + run.attach(pod) + run.begin_case("c") + pod.command({"cmd": "big"}) + trace, calls = run.end_case("c", "failed") + assert trace == "" and calls["big"].to_dict()["n"] == 1 + assert not (tmp_path / "run" / "trace").exists() and not (tmp_path / "run" / "att").exists() + + +def test_trace_cap_truncates_lines(tmp_path): + run = RunRecording(tmp_path / "run", trace_cap=300) + pod = FakePod() + run.attach(pod) + run.begin_case("c") + for i in range(10): + pod.command({"cmd": "can_write", "id": i}) + trace, calls = run.end_case("c", "passed") + lines = _lines(tmp_path / "run" / trace) + assert run.health["trace_truncated"] is True + assert "req" in lines[0] + assert set(lines[-1]) == {"t", "op", "cmd", "ms", "ok"} + assert calls["can_write"].to_dict()["n"] == 10 + + +def test_inactive_run_passes_calls_straight_through(tmp_path): + run = RunRecording(tmp_path / "run") + pod = FakePod() + rec = run.attach(pod) + run.begin_case("c") + run.finish() + pod.command({"cmd": "late"}) + assert not (tmp_path / "run" / "trace").exists() + assert pod.requests[-1] == {"cmd": "late"} + rec.uninstall() + assert "command" not in vars(pod) + + +def test_busy_retries_and_session_calls(tmp_path): + run = RunRecording(tmp_path / "run") + pod = FakePod() + run.attach(pod) + pod.command({"cmd": "busy_twice"}) # outside any case + assert run.health["busy_retries"] == 2 + assert run.session_calls()["busy_twice"].to_dict()["n"] == 1 + assert (tmp_path / "run" / "trace" / "_session.ndjson").exists() + + +def test_uart_proxy_link_and_failed_handshake(tmp_path): + class Refusing(FakePod): + def dap_start(self, *a, **k): + raise RuntimeError("BP0009 no LA voltage") + + run = RunRecording(tmp_path / "run") + pod = Refusing() + run.attach(pod) + run.begin_case("c") + with pod.uart_proxy_start(3, 4, 115200) as link: + link.write(b"hi") + assert link.closed is False # other attributes reach the real link + with pytest.raises(RuntimeError): + pod.dap_start(11, 12) + trace, calls = run.end_case("c", "passed") + lines = _lines(tmp_path / "run" / trace) + assert lines[0]["op"] == "raw" and lines[0]["mode"] == "uart" + assert lines[0]["req"]["baud"] == 115200 and lines[0]["bytes_out"] == 2 + assert lines[1]["mode"] == "dap" and lines[1]["ok"] is False and "BP0009" in lines[1]["error"] + assert calls["dap_start"].to_dict()["errors"] == 1 + + +def test_benchpod_attaches_to_the_active_run(tmp_path): + run = RunRecording(tmp_path / "run") + set_active_run(run) + assert active_run() is run + run.begin_case("tests/t.py::test_pod") + pod = FakePod() + bp = BenchPod(transport=pod, lease=False, api_key="", api_base="") + bp.command({"cmd": "can_write"}) + trace, calls = run.end_case("tests/t.py::test_pod", "passed") + run.finish() + assert active_run() is None + assert "can_write" in calls + block = run.pod() + assert block == {"name": "benchpod-abc123", "board": "analog-v3", "firmware": "3.10.11", + "gateware": 58, "features_rev": 0, + "calibration_sha256": sha256_of_json(CALIBRATION), "connection": "lan"} + # the calibration read is provenance, not part of the test's calls + assert "calibrate" not in calls + + +def test_benchpod_without_a_run_is_not_wrapped(): + pod = FakePod() + BenchPod(transport=pod, lease=False, api_key="", api_base="") + assert "command" not in vars(pod) + + +# -- pytest plugin --------------------------------------------------------------------------------- + +CONFTEST = ''' +import pytest +from embeddedci.benchpod import BenchPod +from runs_fake import FakePod + + +@pytest.fixture +def pod(): + bp = BenchPod(transport=FakePod(), lease=False, api_key="", api_base="") + yield bp + bp.close() +''' + +TESTS = ''' +import pytest + + +def test_pass(pod, measure): + pod.command({"cmd": "can_write"}) + measure("settle_ms", 840, unit="ms", max=1000, drift="20%") + + +def test_fail(pod): + pod.command({"cmd": "can_write"}) + print("output of the failed test") + assert 0x12 == 0x13 + + +def test_limit(measure): + measure("a", 5, max=1) + measure("b", 0, min=1) + measure("c", 3, min=1, max=4) + + +def test_skip(): + pytest.skip("no DUT") + + +@pytest.mark.xfail(reason="known bug") +def test_xfail(): + assert False + + +@pytest.mark.xfail(reason="fixed already") +def test_xpass(): + pass + + +@pytest.fixture +def broken(): + raise RuntimeError("setup broke") + + +def test_setup_error(broken): + pass + + +@pytest.fixture +def bad_teardown(): + yield + raise RuntimeError("teardown broke") + + +def test_teardown_error(bad_teardown): + pass + + +def test_props(record_property): + record_property("vbus_v", 5.02) + record_property("note", "text is ignored") + record_property("flag", True) +''' + + +def _write_suite(pytester, tests: str = TESTS) -> None: + pytester.makepyfile(runs_fake=Path(__file__).with_name("runs_fake.py").read_text()) + pytester.makeconftest(CONFTEST) + pytester.makepyfile(test_suite=tests) + + +def _run_json(path: Path) -> Dict[str, Any]: + return json.loads((path / "run.json").read_text()) + + +def _cases(record: Dict[str, Any]) -> Dict[str, Dict[str, Any]]: + return {c["id"].split("::")[-1]: c for c in record["results"]["cases"]} + + +def test_plugin_writes_a_run_folder(pytester): + _write_suite(pytester) + out = pytester.path / "out" + fw = pytester.path / "app.elf" + fw.write_bytes(b"firmware") + res = pytester.runpytest(f"--embeddedci-run-dir={out}", "--embeddedci-suite=hil", + "--embeddedci-dut=process-node", f"--benchpod-firmware={fw}", "-p", + "no:cacheprovider") + res.assert_outcomes(passed=3, failed=2, skipped=1, xfailed=1, xpassed=1, errors=2) + res.stdout.fnmatch_lines(["*embeddedci: run error, recorded in*"]) + record = _run_json(out) + assert record["schema_version"] == 1 and record["run_id"].startswith("local-") + assert record["key"] == {"suite": "hil", "dut": "process-node", "definition_sha256": ""} + assert record["trigger"] in ("manual", "github", "other") + assert record["started"].endswith("Z") and record["finished"].endswith("Z") + inputs = record["inputs"] + assert set(inputs) == {"dut_firmware", "tests", "pod", "wiring", "tools", "github"} + assert inputs["dut_firmware"]["file"] == "app.elf" and inputs["dut_firmware"]["size"] == 8 + assert inputs["pod"]["firmware"] == "3.10.11" and inputs["pod"]["connection"] == "lan" + assert set(inputs["tools"]) == {"cli", "sdk", "python", "pytest", "os"} + assert set(inputs["github"]) == {"run_id", "run_attempt", "run_url", "workflow", "actor"} + assert set(record["health"]) == {"valid", "reasons", "lost_samples", "rate_misses", "busy_retries", + "reconnects", "lease_lost", "measurements_dropped", "trace_truncated"} + results = record["results"] + assert results["outcome"] == "error" + # pytest counts test_teardown_error as passed and errored; the run counts it once, as error + assert results["counts"] == {"passed": 2, "failed": 2, "error": 2, "skipped": 1, "xfailed": 1, + "xpassed": 1} + assert record["keep"] == {"traces": "all"} + + cases = _cases(record) + assert set(cases["test_pass"]) == {"id", "step", "kind", "outcome", "duration_ms", "message", + "skip_reason", "attempt", "trace", "log", "measurements", "calls"} + ok = cases["test_pass"] + assert ok["step"] == "pytest" and ok["kind"] == "pytest" and ok["attempt"] == 1 + assert ok["outcome"] == "passed" and ok["log"] == "" and ok["message"] == "" + assert ok["measurements"] == [{"name": "settle_ms", "value": 840, "unit": "ms", "min": None, + "max": 1000, "drift": 0.2, "outcome": "passed", "source": "measure"}] + assert ok["calls"]["can_write"]["n"] == 1 and (out / ok["trace"]).exists() + fail = cases["test_fail"] + assert fail["outcome"] == "failed" and "assert" in fail["message"] + assert "output of the failed test" in fail["log"] + limit = cases["test_limit"] + assert limit["outcome"] == "failed" + assert "2 measurement(s) outside their limits" in limit["message"] + assert [m["outcome"] for m in limit["measurements"]] == ["failed", "failed", "passed"] + assert cases["test_skip"]["outcome"] == "skipped" and cases["test_skip"]["skip_reason"] == "no DUT" + assert cases["test_xfail"]["outcome"] == "xfailed" and cases["test_xfail"]["skip_reason"] == "known bug" + assert cases["test_xpass"]["outcome"] == "xpassed" + assert cases["test_setup_error"]["outcome"] == "error" and "setup broke" in cases["test_setup_error"]["message"] + assert cases["test_teardown_error"]["outcome"] == "error" + props = cases["test_props"]["measurements"] + assert props == [{"name": "vbus_v", "value": 5.02, "unit": "", "min": None, "max": None, + "drift": None, "outcome": "recorded", "source": "record_property"}] + + junit = ET.parse(out / "junit.xml").getroot().find("testsuite") + assert junit.get("name") == "hil" and junit.get("tests") == "9" + assert not (out / "parts").exists() + + +def test_plugin_keep_failed_and_message_cap(pytester): + _write_suite(pytester, tests=''' +def test_pass(pod): + pod.command({"cmd": "x"}) + + +def test_fail(pod): + pod.command({"cmd": "x"}) + assert "a" * 40000 == "b" +''') + out = pytester.path / "out" + res = pytester.runpytest(f"--embeddedci-run-dir={out}", "--embeddedci-keep-traces=failed", + "-p", "no:cacheprovider") + res.assert_outcomes(passed=1, failed=1) + record = _run_json(out) + cases = _cases(record) + assert cases["test_pass"]["trace"] == "" and cases["test_pass"]["calls"]["x"]["n"] == 1 + assert (out / cases["test_fail"]["trace"]).exists() + assert len(cases["test_fail"]["message"].encode()) <= 16 * 1024 + assert record["keep"] == {"traces": "failed"} + assert len(list((out / "trace").iterdir())) == 1 + + +def test_plugin_handover_writes_a_part(pytester, monkeypatch): + _write_suite(pytester, tests=''' +def test_pass(pod, measure): + pod.command({"cmd": "x"}) + measure("v", 1.0, unit="V") +''') + out = pytester.path / "ci-run" + monkeypatch.setenv("EMBEDDEDCI_RUN_DIR", str(out)) + monkeypatch.setenv("EMBEDDEDCI_RUN_STEP", "hil") + monkeypatch.setenv("EMBEDDEDCI_RUN_T0", "1000.0") + monkeypatch.setenv("EMBEDDEDCI_KEEP_TRACES", "all") + res = pytester.runpytest("-p", "no:cacheprovider") + res.assert_outcomes(passed=1) + assert not (out / "run.json").exists() and not (out / "junit.xml").exists() + part = json.loads((out / "parts" / "hil.json").read_text()) + assert part["schema_version"] == 1 and part["step"] == "hil" and part["kind"] == "pytest" + assert set(part["health"]) == {"lost_samples", "rate_misses", "busy_retries", "reconnects", + "measurements_dropped", "trace_truncated", "reasons"} + assert set(part["tools"]) == {"sdk", "python", "pytest"} + assert part["pod"]["firmware"] == "3.10.11" + case = part["cases"][0] + assert case["step"] == "hil" and case["measurements"][0]["name"] == "v" + line = _lines(out / case["trace"])[0] + assert line["t"] > 1e9 # counted from EMBEDDEDCI_RUN_T0 + + +def test_plugin_is_off_by_default(pytester): + _write_suite(pytester, tests=''' +def test_pass(pod, measure): + pod.command({"cmd": "x"}) + measure("v", 1.0) +''') + res = pytester.runpytest("-p", "no:cacheprovider") + res.assert_outcomes(passed=1) + assert not (pytester.path / "embeddedci-runs").exists() + assert "embeddedci:" not in res.stdout.str() + + +def test_plugin_env_run_dir_standalone(pytester, monkeypatch): + _write_suite(pytester, tests="def test_a():\n pass\n") + out = pytester.path / "env-run" + monkeypatch.setenv("EMBEDDEDCI_RUN_DIR", str(out)) + monkeypatch.setenv("EMBEDDEDCI_RUN_ID", "local-given") + res = pytester.runpytest("-p", "no:cacheprovider") + res.assert_outcomes(passed=1) + record = _run_json(out) + assert record["run_id"] == "local-given" and record["key"]["suite"] == "pytest" + assert record["inputs"]["pod"]["name"] == "" # no pod connected + + +def test_plugin_auto_run_dir(pytester): + _write_suite(pytester, tests="def test_a():\n pass\n") + res = pytester.runpytest("--embeddedci-run-dir=auto", "--embeddedci-suite=smoke", "-p", + "no:cacheprovider") + res.assert_outcomes(passed=1) + runs = list((pytester.path / "embeddedci-runs").iterdir()) + assert len(runs) == 1 and runs[0].name.endswith("Z-smoke") + assert (runs[0] / "run.json").exists() + + +def test_measure_checks_limits_without_a_run(pytester): + _write_suite(pytester, tests=''' +import pytest + + +def test_limit(measure): + measure("a", 5, max=1) + + +def test_bad_values(measure): + for bad in (True, float("nan"), float("inf")): + with pytest.raises((TypeError, ValueError)): + measure("x", bad) + with pytest.raises(ValueError): + measure("d", 1, drift="lots") + measure("dup", 1) + with pytest.raises(ValueError, match="already recorded"): + measure("dup", 2) + + +def test_after_failure(measure): + measure("a", 5, max=1) + assert False, "the body failed first" +''') + res = pytester.runpytest("-p", "no:cacheprovider", "-rf") + res.assert_outcomes(passed=1, failed=2) + res.stdout.fnmatch_lines(["*a = 5 is outside the limits (max 1)*"]) + assert "error" not in res.parseoutcomes() + + +def test_measure_cap_per_case(pytester): + _write_suite(pytester, tests=''' +def test_many(measure, record_property): + for i in range(205): + measure(f"m{i}", i) + record_property("extra", 1) +''') + out = pytester.path / "out" + res = pytester.runpytest(f"--embeddedci-run-dir={out}", "-p", "no:cacheprovider") + res.assert_outcomes(passed=1) + record = _run_json(out) + assert len(record["results"]["cases"][0]["measurements"]) == 200 + assert record["health"]["measurements_dropped"] == 6 + + +def test_build_report_fixture_still_works(pytester): + _write_suite(pytester, tests=''' +def test_a(build_report): + build_report.record_wiring(target="t.cfg", swclk=11, swdio=12) +''') + res = pytester.runpytest("-p", "no:cacheprovider") + res.assert_outcomes(passed=1)