diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ffc45c7..780f70d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,7 +13,9 @@ Thanks for helping out. Issues and pull requests are welcome. make test PYTHON=.venv/bin/python ``` -- If you have a BenchPod, run the hardware tier too: `make e2e POD=`. +- If you have a BenchPod, run the hardware tier too: `make e2e POD=`, or record it as a + test run with `cd packages/embeddedci/tests && benchpod ci --pod ` + (`packages/embeddedci/tests/benchpod-ci.yaml`; see `packages/embeddedci/tests/e2e/README.md`). - The 2.x SDK API and MCP tool surface are frozen (snapshot tests fail on any change). Additions are fine; renames and removals wait for a major version. - Keep commits small and focused, and describe what changed and why. diff --git a/packages/embeddedci/CHANGELOG.md b/packages/embeddedci/CHANGELOG.md index 5fcdd58..45f7b73 100644 --- a/packages/embeddedci/CHANGELOG.md +++ b/packages/embeddedci/CHANGELOG.md @@ -6,6 +6,23 @@ `bp.set_log_level(level)` sets it (`LOG_LEVELS`: `error`, `warn`, `info`, `debug`; default `info`, stored on the pod). Firmware newer than 3.10.12 (feature `log_level`); older firmware raises `UnsupportedFeatureError`. +- **Upload test runs:** `--embeddedci-upload` (also the `embeddedci_upload` ini option and + `EMBEDDEDCI_UPLOAD=1`) uploads the recorded run to embeddedci.com when the session ends: + `run.json`, `junit.xml` and the traces of the failed and error tests. Credentials: + `BENCHPOD_API_KEY`, else the `benchpod login` session, else GitHub Actions OIDC. A failed upload + is a warning. Not under `benchpod ci`, which uploads the run itself. + `embeddedci.benchpod.runs.upload.upload_run(folder)` uploads a folder later. +- **`build_report` on test runs:** in GitHub Actions a session whose tests use `build_report` + records its run and uploads it linked to the build the fixture created, so the build shows every + test's result. The fixture still sends the pass/fail to the build for older servers. + `BuildReporter.api_base` and `BuildReporter.session_token` are new. +- Run folder file names always pass the server's rule (`[A-Za-z0-9._-]`, no leading dot): a case + id starting with `.` gets a trace name without it, and attachment names keep ASCII letters only. +- `tests/benchpod-ci.yaml` runs this repository's hardware tests with `benchpod ci`. +- Under `benchpod ci`, a stop of the runner's lease from the web app (`BP0026` on a lease joined + through `BENCHPOD_LEASE`) ends the pytest session after the test that saw it (exit code 2); the + step's `parts/.json` still has the cases so far, `lease_lost` and the reason. With the SDK's + own lease the session goes on without it, as before. - Test run recording: a failed case's `message` starts with the crash line (`ValueError: bad CRC: 6f72`), then a blank line and pytest's full report, still capped at 16 KB. - **Pod lease on the LAN and the USB network:** on a pod with the `lease` feature (features table diff --git a/packages/embeddedci/README.md b/packages/embeddedci/README.md index c3b9ab9..40cfbdf 100644 --- a/packages/embeddedci/README.md +++ b/packages/embeddedci/README.md @@ -508,6 +508,7 @@ The `benchpod` fixture is a `BenchPod` instance, not the module — import const | `--embeddedci-suite` | — | `pytest` | suite name recorded in `run.json` | | `--embeddedci-dut` | — | `""` | name of the board under test recorded in `run.json` | | `--embeddedci-keep-traces` | `EMBEDDEDCI_KEEP_TRACES` | `all` | which pod-call traces to keep: `all`, `failed` or `none` | +| `--embeddedci-upload` | `EMBEDDEDCI_UPLOAD` | off | upload the recorded run to embeddedci.com at the end (also the `embeddedci_upload` ini option), see [Uploading a run](#uploading-a-run) | Connection resolution order: `--benchpod-connection` → `benchpod_connection` ini option → `BENCHPOD_CONNECTION` → `--benchpod-discover`. @@ -624,6 +625,25 @@ test passes to pytest's `record_property` are recorded too (with source `record_ values are left out). A test keeps at most 200 measurements; the rest are dropped and counted in `health.measurements_dropped`. +**Uploading a run.** `--embeddedci-upload` (or `embeddedci_upload = true` in the ini file, or +`EMBEDDEDCI_UPLOAD=1`) uploads the run to embeddedci.com when the session ends, where it shows +next to the build it tested and can be compared with earlier runs. Without `--embeddedci-run-dir` it +records into `./embeddedci-runs/-/` first. What goes up: `run.json`, +`junit.xml`, and the traces (with their attachments) of the failed and error tests only; the +folder on disk keeps what `--embeddedci-keep-traces` says. Credentials: `BENCHPOD_API_KEY`, else +the `benchpod login` session, else the workflow's OIDC token in GitHub Actions (`id-token: write`); +`--benchpod-api-base` / `BENCHPOD_API_BASE` picks the server. The run's URL is printed at the end. +A failed upload is a warning: it never changes the test result. Under `benchpod ci` the plugin does +not upload; the runner uploads the whole run. + +The upload can be done later, or from another machine, with the same code: + +```python +from embeddedci.benchpod.runs.upload import upload_run + +print(upload_run("embeddedci-runs/20261118T140211Z-hil")["url"]) +``` + **Inside `benchpod ci`.** The `benchpod ci` runner hands its run folder to the `pytest` steps it starts through the environment; the plugin then writes `parts/.json` (the cases, health counters, tool versions and pod) and its traces into that folder, and the runner merges them into @@ -2299,7 +2319,9 @@ the lease length. starts), `BenchPod` joins the lease with that id instead of taking its own: on the LAN and the USB network it becomes an owner of the pod's lease, over the cloud it uses the id for the server's lease. Either way it never renews or releases it; the runner does. On a pod without the `lease` -feature it logs one warning and runs without a lease. +feature it logs one warning and runs without a lease. When the runner's lease is stopped from the +web app, the pytest plugin ends the session after the test that saw it (exit code 2, "run stopped +by ..."); the cases so far, `health.lease_lost` and the reason still go to the run folder. Over the cloud, ordinary commands go through the server's command channel rather than the byte tunnel, so they keep working while a UART session holds the tunnel. @@ -2334,7 +2356,18 @@ def test_boots(benchpod, firmware, build_report): Reporting is active only inside GitHub Actions with a mintable session token (`id-token: write`, repo trusted in the web app). Everywhere else the fixture yields an inert reporter -(`build_report.active` is false), so the same test runs unchanged. `--benchpod-build-target` / +(`build_report.active` is false), so the same test runs unchanged. + +**Test results on the build.** When `build_report` reports, the session's tests are also recorded +as a [test run](#test-run-recording) and uploaded at the end, linked to the build the fixture +created (`inputs.dut_firmware.build_id`): every test's outcome, duration, measurements and pod-call +timings, and the traces of the failed tests. The build page shows them under Tests. The fixture +still sends the pass/fail with the build, so older servers keep showing it; current servers keep +the build's own status for the build and show the tests from the run, so a failed test no longer +turns the build red. With several tests using the fixture, the run is linked to the first build. The run is uploaded +with the reporter's own credentials (`build_report.api_base`, `build_report.session_token`). +Recording starts by itself in GitHub Actions when a collected test uses `build_report`; pass +`--embeddedci-run-dir` to choose the folder. `--benchpod-build-target` / `BENCHPOD_BUILD_TARGET` records a platform id. Outside pytest, `make_build_reporter()` returns the same `BuildReporter` / `NoopBuildReporter`. diff --git a/packages/embeddedci/src/embeddedci/benchpod/ci.py b/packages/embeddedci/src/embeddedci/benchpod/ci.py index 5ab9ce4..e57c112 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/ci.py +++ b/packages/embeddedci/src/embeddedci/benchpod/ci.py @@ -94,6 +94,16 @@ def active(self) -> bool: def build_id(self) -> Optional[str]: return self._build_id + @property + def api_base(self) -> str: + """The embeddedci server this reporter talks to.""" + return self._api_base + + @property + def session_token(self) -> str: + """The session token the reporter uses (the test run is uploaded with it too).""" + return self._token + # -- public API used by tests ------------------------------------------------- def record_wiring( @@ -169,7 +179,10 @@ def set_result(self, success: bool, reason: str = "") -> None: self._result = (bool(success), reason) def finalize(self) -> None: - """Create the build if needed and post the final pytest status. Idempotent.""" + """Create the build if needed and post the final pytest status. Idempotent. + + The status call is kept for servers that predate test runs; newer servers keep the build's + own status and show the test results from the run linked to the build instead.""" if self._finalized: return self._finalized = True diff --git a/packages/embeddedci/src/embeddedci/benchpod/pod_lease.py b/packages/embeddedci/src/embeddedci/benchpod/pod_lease.py index e2fac32..ec00358 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/pod_lease.py +++ b/packages/embeddedci/src/embeddedci/benchpod/pod_lease.py @@ -152,6 +152,9 @@ def _mark_lost(self, by: str) -> bool: run = active_run() if run is not None: run.lease_lost(by) + if self.joined: + # The runner's lease (benchpod ci) was stopped: the whole run ends. + run.request_stop(f"run stopped by {by or 'someone'}: the pod's lease was stopped") except Exception: # noqa: BLE001 - never fail the session over the run record pass return True diff --git a/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py b/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py index 7f7b00d..8bb05f6 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py +++ b/packages/embeddedci/src/embeddedci/benchpod/pytest_plugin.py @@ -533,6 +533,10 @@ def test_boots(benchpod, firmware, build_report): ... # the pytest pass/fail is captured automatically The build status (pass/fail) is captured automatically from the test result at teardown. + + The session's test run is recorded too and uploaded at the end, linked to the build: every + test's outcome, measurements and pod-call timings, and the traces of the failed tests (see + ``--embeddedci-upload``). The build page shows it under Tests. """ from .ci import make_build_reporter @@ -556,6 +560,11 @@ def test_boots(benchpod, firmware, build_report): if log_text: reporter.upload_logs("pytest.log", log_text) reporter.finalize() + # The session's test run is uploaded at the end, linked to this build. + runs = request.config.pluginmanager.get_plugin(_runs.PLUGIN_NAME) + build_id = getattr(reporter, "build_id", None) + if runs is not None and getattr(reporter, "active", False) and build_id: + runs.link_build(build_id, reporter.api_base, reporter.session_token) def _collect_pytest_log(node: "pytest.Item") -> str: diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/active.py b/packages/embeddedci/src/embeddedci/benchpod/runs/active.py index fcf04aa..6a57322 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/runs/active.py +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/active.py @@ -102,8 +102,16 @@ def default_run_dir(started: float, suite: str = "", base: Union[str, "os.PathLi return Path(base) / "embeddedci-runs" / f"{folder_stamp(started)}-{case_slug_simple(suite or 'pytest')}" +_SIMPLE_OK = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._-") + + def case_slug_simple(text: str) -> str: - return "".join(c if c.isalnum() or c in "._-" else "_" for c in text)[:80] or "pytest" + """``text`` as a short file name part: ASCII ``[A-Za-z0-9._-]`` only (the server's file name + rule), every other character ``_``, no leading dot, at most 80 characters.""" + out = "".join(c if c in _SIMPLE_OK else "_" for c in text)[:80] + if out.startswith("."): + out = "_" + out[1:] + return out or "pytest" @dataclass @@ -140,6 +148,8 @@ def __init__(self, run_dir: Union[str, "os.PathLike[str]"], *, step: str = "pyte self.handover = handover #: The suite name (``key.suite``); a pod lease taken during the run uses it as purpose. self.suite = "" + #: Why the run must end now (the runner's lease was stopped), or "". + self.stop_reason = "" self.started = time.time() self.t0 = float(t0) if t0 is not None else self.started self.run_id = run_id or new_run_id(self.started) @@ -181,6 +191,12 @@ def lease_lost(self, by: str = "") -> None: self.health["lease_lost"] = True self.add_reason(f"the pod lease was stopped by {by or 'someone'} during the run") + def request_stop(self, reason: str) -> None: + """End the run after the current test (the pytest plugin exits the session).""" + with self._lock: + if not self.stop_reason: + self.stop_reason = reason + def add_reason(self, reason: str, *, invalid: bool = True) -> None: """A note on the run's health; ``invalid`` (default) marks the run as not a valid measurement.""" with self._lock: diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/pytest_runs.py b/packages/embeddedci/src/embeddedci/benchpod/runs/pytest_runs.py index 1bb610b..e9f419a 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/runs/pytest_runs.py +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/pytest_runs.py @@ -38,6 +38,8 @@ ) PLUGIN_NAME = "embeddedci-runs" +#: ``EMBEDDEDCI_UPLOAD=1`` uploads the recorded run (like ``--embeddedci-upload``). +UPLOAD_ENV = "EMBEDDEDCI_UPLOAD" #: ``--embeddedci-run-dir auto``: the default folder, ``./embeddedci-runs/-/``. AUTO_RUN_DIR = "auto" @@ -59,6 +61,15 @@ def add_options(parser: "pytest.Parser") -> None: "--embeddedci-dut", action="store", default=None, dest="embeddedci_dut", help="Name of the board under test recorded in run.json (key.dut; default '').", ) + group.addoption( + "--embeddedci-upload", action="store_true", default=False, dest="embeddedci_upload", + help="Upload the recorded run to embeddedci.com at the end (records into " + "./embeddedci-runs/ when --embeddedci-run-dir is not given). Credentials: BENCHPOD_API_KEY, " + "else the benchpod login session, else GitHub Actions OIDC. Also the embeddedci_upload ini " + "option and EMBEDDEDCI_UPLOAD=1.", + ) + parser.addini("embeddedci_upload", type="bool", default=False, + help="Upload the recorded run to embeddedci.com (see --embeddedci-upload).") group.addoption( "--embeddedci-keep-traces", action="store", default=None, dest="embeddedci_keep_traces", choices=list(KEEP_POLICIES), @@ -67,6 +78,13 @@ def add_options(parser: "pytest.Parser") -> None: ) +def _ini_bool(config: Any, name: str) -> bool: + try: + return bool(config.getini(name)) + except (ValueError, KeyError): + return False + + # -- measure ------------------------------------------------------------------------------------- class Measure: @@ -220,6 +238,13 @@ def __init__(self, config: "pytest.Config") -> None: self.dut = "" self.record: Optional[Dict[str, Any]] = None self._reports: Dict[str, Dict[str, Any]] = {} + #: Upload the run at the end (--embeddedci-upload, the ini option, EMBEDDEDCI_UPLOAD=1). + self.upload = False + #: The build a build_report fixture created: (build id, api base, session token). + self.build_link: Optional[Tuple[str, str, str]] = None + #: The upload's result ({"id", "url", "build_id"}) or the warning it ended with. + self.uploaded: Optional[Dict[str, Any]] = None + self.upload_warning = "" # -- start -------------------------------------------------------------------------------- @@ -236,15 +261,19 @@ def _settings(self) -> Optional[Dict[str, Any]]: or os.environ.get(KEEP_TRACES_ENV, "").strip().lower() or "all") if keep not in KEEP_POLICIES: raise pytest.UsageError(f"{KEEP_TRACES_ENV} must be one of {', '.join(KEEP_POLICIES)}, got {keep!r}") + self.upload = bool(config.getoption("embeddedci_upload", False) + or _ini_bool(config, "embeddedci_upload") + or os.environ.get(UPLOAD_ENV, "").strip().lower() in ("1", "true", "yes")) t0_text = os.environ.get(RUN_T0_ENV, "").strip() try: t0 = float(t0_text) if t0_text else None except ValueError: raise pytest.UsageError(f"{RUN_T0_ENV} must be unix seconds, got {t0_text!r}") from None if env_dir and env_step: + self.upload = False # benchpod ci uploads the whole run itself return {"run_dir": env_dir, "step": env_step, "handover": True, "keep": keep, "t0": t0, "run_id": os.environ.get(RUN_ID_ENV, "").strip()} - run_dir = opt_dir or env_dir + run_dir = opt_dir or env_dir or (AUTO_RUN_DIR if self.upload else "") if not run_dir: return None return {"run_dir": run_dir, "step": "pytest", "handover": False, "keep": keep, "t0": t0, @@ -253,8 +282,34 @@ def _settings(self) -> Optional[Dict[str, Any]]: @pytest.hookimpl(tryfirst=True) def pytest_sessionstart(self, session: "pytest.Session") -> None: settings = self._settings() - if settings is None: + if settings is not None: + self._start(settings) + + def pytest_collection_finish(self, session: "pytest.Session") -> None: + """A suite whose tests use ``build_report`` records its run in GitHub Actions (when nothing + else asked for it), so the run can be uploaded linked to the build the fixture reports.""" + if self.run is not None or hasattr(self.config, "workerinput"): + return + if os.environ.get(RUN_STEP_ENV, "").strip(): + return + from ..ci import running_in_github_actions + + if not running_in_github_actions(): return + if not any("build_report" in getattr(item, "fixturenames", ()) for item in session.items): + return + self._settings() + self._start({"run_dir": AUTO_RUN_DIR, "step": "pytest", "handover": False, + "keep": os.environ.get(KEEP_TRACES_ENV, "").strip().lower() or "failed", + "t0": None, "run_id": ""}) + + def link_build(self, build_id: str, api_base: str, token: str) -> None: + """The ``build_report`` fixture created ``build_id``: link the run to it (the first one) and + upload it at the end with the fixture's credentials.""" + if build_id and self.build_link is None: + self.build_link = (build_id, api_base, token) + + def _start(self, settings: Dict[str, Any]) -> None: run_dir = settings["run_dir"] if run_dir == AUTO_RUN_DIR: import time @@ -298,6 +353,18 @@ def pytest_runtest_logfinish(self, nodeid: str, location: Any) -> None: skip_reason=reason, attempt=1, trace=trace, log=_captured_output(reports) if outcome in ("failed", "error") else "", measurements=run.measurements(nodeid)[:MEASUREMENTS_PER_CASE], calls=calls)) + self._exit_if_stopped() + + def _exit_if_stopped(self) -> None: + """The runner's lease was stopped from the web app: end the session now. The cases so far + and the run's health still go to the run folder (``pytest_sessionfinish``).""" + run = self.run + if run is not None and run.stop_reason: + pytest.exit(run.stop_reason, returncode=pytest.ExitCode.INTERRUPTED) + + @pytest.hookimpl(tryfirst=True) + def pytest_runtest_setup(self, item: "pytest.Item") -> None: + self._exit_if_stopped() def pytest_collectreport(self, report: Any) -> None: if self.run is None or not report.failed: @@ -335,8 +402,32 @@ def pytest_sessionfinish(self, session: "pytest.Session", exitstatus: int) -> No "tools": tools, "github": provenance.github_info(), } + if self.build_link is not None: + inputs["dut_firmware"] = {**inputs["dut_firmware"], "build_id": self.build_link[0]} key = {"suite": self.suite, "dut": self.dut, "definition_sha256": ""} self.record = run.write_run(cases=self.cases, key=key, inputs=inputs, suite=self.suite) + if self.upload or self.build_link is not None: + self._upload(run) + + def _upload(self, run: RunRecording) -> None: + from .upload import RunUploadError, upload_run + + problems: List[str] = [] + api_base = (self.config.getoption("benchpod_api_base", None) + or os.environ.get("BENCHPOD_API_BASE") or None) + token = None + build_id = None + if self.build_link is not None: + build_id, base, token = self.build_link + api_base = api_base or base + try: + self.uploaded = upload_run(run.run_dir, api_base=api_base, token=token, + build_id=build_id, warn=problems.append) + except RunUploadError as exc: + self.upload_warning = str(exc) + return + if problems: + self.upload_warning = "some files were not uploaded: " + "; ".join(problems[:5]) def pytest_terminal_summary(self, terminalreporter: Any) -> None: run = self.run @@ -347,6 +438,10 @@ def pytest_terminal_summary(self, terminalreporter: Any) -> None: elif self.record is not None: outcome = self.record["results"]["outcome"] terminalreporter.write_line(f"embeddedci: run {outcome}, recorded in {run.run_dir}") + if self.uploaded is not None: + terminalreporter.write_line(f"embeddedci: run uploaded: {self.uploaded.get('url') or self.uploaded.get('id')}") + if self.upload_warning: + terminalreporter.write_line(f"embeddedci: warning: run upload: {self.upload_warning}", yellow=True) def pytest_unconfigure(self, config: "pytest.Config") -> None: if self.run is not None and self.run.active: diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/schema.py b/packages/embeddedci/src/embeddedci/benchpod/runs/schema.py index 7b21445..bbfa22b 100644 --- a/packages/embeddedci/src/embeddedci/benchpod/runs/schema.py +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/schema.py @@ -51,8 +51,11 @@ def case_slug(case_id: str) -> str: """The file name stem for a case id: every character outside ``[A-Za-z0-9._-]`` replaced by - ``_``, at most 120 characters, plus ``-`` when it was cut or changed.""" + ``_`` (a leading ``.`` too), at most 120 characters, plus ``-`` when it + was cut or changed.""" slug = _SLUG_BAD.sub("_", case_id) + if slug.startswith("."): # a file name may not start with a dot (hidden; the server refuses it) + slug = "_" + slug[1:] changed = slug != case_id if len(slug) > SLUG_MAX_CHARS: slug = slug[:SLUG_MAX_CHARS] @@ -377,6 +380,11 @@ def build_part_record(*, step: str, cases: Sequence[Case], health: Mapping[str, part_health = {k: health.get(k, empty_health()[k]) for k in ("lost_samples", "rate_misses", "busy_retries", "reconnects", "measurements_dropped", "trace_truncated", "reasons")} + # Not in the spec's part shape, honored by the runner: a lost lease invalidates the whole run. + if health.get("lease_lost"): + part_health["lease_lost"] = True + if health.get("valid") is False: + part_health["valid"] = False out: Dict[str, Any] = { "schema_version": SCHEMA_VERSION, "step": step, diff --git a/packages/embeddedci/src/embeddedci/benchpod/runs/upload.py b/packages/embeddedci/src/embeddedci/benchpod/runs/upload.py new file mode 100644 index 0000000..0c2410b --- /dev/null +++ b/packages/embeddedci/src/embeddedci/benchpod/runs/upload.py @@ -0,0 +1,167 @@ +"""Upload a run folder to embeddedci.com: ``POST /api/runs`` with ``run.json``, then its files with +``POST /api/runs/{id}/files?name=``. + +What goes up: ``run.json``, ``junit.xml``, the traces of the failed and error cases (the ``failed`` +keep policy, whatever the folder kept locally) with the attachments those traces point to, and +``logs/``. The uploaded ``run.json`` says so: its ``keep.traces`` is ``failed`` (or ``none``) and the +cases whose trace stays local have ``trace: ""``. The folder on disk is not changed. +""" + +from __future__ import annotations + +import json +import os +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path +from typing import Any, Callable, Dict, Iterable, List, Optional, Set, Union + +from embeddedci import _http + +from ..errors import BenchPodError + +_HTTP_TIMEOUT = 60.0 +#: The case outcomes whose traces are uploaded. +_UPLOADED_TRACES = frozenset({"failed", "error"}) + + +class RunUploadError(BenchPodError): + """Uploading a run failed (no credentials, the server refused, the network).""" + + +def upload_files(run_dir: Path, record: Dict[str, Any]) -> List[str]: + """The files of ``run_dir`` to upload next to ``record``, as names relative to the folder.""" + names: List[str] = [] + if (run_dir / "junit.xml").is_file(): + names.append("junit.xml") + for case in record.get("results", {}).get("cases", []) or []: + trace = case.get("trace") or "" + if not trace or case.get("outcome") not in _UPLOADED_TRACES: + continue + path = run_dir / trace + if not path.is_file(): + continue + names.append(trace) + names.extend(sorted(_attachments(path) - set(names))) + logs = run_dir / "logs" + if logs.is_dir(): + names.extend(f"logs/{p.name}" for p in sorted(logs.iterdir()) if p.is_file()) + return [n for n in names if (run_dir / n).is_file()] + + +def _attachments(trace: Path) -> Set[str]: + out: Set[str] = set() + try: + with trace.open("r", encoding="utf-8") as f: + for line in f: + if '"att"' not in line: + continue + try: + att = json.loads(line).get("att") + except ValueError: + continue + if isinstance(att, str) and att.startswith("att/"): + out.add(att) + except OSError: + pass + return out + + +def upload_record(record: Dict[str, Any], *, build_id: Optional[str] = None) -> Dict[str, Any]: + """The ``run.json`` as uploaded: traces only for failed and error cases, and ``build_id`` (when + given) in ``inputs.dut_firmware`` so the server links the run to that build.""" + doc = json.loads(json.dumps(record)) + keep = (doc.get("keep") or {}).get("traces", "all") + if keep == "all": + doc["keep"] = {"traces": "failed"} + for case in doc.get("results", {}).get("cases", []) or []: + if case.get("outcome") not in _UPLOADED_TRACES: + case["trace"] = "" + if build_id: + inputs = doc.setdefault("inputs", {}) + firmware = inputs.get("dut_firmware") + if not isinstance(firmware, dict): + firmware = {} + firmware["build_id"] = str(build_id) + inputs["dut_firmware"] = firmware + return doc + + +def _token(api_base: str) -> str: + """A credential for the runs API: ``BENCHPOD_API_KEY``, else the ``benchpod login`` session, + else GitHub Actions OIDC (see :func:`~embeddedci.benchpod.cloud_auth.mint_session_token`).""" + from ..cloud_auth import mint_session_token + + try: + return mint_session_token(api_base)[0] + except BenchPodError as exc: + raise RunUploadError(f"no credentials to upload the run: {exc}") from exc + + +def _request(url: str, token: str, body: bytes, content_type: str) -> Dict[str, Any]: + from ..cloud_auth import USER_AGENT + + req = urllib.request.Request(url, data=body, method="POST", headers={ + "Authorization": f"Bearer {token}", "Content-Type": content_type, + "Accept": "application/json", "User-Agent": USER_AGENT}) + try: + with _http.urlopen(req, timeout=_HTTP_TIMEOUT) as resp: + raw = resp.read().decode("utf-8") + except urllib.error.HTTPError as exc: + msg = exc.read().decode("utf-8", "replace") + try: + msg = json.loads(msg).get("error", msg) + except ValueError: + pass + raise RunUploadError(f"run upload failed (HTTP {exc.code}): {msg}") from exc + except Exception as exc: # noqa: BLE001 - network, TLS + raise RunUploadError(f"run upload failed: {exc}") from exc + try: + data = json.loads(raw) if raw.strip() else {} + except ValueError: + data = {} + return data if isinstance(data, dict) else {} + + +def upload_run(run_dir: Union[str, "os.PathLike[str]"], *, api_base: Optional[str] = None, + token: Optional[str] = None, build_id: Optional[str] = None, + warn: Optional[Callable[[str], None]] = None) -> Dict[str, Any]: + """Upload the run folder ``run_dir``. Returns the server's ``{"id", "url", "build_id"}``. + + ``token`` is the credential (default: ``BENCHPOD_API_KEY``, else the ``benchpod login`` + session, else GitHub Actions OIDC); ``api_base`` the server (default ``BENCHPOD_API_BASE``, + else ``https://www.embeddedci.com``). A file the server refuses is reported to ``warn`` and + skipped; a refused ``run.json`` raises :class:`RunUploadError`.""" + from ..cloud_auth import DEFAULT_API_BASE + + folder = Path(run_dir) + try: + record = json.loads((folder / "run.json").read_text(encoding="utf-8")) + except (OSError, ValueError) as exc: + raise RunUploadError(f"{folder}: no readable run.json ({exc})") from exc + base = (api_base or os.environ.get("BENCHPOD_API_BASE") or DEFAULT_API_BASE).rstrip("/") + tok = token or _token(base) + doc = upload_record(record, build_id=build_id) + created = _request(f"{base}/api/runs", tok, json.dumps(doc).encode("utf-8"), "application/json") + run_id = str(created.get("id") or "") + if not run_id: + raise RunUploadError("run upload: the server returned no run id") + for name in upload_files(folder, doc): + url = f"{base}/api/runs/{urllib.parse.quote(run_id, safe='')}/files?name={urllib.parse.quote(name, safe='')}" + try: + _request(url, tok, (folder / name).read_bytes(), "application/octet-stream") + except (RunUploadError, OSError) as exc: + if warn is not None: + warn(f"{name}: {exc}") + return created + + +def names_valid(names: Iterable[str]) -> bool: + """Whether every name passes the server's file name rule (``run.json``, ``junit.xml``, or one + level of ``trace/``, ``att/``, ``logs/`` with ``[A-Za-z0-9._-]`` names not starting with a dot, + at most 200 characters).""" + import re + + rule = re.compile(r"^(?:run\.json|junit\.xml|(?:trace|att|logs)/[A-Za-z0-9_-][A-Za-z0-9._-]*)$") + return all(len(n) <= 200 and rule.match(n) for n in names) diff --git a/packages/embeddedci/tests/api_surface.json b/packages/embeddedci/tests/api_surface.json index 42726b2..4a4e04f 100644 --- a/packages/embeddedci/tests/api_surface.json +++ b/packages/embeddedci/tests/api_surface.json @@ -204,9 +204,11 @@ "BuildReporter": { "__init__": "(self, *, api_base: 'str', session_token: 'str', metadata: 'Dict[str, str]', target: 'str' = '', name: 'str' = '') -> 'None'", "active": "property", + "api_base": "property", "build_id": "property", "finalize": "(self) -> 'None'", "record_wiring": "(self, *, target: 'Optional[str]' = None, swclk: 'Optional[int]' = None, swdio: 'Optional[int]' = None, nreset: 'Optional[bool]' = None, efuse: 'Optional[int]' = None) -> 'None'", + "session_token": "property", "set_result": "(self, success: 'bool', reason: 'str' = '') -> 'None'", "upload_artifact": "(self, path: 'str') -> 'None'", "upload_artifacts": "(self, paths: 'Iterable[str]') -> 'None'", diff --git a/packages/embeddedci/tests/benchpod-ci.yaml b/packages/embeddedci/tests/benchpod-ci.yaml new file mode 100644 index 0000000..b69ad07 --- /dev/null +++ b/packages/embeddedci/tests/benchpod-ci.yaml @@ -0,0 +1,23 @@ +# The SDK's own hardware tests with `benchpod ci` (needs a benchpod-cli that has `ci`): +# +# cd packages/embeddedci/tests && benchpod ci --pod +# +# The same tests as `make e2e POD=...` from the repository root: the pod, GPIO, rev3, digital and +# USB tiers of e2e/ and the examples. Without --benchpod-firmware the DUT tier (test_e2e_dut.py) +# skips, so the suite needs no DUT firmware; add `--benchpod-firmware=` to `args` (and name +# the board in `dut`) to run it. The bench wiring comes from BENCHPOD_E2E_* (see e2e/README.md). +# Results land in ./embeddedci-runs/