Skip to content

Upload test runs, build_report on runs (phase 3) - #122

Merged
wardviaene merged 6 commits into
mainfrom
ci-runs-upload
Nov 19, 2026
Merged

wardviaene merged 6 commits into
mainfrom
ci-runs-upload

Conversation

@wardviaene

Copy link
Copy Markdown
Member

What

  • --embeddedci-upload (also the embeddedci_upload ini option and EMBEDDEDCI_UPLOAD=1) uploads the recorded run at session end: POST /api/runs with run.json, then POST /api/runs/{id}/files?name= for junit.xml, the traces of failed and error cases with the attachments they point to, and logs/. The uploaded run.json says so (keep.traces: failed, trace: "" for cases kept local); the folder on disk is unchanged. Without --embeddedci-run-dir it records into ./embeddedci-runs/ first. Credentials: BENCHPOD_API_KEY, else the benchpod login session, else GitHub Actions OIDC (mint_session_token). The run URL is printed; a failed upload is a warning and never changes the result. Under benchpod ci (EMBEDDEDCI_RUN_STEP) the plugin never uploads.
  • embeddedci.benchpod.runs.upload.upload_run(folder, api_base=None, token=None, build_id=None) uploads a folder later.
  • build_report on runs: in GitHub Actions, a session whose collected tests use build_report starts recording by itself; at the end the run is uploaded with inputs.dut_firmware.build_id set to the build the fixture created (the first one), with the reporter's own session token. The /api/cloud/builds/{id}/status call stays, so older servers keep the pass/fail. BuildReporter.api_base and BuildReporter.session_token are new (additive).
  • File names: case slugs never start with a dot, and attachment/part names keep ASCII [A-Za-z0-9._-] only, so every file passes the server's name rule.
  • packages/embeddedci/tests/benchpod-ci.yaml: the SDK's hardware tests (e2e/ and examples/, the make e2e tiers) as one pytest step, dut: none (the DUT tier skips without --benchpod-firmware). Mentioned in CONTRIBUTING and the e2e README.
  • Docs: README (Uploading a run, the option row, build reporting's test results) and CHANGELOG.

Tests

tests/test_runs_upload.py against an in-process fake runs API: only failed traces and their attachments go up, upload order and auth header, a refused upload, the option and the env var, a failed upload is a warning, no upload under benchpod ci, build_report linking the run to its build (local run.json too), nothing recorded outside GitHub Actions, trace/attachment names (long, unicode, leading dot) against the server's rule, and the repo's benchpod-ci.yaml. SDK (1089 passed, bundled-copy check against firmware main), MCP and scripts suites pass.

🤖 Generated with Claude Code

wardviaene and others added 6 commits November 18, 2026 23:25
A case slug never starts with a dot, and attachment and part names keep
ASCII [A-Za-z0-9._-] only (str.isalnum() let other letters through).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--embeddedci-upload (ini embeddedci_upload, EMBEDDEDCI_UPLOAD=1) uploads
the recorded run at session end: POST /api/runs, then junit.xml, the
traces of failed and error cases with their attachments, and logs/.
Credentials: BENCHPOD_API_KEY, else the benchpod login session, else
GitHub Actions OIDC. A failed upload is a warning; under benchpod ci the
runner uploads instead.

In GitHub Actions a session whose tests use build_report records its run
and uploads it with inputs.dut_firmware.build_id set to the build the
fixture created, using the reporter's credentials. The build status call
stays for older servers. BuildReporter gains api_base and session_token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fake runs API: only failed traces and their attachments go up, the
upload order, a refused upload, the plugin option and env, a failed upload
as a warning, no upload under benchpod ci, build_report linking the run
to its build, and trace/attachment names against the server's rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One pytest step over e2e/ and examples/ (the make e2e tiers), dut: none:
the DUT tier skips without --benchpod-firmware. Mentioned in CONTRIBUTING
and the e2e README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BP0026 on a lease joined through BENCHPOD_LEASE asks the run to stop; the
plugin exits the session after the test that saw it (exit code 2, "run
stopped by <by>") and still writes parts/<step>.json with the cases so
far and health lease_lost/valid/reasons. The SDK's own lease keeps going
on without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wardviaene
wardviaene merged commit 5c9728a into main Nov 19, 2026
20 checks passed
@wardviaene wardviaene mentioned this pull request Nov 19, 2026
wardviaene added a commit that referenced this pull request Nov 19, 2026
Test run recording (#118), pod lease on every transport (#119), run upload
and build_report on runs (#122), console log level (#121), runs API (#124).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@wardviaene
wardviaene deleted the ci-runs-upload branch November 20, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant