Skip to content

Test run recording: run folder, trace, measure (phase 1) - #118

Merged
wardviaene merged 4 commits into
mainfrom
ci-runs
Nov 19, 2026
Merged

wardviaene merged 4 commits into
mainfrom
ci-runs

Conversation

@wardviaene

Copy link
Copy Markdown
Member

What

The pytest plugin can now record a whole test run into one folder (opt-in):

  • Run folder (--embeddedci-run-dir DIR, auto, or EMBEDDEDCI_RUN_DIR): run.json (schema version 1, written last and atomically), junit.xml, trace/<case-slug>.ndjson per test, att/ for large payloads (zstd when zstandard is installed). --embeddedci-suite, --embeddedci-dut, --embeddedci-keep-traces all|failed|none. Off by default.
  • Recorder (embeddedci.benchpod.runs): wraps a transport instance (command, command_sequence, samples, stream_chunks, status, status_without_caps, target_power, dap_start, uart_proxy_start), so classes and isinstance are unchanged. Per-case call stats (n, p50, p95, max, errors), redaction at any depth, 4 KB inline limit, 50 MB per-run cap with truncated lines past it, raw links log one raw line with byte counts on close. Nested calls (status sending command) are recorded once. When no run is active nothing is wrapped; after a run ends a wrapper is one flag check.
  • Active run: BenchPod.__init__ attaches to the process's active run right after the transport opens, so pods a test opens itself are recorded too. The pod block (name, board, firmware, gateware, features_rev, calibration sha256, connection kind) is read once, outside any case's trace.
  • Handover for benchpod ci: with EMBEDDEDCI_RUN_DIR and EMBEDDEDCI_RUN_STEP the plugin writes parts/<step>.json and its traces instead of run.json/junit.xml; EMBEDDEDCI_RUN_T0 sets the trace timeline, EMBEDDEDCI_RUN_ID the run id, EMBEDDEDCI_KEEP_TRACES the policy.
  • measure fixture: measure(name, value, unit="", min=None, max=None, drift=None); numbers only (bool, NaN, inf refused), drift as "20%" or 0.2, at most 200 per test (rest counted in health.measurements_dropped). A value outside its limits fails the test after the body (listing every failure), also when no run is recorded. Numeric record_property values become measurements with source record_property.
  • CloudTransport.busy_retries counts "device busy" resends for the run's health.
  • build_report is unchanged.

Tests

pytest -q packages/embeddedci: all new tests pass (tests/test_runs.py, fakes only, including pytester suites for the run folder, handover parts, keep policy, outcome rules, measure limits and cap). Existing SDK, MCP and scripts suites pass. The only failure locally is test_feature_map.py::test_bundled_copy_matches_the_firmware, which compares against a sibling firmware checkout and fails the same way on main (it skips in CI). The zstd attachment path was run with zstandard installed and without it.

No version bump; release later.

🤖 Generated with Claude Code

wardviaene and others added 4 commits November 18, 2026 21:14
New package embeddedci.benchpod.runs:
- schema: run.json / parts / junit.xml writers (atomic), Case, Measurement,
  CallStats, redaction, caps, case slug
- recorder: Recorder wraps a transport instance (command, sequence, samples,
  stream_chunks, status, target_power, dap/uart raw links) into per-case
  NDJSON traces, large payloads to att/ (zstd when available)
- active: RunRecording and the process-wide active run
- provenance: git, tools, GitHub, firmware file and pod blocks

BenchPod attaches to the active run after its transport opens, so pods a
test opens itself are recorded too. CloudTransport counts busy retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--embeddedci-run-dir DIR (or EMBEDDEDCI_RUN_DIR) writes run.json, junit.xml
and per-test traces; off by default. With EMBEDDEDCI_RUN_STEP the plugin
writes parts/<step>.json for benchpod ci instead. --embeddedci-suite,
--embeddedci-dut and --embeddedci-keep-traces set the key and trace policy.

measure(name, value, unit, min, max, drift) records numbers with limits; a
value outside them fails the test. Numeric record_property values are kept
as measurements too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fake transports only: schema fields, redaction, caps and truncation, case
slugs, junit.xml, the recorder's stats and attachments, the parts file under
the benchpod ci handover, measure limits and cap, record_property and the
case outcome rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README: the --embeddedci-* options, the run folder layout, run.json and
trace fields, keep policy, redaction and caps, the measure fixture and
record_property, and the benchpod ci environment handover. CHANGELOG entry
under Unreleased.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wardviaene
wardviaene merged commit a4e0dab into main Nov 19, 2026
20 checks passed
@wardviaene wardviaene mentioned this pull request Nov 19, 2026
wardviaene added a commit that referenced this pull request Nov 19, 2026
Test run recording (#118), pod lease on every transport (#119), run upload
and build_report on runs (#122), console log level (#121), runs API (#124).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@wardviaene
wardviaene deleted the ci-runs branch November 20, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant