Repository navigation
Test run recording: run folder, trace, measure (phase 1) - #118
Merged
Merged
Conversation
New package embeddedci.benchpod.runs: - schema: run.json / parts / junit.xml writers (atomic), Case, Measurement, CallStats, redaction, caps, case slug - recorder: Recorder wraps a transport instance (command, sequence, samples, stream_chunks, status, target_power, dap/uart raw links) into per-case NDJSON traces, large payloads to att/ (zstd when available) - active: RunRecording and the process-wide active run - provenance: git, tools, GitHub, firmware file and pod blocks BenchPod attaches to the active run after its transport opens, so pods a test opens itself are recorded too. CloudTransport counts busy retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--embeddedci-run-dir DIR (or EMBEDDEDCI_RUN_DIR) writes run.json, junit.xml and per-test traces; off by default. With EMBEDDEDCI_RUN_STEP the plugin writes parts/<step>.json for benchpod ci instead. --embeddedci-suite, --embeddedci-dut and --embeddedci-keep-traces set the key and trace policy. measure(name, value, unit, min, max, drift) records numbers with limits; a value outside them fails the test. Numeric record_property values are kept as measurements too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fake transports only: schema fields, redaction, caps and truncation, case slugs, junit.xml, the recorder's stats and attachments, the parts file under the benchpod ci handover, measure limits and cap, record_property and the case outcome rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README: the --embeddedci-* options, the run folder layout, run.json and trace fields, keep policy, redaction and caps, the measure fixture and record_property, and the benchpod ci environment handover. CHANGELOG entry under Unreleased. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The pytest plugin can now record a whole test run into one folder (opt-in):
--embeddedci-run-dir DIR,auto, orEMBEDDEDCI_RUN_DIR):run.json(schema version 1, written last and atomically),junit.xml,trace/<case-slug>.ndjsonper test,att/for large payloads (zstd whenzstandardis installed).--embeddedci-suite,--embeddedci-dut,--embeddedci-keep-traces all|failed|none. Off by default.embeddedci.benchpod.runs): wraps a transport instance (command, command_sequence, samples, stream_chunks, status, status_without_caps, target_power, dap_start, uart_proxy_start), so classes andisinstanceare unchanged. Per-case call stats (n, p50, p95, max, errors), redaction at any depth, 4 KB inline limit, 50 MB per-run cap with truncated lines past it, raw links log onerawline with byte counts on close. Nested calls (status sending command) are recorded once. When no run is active nothing is wrapped; after a run ends a wrapper is one flag check.BenchPod.__init__attaches to the process's active run right after the transport opens, so pods a test opens itself are recorded too. The pod block (name, board, firmware, gateware, features_rev, calibration sha256, connection kind) is read once, outside any case's trace.benchpod ci: withEMBEDDEDCI_RUN_DIRandEMBEDDEDCI_RUN_STEPthe plugin writesparts/<step>.jsonand its traces instead ofrun.json/junit.xml;EMBEDDEDCI_RUN_T0sets the trace timeline,EMBEDDEDCI_RUN_IDthe run id,EMBEDDEDCI_KEEP_TRACESthe policy.measurefixture:measure(name, value, unit="", min=None, max=None, drift=None); numbers only (bool, NaN, inf refused),driftas"20%"or0.2, at most 200 per test (rest counted inhealth.measurements_dropped). A value outside its limits fails the test after the body (listing every failure), also when no run is recorded. Numericrecord_propertyvalues become measurements with sourcerecord_property.CloudTransport.busy_retriescounts "device busy" resends for the run's health.build_reportis unchanged.Tests
pytest -q packages/embeddedci: all new tests pass (tests/test_runs.py, fakes only, including pytester suites for the run folder, handover parts, keep policy, outcome rules, measure limits and cap). Existing SDK, MCP and scripts suites pass. The only failure locally istest_feature_map.py::test_bundled_copy_matches_the_firmware, which compares against a sibling firmware checkout and fails the same way onmain(it skips in CI). The zstd attachment path was run withzstandardinstalled and without it.No version bump; release later.
🤖 Generated with Claude Code