From 8303130a5c30079489435c81eb7f035a038a2a55 Mon Sep 17 00:00:00 2001 From: Matt <47545907+SoundMatt@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:03:57 -0700 Subject: [PATCH] =?UTF-8?q?rewrite(phase7):=20batch=202=20=E2=80=94=20add?= =?UTF-8?q?=20real=20MC/DC=20coverage=20ratchet=20gate=20(LLVM)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Part of the v3.0.0 ground-up rewrite (cpp-RCP #129, ROADMAP.md Phase 17), branch rewrite/v3-from-c-rcp. Phase 7 (formal verification + MC/DC + release-pipeline hardening), batch 2: real MC/DC (Modified Condition/ Decision Coverage) evidence in CI. cpp-RCP had zero MC/DC infrastructure before this batch (verified: no mcdc/MCDC/coverage-mcdc references anywhere in .github/workflows/ci.yml). ## What this adds Ported c-RCP's existing `mcdc:` CI job (LLVM 18, real llvm-cov/llvm-profdata condition/decision instrumentation, ratchet-gated -- not a 100% gate), adapted for cpp-RCP's header-only C++ architecture: - CMAKE_C_FLAGS -> CMAKE_CXX_FLAGS, CXX: clang++-18 (not CC:). - -sources scoped to include/rcp/*.hpp (cpp-RCP has no src/ dir -- verified; all real implementation logic lives in headers, not the test files), matching c-RCP's own intent of measuring implementation code, not test code. - `cpfusa coverage --mcdc-file` confirmed to have the same class of bug as c-RCP's `cfusa` equivalent: it parses `covered_true_count`/ `covered_false_count` object keys that real `llvm-cov export -format=text` output never produces (verified directly against both cpp-FuSa's source and real export output -- the schema is a positional array, no such keys anywhere). Followed c-RCP's precedent: the job reads llvm-cov export's own totals.mcdc block directly. ## Floor: 60%, set from a real freshly-measured baseline Not copied from c-RCP's own 93% (that number reflects c-RCP's C codebase's own current state, meaningless for cpp-RCP's). Measured locally end-to-end (instrumented build -> 58/58 tests passing under instrumentation -> profile merge -> llvm-cov export) on Homebrew LLVM 18.1.8 (matching the CI job's clang-18 major version): 313/466 = 67.17%. Cross-checked on LLVM 22.1.8: 426/629 = 67.73% -- close agreement despite a 4-major-version gap, evidence the number reflects the test suite, not toolchain noise. Floor set with ~7-point margin below the clang-18-matched measurement, wider than c-RCP's own 2.3-point margin because this measurement also crosses macOS/arm64 (local) vs Ubuntu 22.04/x86_64 (CI) and Homebrew-built vs apt.llvm.org-built clang-18 -- either could plausibly shift condition-pair counts on a header-only codebase (many decisions live in platform-conditional code, e.g. rcp/l2.hpp's raw-socket paths) more than a same-OS compiler bump would. Provenance written into the gate step's own comment and AUDIT_PACK.md §3/§5. ## Verification Independently re-verified (not just the porting agent's own measurement): reproduced the entire instrumented pipeline myself from scratch on the same Homebrew LLVM 18.1.8 toolchain -- 58/58 tests pass under instrumentation, real llvm-cov export produces a non-empty totals.mcdc block: 315/468 = 67.31%, matching the report's 313/466 = 67.17% almost exactly (trivial build-to-build variance, well within the ~7-point floor margin either way). Directly confirmed the cpfusa --mcdc-file schema mismatch by inspecting both cpp-FuSa's parser source and a real llvm-cov export's actual JSON structure side by side. Validated the new/modified ci.yml job's YAML parses cleanly. Full non-instrumented tree rebuilt from scratch separately: 0 errors, 0 warnings, 58/58 ctest suites pass (100%) -- confirming this batch doesn't affect the normal build path at all. .github/workflows/ci.yml: +192 lines (809 -> 1001). AUDIT_PACK.md: +18 lines (140 -> 158, real-MC/DC provenance paragraph in §3 + new row in the §5 CI Gate Summary table). Co-Authored-By: Claude Sonnet 5 Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com> --- .github/workflows/ci.yml | 192 +++++++++++++++++++++++++++++++++++++++ AUDIT_PACK.md | 18 ++++ 2 files changed, 210 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 90e84bd..e24fdb4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -371,6 +371,198 @@ jobs: name: coverage-info path: coverage.info + # ── LLVM MC/DC (real condition/decision coverage, ratchet-gated) ───────────── + # + # Phase 7 batch 2 (cpp-RCP #129): real MC/DC evidence per ISO 26262-6:2018 + # Table 12 (MC/DC is "++" recommended at ASIL-C/D), distinct from the + # `cpfusa coverage --mcdc`/`--dal DAL-B` branch-coverage fallback used in + # cpfusa-report below, which is NOT verified MC/DC evidence (see + # AUDIT_PACK.md §3). + # + # `cpfusa coverage --mcdc-file` (the mechanism this job would otherwise + # feed) cannot be used here: confirmed by direct local reproduction that + # cpp-FuSa v0.18.0's parser (src/coverage/coverage.cpp's apply_mcdc) + # scans for `mcdc_records[].conditions[].covered_true_count`/ + # `covered_false_count` object fields, while real `llvm-cov export + # -format=text` emits `mcdc_records` as positional arrays (LineStart/ + # ColumnStart/LineEnd/ColumnEnd/.../a bool TestVectors array) with no such + # keys anywhere in the schema, at any LLVM version — the same bug class + # sibling repo c-RCP hit and filed as SoundMatt/c-FuSa#129. This job does + # not wait on an upstream fix — it reads `llvm-cov export`'s own + # `totals.mcdc` block directly instead of routing through cpfusa's + # mismatched parser, so the number below is genuine, tool-independent + # MC/DC evidence today. + # + # Landed ratchet-gated from the start (not informational-only): unlike + # c-RCP's mcdc job, which shipped informational before a later issue + # added its gate, this job ships its regression floor in the same PR that + # introduces it, mirroring this file's own "Coverage regression gate + # (line floor, not DAL-B)" step further down in cpfusa-report — a + # ratchet against regression, not a claim of 100% (or even 80%) MC/DC. + # See the "MC/DC regression gate" step at the end of this job for the + # floor itself and its provenance comment. + mcdc: + name: MC/DC coverage (LLVM, ratchet-gated) + runs-on: ubuntu-22.04 + needs: build-and-test + steps: + - uses: actions/checkout@v4 + + - name: Install LLVM 18 (clang/llvm-cov/llvm-profdata with MC/DC support) + run: | + wget -qO /tmp/llvm.sh https://apt.llvm.org/llvm.sh + chmod +x /tmp/llvm.sh + sudo /tmp/llvm.sh 18 + sudo apt-get install -y cmake ninja-build + + - name: Configure (MC/DC-instrumented build) + env: + CXX: clang++-18 + run: | + cmake -B build-mcdc \ + -DCMAKE_BUILD_TYPE=Debug \ + -DCMAKE_CXX_STANDARD=17 \ + -DCMAKE_CXX_FLAGS="-fprofile-instr-generate -fcoverage-mapping -fcoverage-mcdc -O0" \ + -DCMAKE_EXE_LINKER_FLAGS="-fprofile-instr-generate -fcoverage-mapping" \ + -G Ninja + + - name: Build + run: cmake --build build-mcdc --parallel + + - name: Test (profiled; one .profraw per test binary, %p/%m-namespaced) + env: + LLVM_PROFILE_FILE: mcdc-profiles/%p-%16m.profraw + run: | + mkdir -p build-mcdc/tests/mcdc-profiles + ctest --test-dir build-mcdc --output-on-failure -j1 + + - name: Merge profile data + run: | + llvm-profdata-18 merge -sparse \ + build-mcdc/tests/mcdc-profiles/*.profraw \ + -o mcdc.profdata + + - name: Export real MC/DC totals (llvm-cov, not cpfusa --mcdc-file -- see job header) + run: | + bins=() + while IFS= read -r f; do bins+=("$f"); done < <(find build-mcdc/tests -maxdepth 1 -type f -perm -u+x -name 'test_*') + primary="${bins[0]}" + objargs=() + for b in "${bins[@]:1}"; do objargs+=(-object "$b"); done + # Header-only: the real implementation lives in include/rcp/*.hpp, + # not in any src/ dir (cpp-RCP has none) -- scope -sources there, + # not to tests/*.cpp, so this measures whether the implementation's + # own conditions/decisions are independently exercised by the test + # suite, matching c-RCP's own src/*.c-scoped intent. + srcs=() + while IFS= read -r f; do srcs+=("$f"); done < <(find include/rcp -name '*.hpp') + llvm-cov-18 export "$primary" "${objargs[@]}" \ + -instr-profile=mcdc.profdata -format=text \ + -sources "${srcs[@]}" \ + > mcdc-export.json + + - name: Summarize MC/DC totals + run: | + python3 - <<'PYEOF' + import json + d = json.load(open("mcdc-export.json")) + totals = d["data"][0]["totals"] + mcdc = totals.get("mcdc", {}) + branch = totals.get("branches", {}) + summary = { + "mcdc_condition_pairs_covered": mcdc.get("covered"), + "mcdc_condition_pairs_total": mcdc.get("count"), + "mcdc_percent": mcdc.get("percent"), + "branch_percent_for_comparison": branch.get("percent"), + } + json.dump(summary, open("mcdc-summary.json", "w"), indent=2) + print(json.dumps(summary, indent=2)) + note = ( + f"Real LLVM MC/DC (condition/decision) coverage: " + f"{mcdc.get('percent', 0):.2f}% " + f"({mcdc.get('covered')}/{mcdc.get('count')} condition " + f"independence pairs) across include/rcp/*.hpp -- " + f"ratchet-gated (regression-only, not a 100% gate; see the " + f"gate step below). For comparison, branch coverage over the " + f"same instrumented binaries: " + f"{branch.get('percent', 0):.2f}%. See job header for why " + f"this is measured directly via llvm-cov rather than " + f"cpfusa coverage --mcdc-file." + ) + print(f"::notice title=MC/DC Coverage::{note}") + with open("mcdc-summary.md", "w") as f: + f.write("### MC/DC coverage (ratchet-gated, not a 100% gate)\n\n") + f.write(note + "\n") + PYEOF + cat mcdc-summary.md >> "$GITHUB_STEP_SUMMARY" + + # Uploaded *before* the gate step below so a failing gate still + # leaves this diagnostic artifact inspectable -- matching how this + # file's own cpfusa-report job uploads its compliance-report artifact + # ahead of any of its own non-gating steps failing. + - name: Upload MC/DC report + uses: actions/upload-artifact@v4 + with: + name: mcdc-report + path: | + mcdc-summary.json + mcdc-export.json + + # The real gate: hard-fails if MC/DC condition/decision coverage + # drops below a floor set with real margin under the currently- + # measured percentage, mirroring this file's own "Coverage + # regression gate (line floor, not DAL-B)" step in cpfusa-report -- + # a ratchet against regression, not a claim of complete MC/DC (see + # AUDIT_PACK.md §3). Reads mcdc-summary.json already written by the + # step above rather than recomputing anything. + # + # Floor set at 60% (Phase 7 batch 2 / cpp-RCP #129): cpp-RCP had ZERO + # MC/DC infrastructure before this PR, so there is no prior + # measurement to ratchet against -- this is the first one. Local + # instrumented build+test+export (same mechanism and flags as this + # job) run twice: Homebrew LLVM 18.1.8 (matching this job's own + # clang-18 major version) measured 313/466 = 67.17% over + # include/rcp/*.hpp with all 58 tests passing under instrumentation; + # a second run on Homebrew LLVM 22.1.8 measured 426/629 = 67.73%, + # close agreement despite a 4-major-version toolchain gap, which is + # itself evidence the percentage is a stable property of the test + # suite rather than a toolchain artifact. 60% keeps ~7 points of + # margin below the clang-18-matched 67.17% figure -- wider than + # would be needed for gcc/clang version drift alone, because this + # measurement also crosses macOS/arm64 (local) vs Ubuntu 22.04/x86_64 + # (this job) and Homebrew's clang-18 build vs apt.llvm.org's, either + # of which could plausibly shift condition-pair counts on a + # header-only codebase (many decisions live in platform-conditional + # code, e.g. rcp/l2.hpp's raw-socket paths) more than a same-OS + # version bump would. Raise this floor deliberately, with an updated + # provenance comment, once a fresh re-measurement on this job's own + # runner confirms real headroom -- do not raise it from a + # locally-measured number alone. + - name: MC/DC regression gate (ratchet floor, not 100% -- Phase 7 batch 2 / cpp-RCP #129) + run: | + python3 - <<'PYEOF' + import json + import sys + + floor = 60.0 # see step name's comment above for provenance + + summary = json.load(open("mcdc-summary.json")) + actual = summary["mcdc_percent"] + + if actual is None or actual < floor: + print(f"::error::MC/DC coverage regression: {actual}% < floor {floor}% " + f"({summary['mcdc_condition_pairs_covered']}/" + f"{summary['mcdc_condition_pairs_total']} condition " + f"independence pairs). This is a ratchet floor (Phase 7 " + f"batch 2 / cpp-RCP #129), not a 100% gate -- if this drop " + f"is an intentional tradeoff, lower the floor explicitly " + f"with a provenance comment matching this step's own " + f"convention; do not just delete or skip this step.") + sys.exit(1) + + print(f"OK: {actual:.2f}% >= floor {floor}%") + PYEOF + # ── Build cpp-FuSa binary (shared by all cpfusa-* jobs) ────────────────────── cpfusa-build: name: Build cpp-FuSa diff --git a/AUDIT_PACK.md b/AUDIT_PACK.md index e5b68e1..e7c0a27 100644 --- a/AUDIT_PACK.md +++ b/AUDIT_PACK.md @@ -69,6 +69,23 @@ itself was written to correct. Required threshold: 80% branch coverage. MC/DC coverage target of 80% is tracked as an open item for an ASIL-C upgrade path. +Real MC/DC (condition/decision) evidence, distinct from the `cpfusa +coverage --mcdc`/`--dal DAL-B` branch-coverage fallback above, is now +measured in CI by `.github/workflows/ci.yml`'s `mcdc` job (Phase 7 batch +2 / cpp-RCP #129): LLVM's own `-fcoverage-mcdc` instrumentation, built +and run against the full `ctest` suite, exported via `llvm-cov export` +(not `cpfusa coverage --mcdc-file`, whose parser expects JSON keys real +`llvm-cov export` output does not produce — see the job's own header +comment; filed upstream as SoundMatt/cpp-FuSa#64-class). Freshly measured +immediately before this PR (Homebrew LLVM 18.1.8, matching the CI job's +own clang-18, on `include/rcp/*.hpp`): 313/466 = 67.17% real MC/DC +condition-pair coverage (corroborated by a second local run on LLVM +22.1.8: 426/629 = 67.73%, close agreement across a 4-major-version +toolchain gap). The `mcdc` job ratchet-gates a 60% floor — real margin +below that measurement, not a 100% or 80% claim — so this stays +open-item/informational for the 80% ASIL-C target above while still +catching a real regression today. + --- ## 4. DO-178C (DAL-C) Applicability @@ -101,6 +118,7 @@ All of the following gates must pass for a tagged release: | IEC 61508 report | `cpfusa iec61508` | Gap report generated (advisory) | | DO-178C report | `cpfusa do178` | Gap report generated (advisory) | | Coverage | `cpfusa coverage` | ≥ 80% branch | +| MC/DC (real, LLVM) | `ci.yml`'s `mcdc` job (`llvm-cov export`) | Ratchet floor: ≥ 60% (not 100%; see §3) | | SCI (Software Change Impact) | `cpfusa sci` | No unmitigated impacts | | Audit pack | `cpfusa audit-pack` | Generated | | Release badge | `cpfusa badge` | Green |