Add the gcov backend for QNX on-target coverage by dcalavrezo-qorix · Pull Request #10 · eclipse-score/coverage_tool

dcalavrezo-qorix · 2026-09-28T06:49:08Z

Adds QNX on-target coverage as a second backend of the tool (tooling#427, track 1). QCC is GCC-based and cannot emit LLVM coverage mapping; QNX tests run in QEMU. The backend changes the collection, not the report.

How it works

Step	Linux (LLVM backend)	QNX / GCC (gcov backend)
Instrumentation	Clang/rustc covmap	`-fprofile-arcs -ftest-coverage` (toolchain `coverage` feature)
Test execution	host	QEMU via `score_qnx_unit_tests` (`--run_under`), which tars the `.gcda` counters and extracts them into Bazel's `COVERAGE_DIR`
Per-test collection	`merger.py` (profraw → profdata)	Bazel's own `collect_coverage.sh` + `lcov_merger` (kept as is)
Final report	`reporter.py` (llvm-cov)	new `gcov_reporter.py`: sums per-test LCOV per file, applies the scope, zero-coverage baseline via `gcov --json-format` over the `.gcno` notes of in-scope translation units, HTML + summary via gcovr 8.6
Phase 2	unchanged	unchanged (same zip layout)

Consumer side: one more score_coverage_reporter(backend = "gcov", gcov = "@score_qcc_x86_64_toolchain_pkg//:gcov") target and a coverage:qnx bazelrc block (user manual, step 4b). Existing consumers need no change.

Limits, documented: C++ only (Rust sources are reported not-instrumented); headers vendored from external repositories have no data on this backend (Bazel's collector drops them); gcov and LLVM count lines differently, so the two reports are compared per file.

Two defects found on the way

effective_coverage's gcovr parser matched across table rows. Real gcovr pages carry bare rows for non-instrumented lines, so every status shifted by one row and no justification ever applied to a gcovr report. The synthetic test HTML had a class on every row. Fixed, with a regression test using the real markup.
The self-coverage gate double counted branches when two test targets import the same module (coverage.py's BRDA branch field encodes whether the arc was taken; Bazel's merger keeps both variants). Arcs are now keyed by (line, block).

Verification

Check	Result
Unit tests	22 targets, 20 new cases (`gcov_reporter_test`), 1 new analysis test
Integration workspace, LLVM section	unchanged, 18 checks
Integration workspace, gcov section (S-CORE GCC 12.2.0, `coverage:gcov`)	gate, hand-derived `expected_lcov_gcov.dat` matches, every gcovr link opens, justification applied (effective 66.66 % > raw 58.33 %), categories incl. `not-instrumented`
lint (ruff, pylint, ty), format, copyright	pass
self coverage	96.02 % lines / 90.61 % branches (gate 95 / 87); `gcov_reporter.py` 96.0 % / 90.9 %
docs	0 warnings

GCC on Linux takes exactly the collection path QCC takes on QNX; the QEMU transport itself is communication's and baselibs' existing run_under_qnx flow. A real QNX run is validated on baselibs (follow-up PR with a git_override to this branch; the QNX credentials live in that repository's CI).

Docs

Architecture (backend section and diagram), tool requirements scope_gcno, backend_select, gcov_merge, gcov_baseline, gcov_html; potential errors ERR-11/ERR-12; constraint CSTR-11; user manual step 4b; known problems; release notes 0.3.0; verification report.

QCC is GCC-based and cannot emit LLVM coverage mapping, and QNX tests run inside QEMU. This adds a second backend that changes the collection, not the report (tooling issue #427, track 1): - score_qnx_unit_tests' run_under_qnx runner brings the .gcda counters back from the guest; Bazel's own per-test collector (gcov + lcov_merger) turns them into LCOV; the new gcov_reporter.py is the --coverage_report_generator: it sums per-test records per file, applies the scope through the selection shared with the LLVM reporter, adds a zero-coverage baseline by running gcov over the .gcno notes of every in-scope translation unit, renders HTML and text summary with gcovr 8.6 (new pip dependency), and writes the same zip layout, so phase 2 (justifications, summary, gate, archive) is unchanged. - score_coverage_scope collects the .gcno files through InstrumentedFilesInfo (<name>_gcno.txt, gcno/gcno_files output groups). score_coverage_reporter gained backend = "llvm"|"gcov" and gcov = <toolchain gcov label>; the LLVM tools are required only for the LLVM backend. Existing consumers need no change. - Rust sources are reported as not-instrumented on this backend (new unmapped category); headers vendored from external repositories have no data because Bazel's collector filters them out (documented). Two defects found on the way, both fixed: - effective_coverage's gcovr parser matched across table rows; real gcovr pages have bare rows for non-instrumented lines, so every line status was attributed to the previous instrumented line and no justification ever applied to a gcovr report. - the self-coverage gate double counted branches when two test targets import the same module: coverage.py's BRDA branch field encodes whether the arc was taken, Bazel's merger keeps both variants. Arcs are now keyed by (line, block). Validation: the integration workspace gained the S-CORE GCC 12.2.0 toolchain, a coverage:gcov config, a hand-derived expected_lcov_gcov.dat and gcov checks (gate, ground truth, every gcovr link opens, justification applied, categories); 20 new unit tests; one new analysis test. GCC on Linux takes exactly the collection path QCC takes on QNX; the QEMU transport is validated on a consumer. Docs: architecture (backend section and diagram), tool requirements (scope_gcno, backend_select, gcov_merge, gcov_baseline, gcov_html), potential errors ERR-11/ERR-12, constraint CSTR-11, user manual step 4b, known problems, release notes 0.3.0, verification report. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

github-actions · 2026-09-28T06:50:11Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run //:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
2026/09/28 10:03:41 Downloading https://releases.bazel.build/8.6.0/release/bazel-8.6.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server (8.6.0) and connecting to it...
INFO: Invocation ID: f4ffb1c6-7657-4094-afdb-e4a10f06f862
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (5 packages loaded, 6 targets configured)

Analyzing: target //:license-check (13 packages loaded, 10 targets configured)

Analyzing: target //:license-check (66 packages loaded, 10 targets configured)

Analyzing: target //:license-check (79 packages loaded, 10 targets configured)

Analyzing: target //:license-check (132 packages loaded, 273 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3138 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3138 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3138 targets configured)

Analyzing: target //:license-check (151 packages loaded, 3138 targets configured)

Analyzing: target //:license-check (152 packages loaded, 3141 targets configured)

Analyzing: target //:license-check (162 packages loaded, 7148 targets configured)

Analyzing: target //:license-check (163 packages loaded, 9468 targets configured)

Analyzing: target //:license-check (163 packages loaded, 9468 targets configured)

Analyzing: target //:license-check (173 packages loaded, 9525 targets configured)

Analyzing: target //:license-check (173 packages loaded, 9525 targets configured)

Analyzing: target //:license-check (177 packages loaded, 11534 targets configured)

INFO: Analyzed target //:license-check (178 packages loaded, 11661 targets configured).
[1 / 1] no actions running
[12 / 16] checking cached actions
[14 / 16] checking cached actions
[14 / 16] [Prepa] JavaToolchainCompileBootClasspath external/rules_java+/toolchains/platformclasspath.jar
[15 / 16] Building tools/license.check.license_check.jar (); 0s disk-cache, multiplex-worker
INFO: Found 1 target...
Target //tools:license.check.license_check up-to-date:
  bazel-bin/tools/license.check.license_check
  bazel-bin/tools/license.check.license_check.jar
INFO: Elapsed time: 47.619s, Critical Path: 2.50s
INFO: 16 processes: 12 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/tools/license.check.license_check tools/formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

Copilot

Copilot review overview

🟡 Changes recommended

Canonical path variants can lose per-test coverage counts, and the documented Rust build-setting label is invalid.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Duplicate test variants are excluded instead of being merged · New
LLVM setting label is misspelled · New

What changed in this PR

Adds a gcov backend for GCC/QNX coverage while preserving the existing report format and gating pipeline.

Changes:

Adds gcov collection, baseline generation, merging, and gcovr reporting.
Extends Bazel scope/wrapper rules and integration coverage.
Fixes gcovr justification parsing and self-coverage branch deduplication.

File	Description
`tools/self_coverage_gate.py`	Deduplicates detailed LCOV records.
`tools/self_coverage_gate_test.py`	Tests LCOV deduplication.
`score_coverage/tests/starlark/coverage_scope_tests.bzl`	Tests gcno outputs.
`score_coverage/tests/gcov_reporter_test.py`	Tests the gcov reporter.
`score_coverage/tests/effective_coverage_test.py`	Adds gcovr markup regression coverage.
`score_coverage/tests/BUILD`	Registers gcov reporter tests.
`score_coverage/requirements.in`	Adds gcovr.
`score_coverage/requirements_3_12.txt`	Locks gcovr dependencies.
`score_coverage/reporter_wrapper.bzl`	Selects LLVM or gcov backend.
`score_coverage/generate_coverage_html.py`	Generalizes report errors.
`score_coverage/gcov_reporter.py`	Implements gcov report generation.
`score_coverage/effective_coverage.py`	Constrains gcovr row parsing.
`score_coverage/coverage_summary.py`	Adds not-instrumented category.
`score_coverage/coverage_scope.bzl`	Exports gcno baseline inputs.
`score_coverage/BUILD`	Defines gcov reporter targets.
`README.md`	Documents dual backends.
`integration_tests/tools/coverage/BUILD`	Adds gcov wrapper target.
`integration_tests/run_integration_test.sh`	Exercises gcov end to end.
`integration_tests/MODULE.bazel.lock`	Updates integration dependencies.
`integration_tests/MODULE.bazel`	Adds GCC toolchain.
`integration_tests/expected_lcov_gcov.dat`	Defines gcov ground truth.
`integration_tests/.bazelrc`	Adds gcov coverage configuration.
`docs/verification/verification_report.rst`	Records gcov verification.
`docs/requirements/tool_requirements.rst`	Specifies gcov requirements.
`docs/requirements/potential_errors.rst`	Adds gcov failure modes.
`docs/release/release_notes.rst`	Documents release changes.
`docs/manual/user_manual.rst`	Adds QNX adoption instructions.
`docs/manual/known_problems.rst`	Documents backend limitations.
`docs/manual/constraints.rst`	Adds QNX transport constraints.
`docs/architecture/index.rst`	Describes gcov architecture.
`defs.bzl`	Extends the public reporter macro.
`BUILD`	Exports the gcov reporter alias.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

dcalavrezo-qorix · 2026-09-28T10:03:02Z

+    merged: dict[str, FileRecord] = {}
+    for raw, name in selection.staged.items():
+        if raw in tested:
+            merged.setdefault(name, FileRecord()).add(tested[raw])
+    for raw, name in selection.staged.items():
+        if raw in baseline and name not in merged:
+            merged.setdefault(name, FileRecord()).add(baseline[raw].zeroed())
+    return merged


Valid, thanks. For the gcov backend the merge is ours (unlike llvm-cov, which needs one path per file), so dropping the duplicate raw variants lost the second test's counts. Fixed in 55bf112: test and baseline records are now summed per canonical name before the selection (merge_by_name), with a unit test for a header recorded under its declared path by one test and under a _virtual_includes path by another.

dcalavrezo-qorix · 2026-09-28T10:03:04Z

+   coverage:qnx --noexperimental_use_llvm_covmap
+   coverage:qnx --noexperimental_generate_llvm_lcov
+   coverage:qnx --test_env=GENERATE_LLVM_LCOV --test_env=COVERAGE_GCOV_PATH --test_env=LLVM_PROFILE_CONTINUOUS_MODE
+   coverage:qnx --@rules_rust//rust/settings:extra_rustc_flags=


Both settings exist in rules_rust: the repeatable extra_rustc_flag (singular, used by the coverage:llvm_cov block) and the list-typed extra_rustc_flags (plural). The line was not a typo, but you have a point that it was misleading: an empty value resets only the list-typed plural flag, and the singular one accumulates and cannot be reset at all (verified with bazel aquery on the integration workspace). A workspace following Step 4 keeps the LLVM rustc flags in their own config, so the QNX block does not need to reset anything. Removed the line and documented the reset for workspaces that put the flags on the bare coverage command (that is communication's setup, which uses the plural flag).

hallucination

Found on communication: the QNX gcov reporter depends on the QNX SDP package, and bazel build //... on a Linux host without credentials fails on that download. The manual tag keeps wildcards away from the target; --coverage_report_generator still names it explicitly. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

The coverage pipeline that lived under quality/coverage (merger, reporter, reporter_wrapper, coverage_scope aspect, justify.py, effective_coverage.py, lcov_to_html.py, generate_coverage_html.sh) is the origin of the qualified S-CORE coverage tool, score_coverage (https://eclipse-score.github.io/coverage_tool/main/). This switches the repository to that module and deletes the copy: - quality/coverage/BUILD keeps only what is repository-specific: the coverage scope (unchanged dependable elements) and two report generators, LLVM for Linux and, new, gcov for QNX on target. - coverage.bazelrc: the default coverage command is the LLVM backend as before; --config=qnx now keeps Bazel's per-test gcov collector and points the final step at score_coverage's gcov reporter instead of Bazel's stock LCOV report, so the QNX report has the same layout, scope, 0 % baselines, justifications and gate as the Linux one. - workflows: the Linux coverage report is produced by @score_coverage//:generate_coverage_html (same archive name and layout, cpp_coverage_linux/); the QNX workflow gains a coverage step after the tests and uploads coverage_report_qnx. - MODULE.bazel: score_coverage dev dependency; rules_rust 0.68.2-score as required by it; the death-test coverage feature now comes from the module. gcovr leaves requirements.in (pyyaml stays, used elsewhere). - docs and the visibility-guard golden updated. Validated locally on //score/message_passing/... (Linux, LLVM): the scope allowlist is identical (613 files) and all 347 LCOV records are identical in covered/uncovered status to the previous pipeline (the only differences are execution counts of logging/timing code). The QNX run is validated in CI. VALIDATION ONLY: MODULE.bazel carries a git_override to the coverage_tool branch of eclipse-score/coverage_tool#10 for the gcov backend; replace it by the 0.3.0 registry release once published. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

dcalavrezo-qorix · 2026-09-28T08:14:21Z

Real QNX validation: baselibs PR #626 ran the gcov backend on bl-x86_64-qnx in CI (278 tests in QEMU, https://github.com/eclipse-score/baselibs/actions/runs/36388596870): 615 files reported, every index link opens, 81.59 % raw line coverage, baseline for 160 untested files from the gcno notes, categories as designed (50 Rust not-instrumented). communication is migrated to both backends in eclipse-score/communication#1171 (in-repo pipeline deleted; Linux LCOV identical to the old pipeline record by record on message_passing). One consumer-side lesson added to the manual: the QNX reporter target needs tags = ["manual"], otherwise bazel build //... on a Linux host tries to fetch the QNX SDP.

The gcov reporter dropped every raw path variant of a canonical file except one before merging. When one test reports a header by its declared path and another test through a _virtual_includes path, the second test's counts were lost. Merge test and baseline records by canonical name before the selection so all variants are summed. Reported by the Copilot review on PR #10. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

AlexanderLanin

Not sure who is supposed to review this any deeper. LGTM

Only the list-typed extra_rustc_flags setting can be cleared with an empty value; the repeatable extra_rustc_flag accumulates. The QNX block no longer resets a flag that a workspace following Step 4 never sets, and the manual says what to do when the LLVM rustc flags sit on the bare coverage command. Count the new gcov reporter test. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

The coverage pipeline that lived under quality/coverage (merger, reporter, reporter_wrapper, coverage_scope aspect, justify.py, effective_coverage.py, lcov_to_html.py, generate_coverage_html.sh) is the origin of the qualified S-CORE coverage tool, score_coverage (https://eclipse-score.github.io/coverage_tool/main/). This switches the repository to that module and deletes the copy: - quality/coverage/BUILD keeps only what is repository-specific: the coverage scope (unchanged dependable elements) and two report generators, LLVM for Linux and, new, gcov for QNX on target. - coverage.bazelrc: the default coverage command is the LLVM backend as before; --config=qnx now keeps Bazel's per-test gcov collector and points the final step at score_coverage's gcov reporter instead of Bazel's stock LCOV report, so the QNX report has the same layout, scope, 0 % baselines, justifications and gate as the Linux one. - workflows: the Linux coverage report is produced by @score_coverage//:generate_coverage_html (same archive name and layout, cpp_coverage_linux/); the QNX workflow gains a coverage step after the tests and uploads coverage_report_qnx. - MODULE.bazel: score_coverage dev dependency; rules_rust 0.68.2-score as required by it; the death-test coverage feature now comes from the module. gcovr leaves requirements.in (pyyaml stays, used elsewhere). - docs and the visibility-guard golden updated. Validated locally on //score/message_passing/... (Linux, LLVM): the scope allowlist is identical (613 files) and all 347 LCOV records are identical in covered/uncovered status to the previous pipeline (the only differences are execution counts of logging/timing code). The QNX run is validated in CI. VALIDATION ONLY: MODULE.bazel carries a git_override to the coverage_tool branch of eclipse-score/coverage_tool#10 for the gcov backend; replace it by the 0.3.0 registry release once published. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

dcalavrezo-qorix requested review from AlexanderLanin, MaximilianSoerenPollak and nradakovic as code owners September 28, 2026 06:49

dcalavrezo-qorix mentioned this pull request Sep 28, 2026

coverage: QNX on-target coverage through the score_coverage gcov backend eclipse-score/baselibs#626

Open

AlexanderLanin requested a balanced review from Copilot September 28, 2026 07:13

Copilot started reviewing on behalf of AlexanderLanin September 28, 2026 07:14 View session

Copilot AI reviewed Sep 28, 2026

View reviewed changes

dcalavrezo-qorix mentioned this pull request Sep 28, 2026

quality/coverage: replace the in-repo pipeline by score_coverage (Linux LLVM + QNX gcov) eclipse-score/communication#1171

Draft

AlexanderLanin previously approved these changes Sep 28, 2026

View reviewed changes

dcalavrezo-qorix dismissed AlexanderLanin’s stale review via 95a5d8d September 28, 2026 10:03

Count the new gcov reporter test in the verification report

265da7c

Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>

AlexanderLanin approved these changes Sep 28, 2026

View reviewed changes

dcalavrezo-qorix merged commit c2eb26a into main Sep 28, 2026
9 checks passed

dcalavrezo-qorix deleted the dcalavrezo_qnx_gcov branch September 28, 2026 10:13

AlexanderLanin mentioned this pull request Sep 28, 2026

Add coverage summaries per production Bazel target #11

Open

dcalavrezo-qorix mentioned this pull request Sep 30, 2026

Finish migration of coverage tooling #9

Open

Navigation Menu

Add the gcov backend for QNX on-target coverage - #10

Add the gcov backend for QNX on-target coverage#10
dcalavrezo-qorix merged 5 commits into
mainfrom
dcalavrezo_qnx_gcov

dcalavrezo-qorix commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

dcalavrezo-qorix Sep 28, 2026

Uh oh!

dcalavrezo-qorix Sep 28, 2026

Uh oh!

dcalavrezo-qorix Sep 28, 2026

Uh oh!

dcalavrezo-qorix commented Sep 28, 2026

Uh oh!

AlexanderLanin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dcalavrezo-qorix commented Sep 28, 2026

How it works

Two defects found on the way

Verification

Docs

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Check Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

dcalavrezo-qorix Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

dcalavrezo-qorix Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

dcalavrezo-qorix Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

dcalavrezo-qorix commented Sep 28, 2026

Uh oh!

AlexanderLanin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Sep 28, 2026 •

edited

Loading