quality/coverage: replace the in-repo pipeline by score_coverage (Linux LLVM + QNX gcov) - #1171
dcalavrezo-qorix wants to merge 7 commits into
Conversation
The coverage pipeline that lived under quality/coverage (merger, reporter, reporter_wrapper, coverage_scope aspect, justify.py, effective_coverage.py, lcov_to_html.py, generate_coverage_html.sh) is the origin of the qualified S-CORE coverage tool, score_coverage (https://eclipse-score.github.io/coverage_tool/main/). This switches the repository to that module and deletes the copy: - quality/coverage/BUILD keeps only what is repository-specific: the coverage scope (unchanged dependable elements) and two report generators, LLVM for Linux and, new, gcov for QNX on target. - coverage.bazelrc: the default coverage command is the LLVM backend as before; --config=qnx now keeps Bazel's per-test gcov collector and points the final step at score_coverage's gcov reporter instead of Bazel's stock LCOV report, so the QNX report has the same layout, scope, 0 % baselines, justifications and gate as the Linux one. - workflows: the Linux coverage report is produced by @score_coverage//:generate_coverage_html (same archive name and layout, cpp_coverage_linux/); the QNX workflow gains a coverage step after the tests and uploads coverage_report_qnx. - MODULE.bazel: score_coverage dev dependency; rules_rust 0.68.2-score as required by it; the death-test coverage feature now comes from the module. gcovr leaves requirements.in (pyyaml stays, used elsewhere). - docs and the visibility-guard golden updated. Validated locally on //score/message_passing/... (Linux, LLVM): the scope allowlist is identical (613 files) and all 347 LCOV records are identical in covered/uncovered status to the previous pipeline (the only differences are execution counts of logging/timing code). The QNX run is validated in CI. VALIDATION ONLY: MODULE.bazel carries a git_override to the coverage_tool branch of eclipse-score/coverage_tool#10 for the gcov backend; replace it by the 0.3.0 registry release once published. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
|
CI evidence, nightly quality dispatched on this branch (run https://github.com/eclipse-score/communication/actions/runs/36396091677, success): the Linux coverage job ran the whole tree with the new pipeline, 484 tests, scope 613 files, raw line coverage 89.29 %, archive |
The gcov backend for QNX on-target coverage is released. Pin the release commit until the registry entry (bazel_registry PR #519) is merged, then drop the override. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
The sub-workspace pins score_communication by path and records the registry files of its dependencies; the rules_rust bump on this branch left its lock behind, which fails the GCC15 job under --lockfile_mode=error. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
LittleHuba
left a comment
LittleHuba
left a comment
There was a problem hiding this comment.
We are missing the hook into the release workflow and the dashboard.
| @@ -24,7 +24,6 @@ pydata_sphinx_theme | |||
| myst_parser | |||
|
|
|||
| # Python dependencies for coverage report generation (QNX) | |||
There was a problem hiding this comment.
Delete the comment as well please
| - name: Coverage QNX (gcov, on target, C++) | ||
| if: inputs.fetch_only != 'true' && steps.run-tests.outcome == 'success' | ||
| id: run-coverage | ||
| env: | ||
| SCORE_QNX_USER: ${{ secrets.SCORE_QNX_USER }} | ||
| SCORE_QNX_PASSWORD: ${{ secrets.SCORE_QNX_PASSWORD }} | ||
| COVERAGE_THRESHOLD: "0" | ||
| run: | | ||
| bazel coverage --config=ci --config=qnx --build_tests_only -- //score/... | ||
| bazel run --config=ci --config=qnx @score_coverage//:generate_coverage_html -- \ | ||
| --platform qnx \ | ||
| --yaml quality/coverage/coverage_justifications.yaml \ | ||
| --testlogs-subdir score \ | ||
| --archive-dir coverage_artifact_qnx | ||
| - name: Upload QNX coverage report | ||
| if: inputs.fetch_only != 'true' && steps.run-coverage.outcome == 'success' | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: ${{ github.event.repository.name }}_coverage_report_qnx_${{ github.sha }} | ||
| path: coverage_artifact_qnx/ | ||
| include-hidden-files: true | ||
| retention-days: 10 |
There was a problem hiding this comment.
This should ideally be in the coverage workflow or at least feed into it...
There was a problem hiding this comment.
Moved: the QNX coverage is now a second job (coverage-qnx) of _coverage_report.yml, called from nightly_quality.yml with the QNX secrets like the CodeQL job. Its artifact is downloaded by the deploy job and published as quality/coverage_qnx/ next to the Linux report, so the quality pages and the release zip carry both (quality_reports.rst has the link). The job is continue-on-error so a QNX SDP download hiccup does not take the Linux report down. build_and_test_qnx.yml runs tests only again. Commit 5049376.
The release is in the S-CORE Bazel registry; drop the temporary git_override that pinned the release commit. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
Bazel derives the default --instrumentation_filter from the packages of the test targets and strips only a trailing /tests to reach the code under test. Libraries tested from a test/ subpackage fall outside the guess and the gcov collector drops their counters (0 % for score/os in baselibs' QNX report). Instrument everything under //score. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
0.3.1 requires the explicit --instrumentation_filter this branch already sets, and warns when a report shows the pattern of a library tested from a test/ subpackage without data. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
dcalavrezo-qorix
changed the title
Layers the ulhu_coverage_fix workflow structure (determine-qnx gating job, parallel coverage-linux/coverage-qnx jobs in _coverage_report.yml, nightly QNX artifact publishing, doc links) on top of PR #1171's migration to the score_coverage Bazel module, replacing the in-repo coverage pipeline. - _coverage_report.yml: coverage-qnx now runs as its own job (gated on determine-qnx.outputs.available), invoking @score_coverage//:generate_coverage_html --platform qnx instead of the deleted local quality/coverage:generate_coverage_html script. - build_and_test_qnx.yml: reverted PR #1171's inline coverage steps now that QNX coverage lives in _coverage_report.yml. - nightly_quality.yml, generate_quality_links.bzl, docs/sphinx/quality_reports.rst, extract_coverage_artifact.sh, cache-strategy.md: carried over unchanged from ulhu_coverage_fix, updated to reference @score_coverage//:generate_coverage_html. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Layers the ulhu_coverage_fix workflow structure (determine-qnx gating job, parallel coverage-linux/coverage-qnx jobs in _coverage_report.yml, nightly QNX artifact publishing, doc links) on top of PR #1171's migration to the score_coverage Bazel module, replacing the in-repo coverage pipeline. - _coverage_report.yml: coverage-qnx now runs as its own job (gated on determine-qnx.outputs.available), invoking @score_coverage//:generate_coverage_html --platform qnx instead of the deleted local quality/coverage:generate_coverage_html script. - build_and_test_qnx.yml: reverted PR #1171's inline coverage steps now that QNX coverage lives in _coverage_report.yml. - nightly_quality.yml, generate_quality_links.bzl, docs/sphinx/quality_reports.rst, extract_coverage_artifact.sh, cache-strategy.md: carried over unchanged from ulhu_coverage_fix, updated to reference @score_coverage//:generate_coverage_html. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Review feedback: the QNX coverage step lived in the QNX build workflow and its artifact reached neither the dashboard nor the quality pages nor the release's quality-reports zip. - _coverage_report.yml gains a coverage-qnx job (QNX environment action, QNX secrets passed through from nightly_quality.yml, gcov backend, archive cpp_coverage_qnx); continue-on-error so a QNX SDP download hiccup does not take the Linux report down. - nightly_quality.yml downloads the QNX artifact and extracts it to _quality/coverage_qnx, next to the Linux report, so the quality pages and the release zip carry both; extract_coverage_artifact.sh takes the report subdirectory as a third argument. - quality_reports.rst and generate_quality_links.bzl add the QNX coverage link (latest, release artifact, local build). - build_and_test_qnx.yml no longer runs coverage. - requirements.in: leftover comment of the removed gcovr dependency. - README: where the two reports are produced and published; vendored external headers are measured on QNX since score_coverage 0.3.1. Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
|
Addressed the review in 5049376: QNX coverage moved into |
|
@LittleHuba @castler could you review this one, please? It moves the coverage pipeline to the qualified |
This branch has not been deployed
2 participants
Uses the
score_coverage0.3.1 registry release (release notes).What
The coverage pipeline under
quality/coverage/(merger, reporter, scope aspect,justify.py,effective_coverage.py,lcov_to_html.py,generate_coverage_html.sh) is the origin of the qualified S-CORE coverage tool,score_coverage. This PR switches the repository to that module and deletes the copy (10 files, ~3 000 lines).What stays in
quality/coverage/: the coverage scope (unchanged://score/message_passing,//score/mw/com,//score/mw/com/rust:score_com), two report generators (LLVM for Linux, gcov for QNX, the latter taggedmanualso wildcard builds on Linux do not fetch the QNX SDP),coverage.bazelrcand the justification YAML.rules_rustmoves to 0.68.2-score (required by the module); themodule_integration_testlockfile follows.What changes for users
bazel coverage //...+bazel run //quality/coverage:generate_coverage_htmlbazel coverage //...+bazel run @score_coverage//:generate_coverage_html -- --yaml ... --testlogs-subdir scorebazel coverage --config=qnx+generate_coverage_html --platform qnx(Bazel's stock LCOV report, HTML vialcov_to_html.py, hand-maintained baselines)bazel coverage --config=qnx+generate_coverage_html --platform qnx: same report layout, scope, 0 % baselines from the.gcnonotes, justifications, gate as on Linuxcpp_coverage_<platform>/coverage_<platform>/(CI keepscpp_coverage_linuxvia the positional argument, so the published quality pages are unchanged)unmapped_files.txt: in-scope files without any coverage data, categorisedVerification
Local, Linux,
//score/message_passing/...(11 tests), old pipeline vs this branch, LCOV compared record by record:bazel build --nobuild //...,//:format_test, visibility guardCI, Linux: the nightly quality workflow dispatched on this branch (run 36396091677) is green: 484 tests, 613 files in scope, 89.3 % raw line coverage, report published as before.
QNX:
_coverage_report.ymlgains a second job,coverage-qnx(QNX environment action, gcov backend,continue-on-errorso a QNX SDP download hiccup does not take the Linux report down). The nightly quality workflow passes the QNX secrets to it, downloads its artifact and publishes it asquality/coverage_qnx/next to the Linux report, so the quality pages and the release's quality-reports zip carry both; the quality page links it. The QNX report therefore appears with the next nightly run after the merge. The same backend produced a complete report on baselibs' QNX run (eclipse-score/baselibs#626: 649 files, 92 % lines with the instrumentation filter).Known differences of the QNX report, by design of the gcov backend: Rust sources are listed as
not-instrumented(measured on Linux); headers vendored from external repositories have no data (Bazel's collector drops them); gcov counts fewer lines than LLVM (no unused inline functions, no closing braces).Tracking: eclipse-score/tooling#427, eclipse-score/score#3258.
Update: bumped to 0.3.1, which requires the explicit
--instrumentation_filter=^//score[/:]this branch sets. Bazel guesses the filter from the packages of the test targets; libraries tested from atestsubpackage were otherwise compiled without counters or, on QNX, lost their counters (score/os showed 13 % instead of 80 % in the first QNX report). See the 0.3.1 release notes.