diff --git a/docs/architecture/index.rst b/docs/architecture/index.rst index 4f071a8..6725235 100644 --- a/docs/architecture/index.rst +++ b/docs/architecture/index.rst @@ -23,272 +23,426 @@ Architecture :security: NO :realizes: wp__sw_implementation -The pipeline hooks into Bazel's two coverage extension points, -``--coverage_output_generator`` and ``--coverage_report_generator``, and adds a -justification and gating layer on top. It has two phases and, in phase 1, two -backends that produce the same report zip: - -- **llvm**: Clang and rustc coverage mapping, read with ``llvm-cov``; Linux - host tests, C++ and Rust. -- **gcov**: GCC and QNX QCC ``.gcda`` counters, read with the toolchain's - ``gcov``; C++ only. On QNX the tests run inside QEMU through - ``score_qnx_unit_tests``, which carries the counters back to the host, and - Bazel's own per-test collector turns them into LCOV. +The coverage tool shows which production code was exercised by tests and +checks whether line coverage meets a configured minimum. The repository using +the tool selects the production code to assess and may provide reviewed +explanations for code that is not exercised. + +**Line coverage** is the share of measurable source lines that executed during +these tests. Measurable lines are those for which the compiler provides +coverage information; they are not simply all lines in a source file. Comments +and blank lines, for example, do not count as executable code. + +The pipeline has two phases: + +1. **Collect coverage.** Build code with execution counters, run the selected + tests, and combine their measurements into a report. Either the LLVM or + gcov backend performs this phase, depending on the compiler and platform. +2. **Evaluate coverage.** Prepare the HTML report, optionally apply the reviewed + explanations, and compare line coverage with the required minimum. This + comparison is the **coverage gate**. It decides whether coverage is high + enough for the build or CI job to proceed. + +The first diagram shows the inputs and results of these phases. The next two +expand collection and evaluation respectively. In the data-flow diagrams, +blue boxes are inputs or outputs, yellow boxes are processing steps, and +arrows show what each step receives or produces. .. uml:: @startuml + top to bottom direction skinparam componentStyle rectangle skinparam artifactBackgroundColor #EAF2F8 skinparam artifactBorderColor #4E79A7 skinparam componentBackgroundColor #FFF2CC skinparam componentBorderColor #B58B00 - skinparam usecaseBackgroundColor #E2F0D9 - skinparam usecaseBorderColor #548235 - package "Phase 1: bazel coverage --config=llvm_cov" { - artifact "test binaries" as llvm_bins - artifact "profraw per test" as llvm_profraw - artifact "coverage.dat zip\nprofdata + meta.json" as llvm_test_zip - artifact "allowlist.txt\npath_map.txt\nobjects.txt\nsource files" as llvm_scope_files - component "Clang / rustc\ncovmap instrumentation" as llvm_instrumentation - component "merger.py\n--coverage_output_generator" as llvm_merger - component "score_coverage_scope\naspect" as llvm_scope - component "reporter.py\n--coverage_report_generator" as llvm_reporter - - llvm_instrumentation --> llvm_bins - llvm_bins --> llvm_profraw - llvm_profraw --> llvm_merger - llvm_merger --> llvm_test_zip - llvm_scope --> llvm_scope_files - llvm_test_zip --> llvm_reporter - llvm_scope_files --> llvm_reporter - } - package "Phase 1, gcov backend: bazel coverage --config=qnx" { - artifact "test binaries (QEMU on QNX)" as gcov_bins - artifact ".gcda per test" as gcda - artifact "coverage.dat LCOV" as gcov_lcov - artifact "allowlist.txt\npath_map.txt\ngcno.txt\nsource files" as gcov_scope_files - component "GCC / QCC\n-fprofile-arcs -ftest-coverage" as gcov_instrumentation - component "Bazel collector\ngcov + lcov_merger" as gcov_collector - component "gcov_reporter.py\n--coverage_report_generator" as gcov_reporter - - gcov_instrumentation --> gcov_bins - gcov_bins --> gcda - gcda --> gcov_collector - gcov_collector --> gcov_lcov - gcov_lcov --> gcov_reporter - gcov_scope_files --> gcov_reporter - } + artifact "Code whose coverage we assess\n(coverage scope)" as scope + artifact "Tests to run" as tests + component "1. Collect coverage" as collect + artifact "Coverage report zip\nmeasurements + HTML pages" as report + artifact "Optional explanations\nfor unexecuted code" as justifications + artifact "Required minimum\nline coverage" as threshold + component "2. Evaluate coverage" as evaluate + artifact "HTML report" as html + artifact "Gate result" as verdict + + scope --> collect + tests --> collect + collect --> report + report --> evaluate + justifications --> evaluate + threshold --> evaluate + evaluate --> html + evaluate --> verdict + @enduml + +Collection produces the measurements; evaluation applies the coverage policy. +Both backends use the same evaluation phase. + +Code whose coverage we assess +----------------------------- + +The **coverage scope** is the set of production source files whose coverage we +assess, including code that no selected test exercises. It can span several +units under test and their dependencies. The repository defines this set with +``score_coverage_scope(deps = [...])``. +The listed Bazel targets are build units such as libraries or binaries. The +scope follows their dependencies and collects declared source files from +supported targets in the repository. The repository must select production +targets and keep test code out of this scope: the tool follows declarations, +rather than deciding whether a file is production or test code. + +The scope and the selected tests answer different questions: + +- **Scope:** which production code should be assessed? +- **Tests:** which executions provide measurements for that code? + +**Instrumentation** means adding counters to record execution. The supplied +coverage configurations instrument tests as well as production code. This +also captures production code from headers compiled into a test binary. +The report includes only files in the configured scope, so test sources stay +out of the result when the scope contains only production code. + +Why untested code needs a baseline +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A production library can be in scope even if no selected test depends on it. +Using only the test results would omit that library and could make coverage +look better than it is. The tool therefore also builds scoped production code +and reads the compiler's coverage information for it. This **baseline** supplies +zero counts for measurable files missing from the test results. + +For example, if tests cover all 80 measurable lines in library A and never use +library B with 20 measurable lines, the combined report must show 80 %, with +B at 0 %. + +A baseline requires compiler coverage information. A header that is never +included, or template code that is never instantiated, may have none. A file +with no such information cannot be assigned a measured 0 % and is listed +separately in ``unmapped_files.txt``. Files without coverage data do not contribute measurable +lines to the percentage. The report distinguishes unexplained missing data +from cases such as declaration-only headers and Rust code unsupported by the +gcov backend. + +**A passing gate does not establish that every scoped file was measured.** +Missing-data warnings and the scope itself must also be reviewed. Files outside +the scope are excluded entirely. See :doc:`../manual/constraints` and +:doc:`../manual/known_problems` for the required checks and known limitations. + +Phase 1: collect coverage +------------------------- + +``bazel coverage`` builds and runs the selected tests. Each test produces its +own measurements. A **reporter** combines them into a report, applies the +scope, and adds baseline entries where available. The baseline supplies zero counts only for +files missing from the test data, so it does not overwrite measured hits. + +.. uml:: + + @startuml + top to bottom direction + skinparam componentStyle rectangle + skinparam artifactBackgroundColor #EAF2F8 + skinparam artifactBorderColor #4E79A7 + skinparam componentBackgroundColor #FFF2CC + skinparam componentBorderColor #B58B00 - package "Phase 2: bazel run //:generate_coverage_html" { - artifact "_coverage_report.dat zip\nhtml_report, lcov_report, text_report" as report_zip - artifact "manifest.json" as manifest - artifact "report.json\nsummary.txt" as effective_report - artifact "archive dir" as archive_dir - component "generate_coverage_html.py" as generate_coverage_html - component "justify.py" as justify - component "effective_coverage.py" as effective_coverage - component "coverage_summary.py" as coverage_summary - - report_zip --> generate_coverage_html - generate_coverage_html --> justify - justify --> manifest - manifest --> effective_coverage - effective_coverage --> effective_report - generate_coverage_html --> coverage_summary - generate_coverage_html --> (gate verdict\nexit 0 / 1 / 2) - generate_coverage_html --> archive_dir + together { + artifact "Selected tests" as tests + artifact "Code whose coverage we assess\n(production targets)" as scope } - llvm_reporter --> report_zip - gcov_reporter --> report_zip + tests -[hidden]right-> scope + + component "Build with execution counters\nand run tests" as run_tests + artifact "Execution counters\nfrom each test" as counters + component "Collect per-test measurements" as collect + artifact "Per-test measurements" as measurements + + component "Build scoped code and\ncollect its source files" as prepare + artifact "Source files + file selection\nCompiler information for\nuntested code" as baseline + + component "Combine measurements\nKeep scoped files\nAdd baseline where data is missing" as merge + artifact "Coverage report zip\nmeasurements + HTML pages" as report + + tests --> run_tests + run_tests --> counters + counters --> collect + collect --> measurements + scope --> prepare + prepare --> baseline + measurements --> merge + baseline --> merge + merge --> report @enduml -Phase 1: collection -------------------- - -The ``llvm_cov`` bazelrc config, copied by the consumer from the integration -workspace, does four things: - -1. **Swaps the compilers.** C++ is compiled with a hermetic Clang/LLVM toolchain - instead of GCC; Rust with the Ferrocene toolchain of ``score_toolchains_rust``, - which has LLVM coverage tools attached. Both emit the same covmap format. -2. **Turns on instrumentation.** ``--experimental_use_llvm_covmap`` plus the - ``coverage`` feature for C++; ``rules_rust`` adds ``-Cinstrument-coverage`` to - rustc once the toolchain declares coverage tools. Runtime counter relocation - (``-mllvm -runtime-counter-relocation`` via the ``cc_feature``, - ``-Cllvm-args=-runtime-counter-relocation`` for Rust) enables continuous mode - so coverage survives abnormal termination. Rust branch regions need - ``-Zcoverage-options=branch`` on a rolling Ferrocene. -3. **Installs the per-test tool.** ``merger.py`` merges the test's ``profraw`` - files with ``llvm-profdata``, records the instrumented objects, and zips both - as the test's ``coverage.dat``. -4. **Installs the final tool.** The consumer's ``score_coverage_reporter`` target - wraps ``reporter.py`` with the scope, workspace root and LLVM tool labels. - The reporter merges all per-test profiles and runs ``llvm-cov`` three times: - ``show`` (HTML), ``export`` (LCOV), ``report`` (text). - -**Scope.** Covmap instruments everything. Filtering happens at report time -through the allowlist written by ``score_coverage_scope``: an aspect walks the -dependency graph from the listed production targets and collects every source -file a workspace target declares (including headers it vendors from an -external repository). Everything else, test sources, googletest, external -dependencies, headers a wrapper rule only forwards, is excluded. For headers -exposed through ``strip_include_prefix`` / ``include_prefix`` the aspect also -writes a path map from the generated ``_virtual_includes/`` path the compiler -records to the declared header, and it exports the source files themselves. - -**Sources.** The reporter does not read sources through the workspace -directory: generated headers and external repositories are not there at -report time. It links every in-scope file from its runfiles into a staging -directory laid out like the coverage mapping expects, points llvm-cov at that -directory, and afterwards files the HTML pages under canonical paths so the -archive is machine-independent and every index link resolves. - -**Baseline.** A file that no test executes produces no profile data. The scope -aspect therefore also collects the compiled archives and executables, and the -reporter runs ``llvm-cov --empty-profile`` over them, so untested files show up -with exact 0 % entries whose denominators come from the compiler's coverage -map. Rust rlibs are expanded into their object members first because their -leading ``lib.rmeta`` member makes ``llvm-cov`` reject the archive. - -Phase 1, gcov backend ---------------------- - -The gcov backend exists for toolchains that cannot emit LLVM coverage -mapping: GCC on Linux and, the reason it was built, QCC on QNX. It changes -the collection, not the report. - -1. **Compilers.** The ``coverage`` feature of the S-CORE GCC and QCC - toolchains adds ``-fprofile-arcs -ftest-coverage``; the compiler writes a - ``.gcno`` notes file per translation unit and the instrumented binary - writes ``.gcda`` counters when it exits. rustc cannot produce either, so - Rust sources in scope are reported as *not instrumentable* on this backend. -2. **Execution and transport.** On Linux the test writes its ``.gcda`` under - Bazel's ``COVERAGE_DIR``. On QNX the test runs inside a QEMU micro-VM - (``score_qnx_unit_tests``, ``--run_under``); ``GCOV_PREFIX`` points the - counters at a guest directory, the guest tars them at exit, and the host - runner extracts the archive into ``COVERAGE_DIR``. From there both are the - same. -3. **Per-test collection stays Bazel's.** Bazel's ``collect_coverage.sh`` - runs the toolchain's ``gcov`` over the counters and its ``lcov_merger`` - writes one LCOV file per test. Two properties of that collector shape the - backend: it maps generated ``_virtual_includes/`` paths back to the - declared header itself, and it keeps only files of its instrumented-files - manifest, which never lists sources of external repositories, so a header - vendored from an external repository has no data on this backend even when - a test executes it. Tests are instrumented as well - (``--instrument_test_targets``) so header-only code that only a test - translation unit instantiates is measured; the scope allowlist still drops - the test sources. -4. **Final report.** ``gcov_reporter.py`` sums the per-test LCOV records per - file (the same semantics as merging profiles on the LLVM side), applies - the scope through the shared selection logic, and adds the zero-coverage - baseline: ``gcov --json-format`` over the ``.gcno`` of every in-scope - translation unit without its ``.gcda`` yields every line and branch at - zero. The ``.gcno`` files come from ``InstrumentedFilesInfo`` through the - scope aspect, listed in ``_gcno.txt`` and carried in the reporter's - runfiles. gcovr renders the HTML from the merged data (one page per file, - under the canonical path) and the text summary; LCOV and - ``unmapped_files.txt`` are written as on the LLVM side. - -Line semantics differ between the backends and are recorded in the -integration ground truth: gcov counts only lines the compiler emitted code -for (no closing braces, no unused inline functions), while LLVM's mapping -keeps unused functions at 0 %. The justification and gating layer is -backend-agnostic; ``effective_coverage.py`` recognises gcovr's HTML layout. - -Phase 2: report generation and gate ------------------------------------ - -``generate_coverage_html.py`` unpacks the HTML from the zip and, when a -justification YAML is given, calls ``justify.py`` (YAML plus in-code markers to -a manifest of justified lines) and ``effective_coverage.py`` (recolours justified -lines, computes raw and effective figures, flags stale justifications, writes -``report.json`` and ``summary.txt``). The markdown summary is written next, then -the gate compares the unrounded gated percentage against ``COVERAGE_THRESHOLD``. -Optionally the HTML, LCOV, justification report and JUnit XMLs are assembled -into an artifacts tree. - -Exit code 2 is reserved for runs without a verdict, so a broken report, a bad -threshold or a tool failure can never look like a pass. - -Module and consumer split -------------------------- +The two inputs meet at the reporter: test measurements show what executed; +compiler information lets it also report untested files with zero counts. +This is why selecting tests alone is not enough to define the report. + +Two backends, one report interface +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A **backend** is the compiler-specific implementation of collection. Both +backends follow the diagram above and produce the same archive structure. + +.. list-table:: + :header-rows: 1 + :widths: 24 38 38 + + * - Choice + - LLVM backend + - gcov backend + * - Supported code and execution + - C++ and Rust; tests run on Linux + - C++; tests run on Linux or in QEMU for QNX + * - Compiler + - Clang for C++, Ferrocene/rustc for Rust + - GCC on Linux, QCC on QNX + +The archive contains HTML pages for readers, measurements in **LCOV** (a text format for file names, line and branch +counts), a text summary and a list of files without data. +On QNX, the test runner brings execution counters back from the QEMU virtual +machine to the host. Report generation and evaluation then run on the host. + +The common interface does **not** mean identical percentages. LLVM and gcov +can count different sets of lines, for example for unused inline functions. +Compare reports with that difference in mind; do not merge LLVM and gcov +results into one percentage. Rust is not measured by this gcov backend. +Headers vendored from external repositories are measured by gcov when the +workspace target declaring them is included in ``--instrumentation_filter``. +This filter requirement also applies to ordinary source files: Bazel's gcov +collector can discard their measurements even when they are in the coverage +scope. The tool's LLVM collector does not apply this additional file filter. +Both backends still need instrumentation enabled at compile time to produce +measurements. See :doc:`../manual/known_problems`. + +Phase 2: evaluate coverage +-------------------------- + +The ``generate_coverage_html`` command unpacks the report zip and prepares the +HTML output. It can also apply **justifications**: reviewed engineering explanations for +specific code that was not exercised. These are defined in a YAML file and +associated with source locations, including through in-code markers. The +repository's reviewers assess the engineering argument. The tool checks the +input structure and applies the entries to matching code locations; it does +not decide whether the argument is sound. + +- **Raw line coverage** is the proportion of measured lines that executed. +- **Effective line coverage** also credits validly justified, unexecuted lines. + Those lines remain unexecuted; the report marks them separately. + +For libraries A and B combined, the report contains 100 measurable lines: +all 80 lines in A executed, while none of the 20 lines in B executed. +Suppose 5 of B's unexecuted lines have applicable justifications: + +- **Combined raw coverage:** 80 executed lines / 100 total lines = **80 %**. +- **Combined effective coverage:** (80 executed + 5 justified) / 100 total + lines = **85 %**. + +Library A remains at 100 % coverage. The increase from 80 % to 85 % applies to +the combined effective result. Justifications give credit for 5 lines in B; +they do not change execution counts or remove lines from the total. + +The gate checks effective line coverage when a justification YAML is supplied +and raw line coverage otherwise. Branch coverage is reported where available, +but this gate checks lines. The diagram shows the decision for a usable report; +error handling and current limitations are described below. + +.. uml:: + + @startuml + skinparam activityBackgroundColor #FFF2CC + skinparam activityBorderColor #B58B00 + skinparam activityDiamondBackgroundColor #E2F0D9 + skinparam activityDiamondBorderColor #548235 + + start + :Read coverage report zip; + if (Justification YAML supplied?) then (yes) + :Apply justifications and mark them in HTML; + :Use effective line coverage\n(executed + justified lines); + else (no) + :Use raw line coverage\n(executed lines); + endif + if (Coverage meets required minimum?) then (yes) + :Pass; + else (no) + :Fail; + endif + stop + @enduml + +The YAML selects which metric the gate checks; the configured minimum decides +whether that metric passes. The minimum is ``COVERAGE_THRESHOLD`` (100 % by +default). A justification credits a line without making it an executed line. + +The command defines the following exit-code contract. Current deviations are +listed under :doc:`../manual/known_problems`. .. list-table:: :header-rows: 1 - :widths: 40 60 - - * - Lives in ``@score_coverage`` - - Role - * - ``score_coverage/merger.py`` - - per-test profraw to profdata; C++ ``objects_list.txt`` and Rust ELF - manifest discovery - * - ``score_coverage/reporter.py`` - - final merge, llvm-cov show/export/report, allowlist filtering, - ``--empty-profile`` baselines, rlib expansion, path normalisation; the - selection, staging and path helpers shared with the gcov backend - * - ``score_coverage/gcov_reporter.py`` - - gcov backend: per-test LCOV merge, ``.gcno`` baselines, gcovr HTML - * - ``score_coverage/coverage_scope.bzl`` - - the scope aspect and rule (CcInfo and CrateInfo) - * - ``score_coverage/reporter_wrapper.bzl``, ``defs.bzl`` - - the consumer-facing ``score_coverage_scope`` / ``score_coverage_reporter`` - API - * - ``score_coverage/justify.py``, ``effective_coverage.py``, - ``coverage_summary.py``, ``generate_coverage_html.py`` - - justification, summary and gating layer - * - ``//:enable_llvm_coverage_for_death_tests`` - - ``cc_feature`` for continuous-mode profiling + :widths: 12 18 70 + + * - Code + - Meaning + - Example + * - 0 + - Pass + - The selected coverage meets the minimum. + * - 1 + - Fail + - Coverage was evaluated and is below the minimum. + * - 2 + - Error + - A missing report, invalid threshold or justification-processing failure + prevents the run from completing successfully. + +The command can also write a Markdown summary and assemble an archive for +both passing and failing coverage results. The archive can contain HTML, LCOV, the missing-data list, justification details and JUnit test +results. An error can interrupt this process, so a complete archive is not +guaranteed. Missing data for individual source files can be +reported as warnings without causing exit code 2. + +Responsibilities of the tool and its consumer +--------------------------------------------- + +The **consumer** is the repository using ``@score_coverage``. The tool supplies +the collection and evaluation logic; the consumer decides what code to assess, +which tests to run, and what coverage is acceptable. .. list-table:: :header-rows: 1 - :widths: 40 60 - - * - Lives in the consumer repository - - Why it cannot move - * - ``score_coverage_scope(deps = [...])`` - - names the repository's production targets - * - ``score_coverage_reporter(...)`` - - carries the repository's LLVM tool labels (or, with - ``backend = "gcov"``, the toolchain's gcov) and workspace root; one - target per backend - * - ``coverage_justifications.yaml`` - - reviewed, repository-specific engineering arguments - * - MODULE.bazel toolchain blocks - - LLVM and Ferrocene pins are per-repository decisions - * - the ``coverage:llvm_cov`` and ``coverage:qnx`` bazelrc blocks - - bazelrc cannot be imported across modules - -Two wiring details make the external hosting work: every path in the generated -reporter launcher uses rlocation form because it mixes files from the consumer -(``_main``), ``score_coverage`` and toolchain repositories, and the baseline -manifest is resolved against ``_main`` explicitly because its entries are -consumer files. - -Design decisions ----------------- - -- **Report-time filtering on top of full instrumentation.** The scope - allowlist decides what the report shows; ``--instrumentation_filter`` is - set to the module's root package (``^//score[/:]``) so that every target is - compiled with counters. Bazel's guessed default covers only the packages of - the test targets: a target outside it is compiled without counters unless a - direct dep is instrumented (both backends), and on the gcov backend Bazel's - collector additionally drops the counters of every target outside the - filter. The reporters warn when files without test data sit in a directory - that is tested from a ``test/`` or ``tests/`` subdirectory (ERR-13). -- **Fail loud, never fail green.** Every input problem ends in exit 2. The gate - compares unrounded values and floors displayed percentages. -- **Gate on the LCOV, not on llvm-cov's text summary.** The text summary omits - baseline-only files; the LCOV includes them. -- **In-process tool calls.** ``generate_coverage_html`` imports the justification - tools instead of nesting ``bazel run``; this keeps one process, one exit code - and testable seams. -- **One report format for both backends.** The gcov backend replaces only the - final report step and produces the same zip layout, so phase 2, the - archive, the job summary and the qualification evidence are shared. Bazel's - per-test gcov collector is kept as it is: it is where the QNX transport - hands over, and re-implementing it would move the ``.gcda`` handling into - the qualified tool for no gain. + :widths: 30 35 35 + + * - Concern + - Provided by the tool + - Defined by the consumer + * - Scope + - Dependency traversal and file selection + - Production targets in ``score_coverage_scope`` + * - Collection + - Per-test LLVM collector and final reporters for both backends + - Test selection, compiler/tool versions and coverage configuration + * - Evaluation + - Justification processing, HTML annotation and coverage gate + - Reviewed justification YAML and ``COVERAGE_THRESHOLD`` + * - Publication + - Summary and archive generation + - Output locations and CI artifact upload + +Configuration examples and commands belong to the +:doc:`../manual/user_manual`. The following details are intended for readers +maintaining or extending the integration. + +Implementation details and design decisions +------------------------------------------- + +Scope traversal and source paths +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``coverage_scope.bzl`` implements a Bazel **aspect**, which visits supported +build targets along their dependencies. It collects checked-in files declared +in C++ ``srcs`` / ``hdrs`` and supported Rust source declarations. Generated +source files are not collected by this traversal. Files belonging to external +targets are excluded, but a workspace target can explicitly declare a header +from an external repository and thereby include it in scope. Merely forwarding +an external library's headers does not include them. + +The scope exports a file allowlist, the source files, a header path map and +backend-specific baseline manifests. The path map translates Bazel-generated +``_virtual_includes/`` names back to declared header paths. This lets a header +appear under its source name even when include-prefix settings change the +path seen by the compiler. + +Reporters stage sources using Bazel **runfiles**, the files made available to a +tool when it runs. Source lookup prefers runfiles and falls back to the +consumer workspace for local files. HTML pages are placed under canonical +source paths so report links do not depend on the original build directory. +Missing sources are warned about rather than guaranteed to produce a page. + +Instrumentation and baseline collection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The consumer's coverage configuration must instrument the production packages +as well as the tests. In the integration workspace, +``--instrumentation_filter=^//`` covers workspace packages; a consumer with +production code under ``//score`` can use ``^//score[/:]``. Relying on Bazel's +guessed filter can omit libraries tested from a separate ``test/`` package. +Report-time scope filtering then selects which instrumented files count. +The reporters warn about suspected omissions of this kind (ERR-13). + +For LLVM, ``--experimental_use_llvm_covmap`` and the C++ ``coverage`` feature +enable coverage maps; ``rules_rust`` enables Rust coverage instrumentation when +the toolchain provides coverage tools. Continuous profiling and runtime counter +relocation preserve counters through abnormal termination, such as death +tests. Rust branch coverage additionally needs the unstable +``-Zcoverage-options=branch`` option supported by the configured rolling +Ferrocene toolchain. + +For each test, ``merger.py`` merges raw LLVM profiles (``.profraw``) with +``llvm-profdata`` and records the instrumented objects in a zip. The final +``reporter.py`` merges the profiles from all tests and invokes ``llvm-cov`` +for HTML, LCOV and text output. + +The LLVM baseline uses ``llvm-cov --empty-profile`` on compiled scope objects. +Archives with members lacking coverage maps are expanded, and only members +with maps are passed on. This handles Rust's ``lib.rmeta`` member as well as +empty C++ translation units. Files found only in the baseline are added to the +LCOV output with zero counts. + +For gcov, the compiler emits ``.gcno`` notes and the running binary writes +``.gcda`` counters. The scope obtains notes through Bazel's +``InstrumentedFilesInfo``. Running ``gcov --json-format`` on notes without the +runtime counters supplies the baseline. On QNX, the runner sets +``GCOV_PREFIX`` in the guest and transports the counter archive back into the +host's ``COVERAGE_DIR``. Bazel's existing collector then handles the per-test +conversion to LCOV. ``gcov_reporter.py`` merges these per-test LCOV records, +adds baseline-only files and uses gcovr to render HTML and the text summary. +The QNX transport is provided by ``score_qnx_unit_tests``. + +Bazel integration and the report interface +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Bazel exposes two extension points: ``--coverage_output_generator`` for +per-test output and ``--coverage_report_generator`` for the combined report. +LLVM uses ``merger.py`` for the former; gcov keeps Bazel's ``lcov_merger``. +The consumer defines a ``score_coverage_reporter`` target for each backend in +use and connects it to the scope and tool labels. The final-report extension +point selects that target, which +launches ``reporter.py`` or ``gcov_reporter.py`` with the scope and tool paths. + +The launcher resolves files from the consumer, the coverage module and the +toolchain repositories using Bazel runfile names. Baseline manifest entries +are resolved against the consumer repository (``_main``). The consumer keeps +its toolchain pins in ``MODULE.bazel`` and its coverage flags in its own +bazelrc; the integration workspace provides configuration examples. + +Both reporters write a zip at ``bazel-out/_coverage/_coverage_report.dat``. +Despite the ``.dat`` extension, this is an archive containing ``html_report/``, +``lcov_report/lcov.dat`` and ``text_report/``. The latter includes the text +summary and ``unmapped_files.txt``. This shared interface allows both backends +to use the same evaluation and archive code. + +Evaluation modules +~~~~~~~~~~~~~~~~~~ + +``generate_coverage_html.py`` calls the evaluation modules in the same Python +process, avoiding nested ``bazel run`` invocations: + +- ``justify.py`` resolves YAML entries and code markers into ``manifest.json``. +- ``effective_coverage.py`` reads the HTML, annotates justified lines, flags + stale justifications and writes ``report.json`` and ``summary.txt``. +- ``coverage_summary.py`` reads LCOV, the optional justification report and + the missing-data list to produce the requested Markdown summary. + +Without YAML, the gate calculates raw line coverage from LCOV, which includes +baseline-only files. LLVM's text summary omits those files and is therefore +unsuitable for this decision. With YAML, the gate reads effective coverage +from ``report.json``, computed from the HTML report and applied justifications. +Thus the two paths use different report representations. + +The optional Markdown summary is written before the gate decision. The archive +is assembled afterwards for either verdict, so a coverage failure still leaves +reviewable results unless a separate processing error interrupts the run. diff --git a/docs/manual/known_problems.rst b/docs/manual/known_problems.rst index 40d7710..e7e9372 100644 --- a/docs/manual/known_problems.rst +++ b/docs/manual/known_problems.rst @@ -56,14 +56,22 @@ stay listed with their upstream references. - Rust sources listed as ``not-instrumented`` in ``unmapped_files.txt`` of a QNX report. - Measure Rust with the Linux (LLVM) run of the same tree. - * - **Vendored external headers have no data on the gcov backend.** Bazel's - per-test collector keeps only files of its instrumented-files manifest, - which never lists sources from external repositories, although gcov - itself recorded them. - - Such a header is ``no-data`` in a QNX report and measured in the Linux - report of the same tree. - - Known limitation of Bazel's collector; use the Linux report for those - headers. + * - **gcov discards test measurements when the instrumentation filter is + too narrow.** This affects source files and headers, including headers + imported from another repository. Bazel's gcov collector keeps only + measurements for files declared by targets included in + ``--instrumentation_filter``. Adding files to the coverage scope alone + is not enough. The tool's LLVM collector does not apply this additional + file filter, but both backends need instrumentation enabled at compile + time to produce measurements. + - Tests execute code in a file, but the gcov report shows no test data + for it, or only zero counts from the baseline. + - Make sure ``--instrumentation_filter`` includes the package of the + target declaring the affected files. For example, if + ``//third_party:headers`` declares imported + headers, a filter of ``^//score[/:]`` misses it. Use + ``--instrumentation_filter=^//`` to include all workspace packages, + or extend the narrower filter to include ``//third_party``. * - **gcov and LLVM count different lines.** gcov reports only lines the compiler emitted code for: unused inline functions and closing braces have no line, while LLVM's mapping keeps unused functions at 0 %. @@ -123,3 +131,36 @@ stay listed with their upstream references. output. - Set ``--instrumentation_filter=^//[/:]`` in every coverage config (user manual, step 4). + * - **Effective coverage can pass despite missing untested files.** If LLVM + cannot render HTML with the baseline archives, the reporter retries + using only test binaries. LCOV still includes baseline-only files, but + effective coverage uses the reduced HTML totals. + - Untested files appear in LCOV but are missing from HTML. Even an empty + justification YAML can change a failing raw result into a passing + effective result. + - Check that HTML includes the untested files listed in LCOV before + relying on the effective gate. Without justification YAML, the raw gate + uses LCOV and includes baseline-only files. + Tracked in `issue #14 `_. + * - **Missing effective-coverage totals can be treated as 0 %.** Missing or + unparseable HTML totals do not reliably produce an error. + - The command returns exit 1 instead of exit 2, or even exit 0 when the + threshold is 0, despite having no usable effective-coverage totals. + - Check that the HTML contains usable totals. The raw LCOV path rejects + zero measurable lines. + Tracked in `issue #15 `_. + * - **Malformed YAML can bypass the error-code handling.** A syntax error + in the justification YAML can terminate the command before it reports + the expected error status. + - A traceback and exit 1 instead of exit 2. + - Correct the YAML syntax; treat the traceback as a processing error, + rather than a failed coverage threshold. + Tracked in `issue #16 `_. + * - **The effective gate uses a floored percentage.** It reads the value + from ``report.json``, already floored to two decimal places. The raw + gate compares without display rounding. + - For example, effective coverage of 99.999 % becomes 99.99 % and fails + a threshold of 99.995 %. + - Account for the two-decimal precision when interpreting failures close + to the threshold. Both paths are intended to compare unrounded values. + Tracked in `issue #17 `_.