Require the instrumentation filter; warn when it was left to Bazel

docs/architecture/index.rst

@@ -236,9 +236,15 @@ consumer files.
 Design decisions
 ----------------
 
-- **Report-time filtering, not instrumentation filtering.** Instrumenting
-  everything and filtering by allowlist is what makes exact 0 % baselines
-  possible; ``--instrumentation_filter`` would hide untested files.
+- **Report-time filtering on top of full instrumentation.** The scope
+  allowlist decides what the report shows; ``--instrumentation_filter`` is
+  set to the module's root package (``^//score[/:]``) so that every target is
+  compiled with counters. Bazel's guessed default covers only the packages of
+  the test targets: a target outside it is compiled without counters unless a
+  direct dep is instrumented (both backends), and on the gcov backend Bazel's
+  collector additionally drops the counters of every target outside the
+  filter. The reporters warn when files without test data sit in a directory
+  that is tested from a ``test/`` or ``tests/`` subdirectory (ERR-13).
 - **Fail loud, never fail green.** Every input problem ends in exit 2. The gate
   compares unrounded values and floors displayed percentages.
 - **Gate on the LCOV, not on llvm-cov's text summary.** The text summary omits

docs/manual/known_problems.rst

@@ -110,7 +110,16 @@ stay listed with their upstream references.
      - ``WARNING: N in-scope sources were not found`` in the reporter log, an
        index row without a link.
      - Report it; every declared source is expected to be exported.
-   * - **Instrumentation filter appears ignored.**
-     - ``--instrumentation_filter`` has no visible effect.
-     - Expected: ``--experimental_use_llvm_covmap`` instruments everything;
-       filtering happens at report time through the scope allowlist.
+   * - **Libraries tested from a** ``test`` **subpackage show 0 % or no-data.**
+       Bazel guesses ``--instrumentation_filter`` from the packages of the
+       test targets and strips only a trailing ``/tests``; ``score/os`` tested
+       from ``score/os/test`` is outside the guess. On the gcov backend Bazel's
+       collector then drops the library's counters; on both backends a library
+       without an instrumented direct dep is compiled without counters.
+     - Whole directories at 0 % on QNX that are covered on Linux;
+       ``WARNING: N in-scope files have no test data although their directory
+       is tested from a test/ or tests/ subdirectory`` in the reporter log;
+       ``Using default value for --instrumentation_filter`` in the Bazel
+       output.
+     - Set ``--instrumentation_filter=^//<root>[/:]`` in every coverage
+       config (user manual, step 4).

docs/manual/user_manual.rst

@@ -176,6 +176,26 @@ Do **not** combine ``--config=llvm_cov`` with configs that append other
 ``--extra_toolchains`` (for example a GCC host config): the last toolchain wins
 resolution and a GCC toolchain produces no covmap data.
 
+.. _instrumentation_filter:
+
+Set the instrumentation filter explicitly, to the module's root package:
+
+.. code-block:: text
+
+   coverage:llvm_cov --instrumentation_filter=^//score[/:]
+
+Left unset, Bazel guesses the filter from the **packages of the test
+targets** and strips only a trailing ``/tests`` (plural) to reach the code
+under test; ``bazel coverage`` prints the guess as ``Using default value for
+--instrumentation_filter``. A library whose tests live in a ``test``
+subpackage (``score/os`` tested from ``score/os/test``) is then outside the
+filter. rules_cc still compiles it with counters when one of its direct deps
+is instrumented, which hides the problem for most libraries, but a library
+without such a dep is compiled without instrumentation and appears as
+``no-data``. On the gcov backend the consequence is worse (see step 4b). The
+reporters warn when the pattern is visible in the data (files without test
+data whose directory is tested from a ``test/`` or ``tests/`` subdirectory).
+
 Step 4b (optional): QNX on-target coverage, the gcov backend
 ------------------------------------------------------------
 
@@ -216,12 +236,20 @@ collector and points the final step at that reporter (copy and adapt the
    coverage:qnx --run_under=@score_qnx_unit_tests//src:run_under_qnx
    coverage:qnx --test_lang_filters=cc
    coverage:qnx --instrument_test_targets
+   coverage:qnx --instrumentation_filter=^//score[/:]
    coverage:qnx --noexperimental_use_llvm_covmap
    coverage:qnx --noexperimental_generate_llvm_lcov
    coverage:qnx --test_env=GENERATE_LLVM_LCOV --test_env=COVERAGE_GCOV_PATH --test_env=LLVM_PROFILE_CONTINUOUS_MODE
    coverage:qnx --coverage_output_generator=@bazel_tools//tools/test:lcov_merger
    coverage:qnx --coverage_report_generator=//tools/coverage:gcov_reporter_wrapper
 
+The instrumentation filter is **required** on this backend, not only
+advisable: Bazel's own collector converts counters only for the targets
+inside the filter, so a library outside Bazel's guessed filter shows 0 % even
+though its counters came back from the target (baselibs' ``score/os``: 13 %
+on QNX against 80 % on Linux before the line was added; see
+:ref:`the instrumentation filter <instrumentation_filter>`).
+
 The LLVM-only rustc flags (``-Zcoverage-options=branch`` and friends) stay
 out of the way as long as they live in their own ``coverage:llvm_cov`` config,
 as in Step 4. If your workspace puts them on the bare ``coverage`` command
@@ -240,10 +268,14 @@ Run and report as on Linux, with the platform filter for justifications:
        --yaml tools/coverage/coverage_justifications.yaml --archive-dir coverage_qnx_artifacts
 
 Known differences to the LLVM backend: gcov has no lines for unused inline
-functions and for closing braces; headers a workspace target vendors from an
-external repository are not measured (Bazel's collector filters them out) and
-appear as ``no-data``; the report lists the toolchain's line semantics, not
-LLVM's, so the two reports are compared per file, not merged.
+functions and for closing braces; gcov counts every conditional jump the
+compiler emits as a branch, including exception-handling edges, so branch
+percentages are structurally lower than LLVM's; the report lists the
+toolchain's line semantics, not LLVM's, so the two reports are compared per
+file, not merged. Headers a workspace target vendors from an external
+repository are measured as long as the vendoring target is inside the
+instrumentation filter (Bazel's collector keeps the sources of instrumented
+targets, external headers included).
 
 Step 5 (optional): justifications
 ---------------------------------

docs/release/release_notes.rst

@@ -23,7 +23,45 @@ Release notes
    :security: NO
    :realizes: wp__module_sw_release_note
 
-0.3.0 (unreleased)
+0.3.1 (unreleased)
+------------------
+
+In plain words
+~~~~~~~~~~~~~~
+
+**Libraries tested from a** ``test`` **subpackage are measured on QNX.** The
+first QNX report of baselibs showed ``score/os`` at 13 % where Linux showed
+80 %. The tests had run; their counters were thrown away on the way back.
+Bazel decides which packages take part in a coverage run by guessing from
+where the tests are, and it does not look one directory up from a ``test``
+subpackage. On Linux our own merger ignores that guess; on QNX Bazel's own
+collector obeys it. The fix is one line in the coverage config,
+``--instrumentation_filter=^//score[/:]``, now required by the user manual
+for both backends. The reporters warn when a report shows the pattern
+(files without data in a directory that is tested from ``test/`` or
+``tests/``), and the integration workspace contains such a library so the
+case stays covered.
+
+Details for integrators
+~~~~~~~~~~~~~~~~~~~~~~~
+
+- Add ``--instrumentation_filter=^//<root package>[/:]`` to the
+  ``coverage:llvm_cov`` and QNX configs (user manual, steps 4 and 4b). Without
+  it a library without an instrumented direct dependency is compiled without
+  counters on both backends, and on the gcov backend every library outside
+  the guessed filter loses its counters.
+- New warning in both reporters (``tool_req__coverage_instrumentation_hint``,
+  potential error ERR-13); ``known_problems`` and the architecture's design
+  decisions corrected: the filter is not irrelevant, it must name the module.
+- Integration workspace: ``//lib:cross_pkg`` tested from ``//lib/test``, in
+  both goldens; a gcov run with the guessed filter checks the warning.
+- Correction to the 0.3.0 notes: headers vendored from an external repository
+  **are** measured on the gcov backend once the vendoring target is inside
+  the instrumentation filter. The 0.3.0 statement described the guessed
+  filter, which left the vendoring target uninstrumented. The gcov golden of
+  the integration workspace now contains the vendored header.
+
+0.3.0 (2026-09-28)
 ------------------
 
 In plain words

docs/requirements/potential_errors.rst

@@ -135,6 +135,18 @@ requirements and constraints of use that mitigate the error.
        :need:`tool_req__coverage_gcov_baseline`,
        :need:`tool_req__coverage_report_unmapped`,
        :ref:`CSTR-08 <cstr_coverage_check_baselines>`
+   * - ERR-13
+     - Both backends: Bazel's guessed ``--instrumentation_filter`` leaves a
+       library outside the instrumented set (its tests live in a ``test``
+       subpackage). The library is compiled without counters, or on the gcov
+       backend its counters are dropped by Bazel's collector, and the report
+       shows 0 % or no-data instead of the measured coverage.
+     - low (coverage is under-reported, the safe direction; costs review
+       effort and hides the real state on QNX)
+     - good
+     - :need:`tool_req__coverage_instrumentation_hint`,
+       :need:`tool_req__coverage_validation_ground_truth`,
+       user manual step 4 (the filter line)
 
 Classification
 --------------
@@ -143,6 +155,7 @@ Tool impact: **yes**. An error in the *more coverage than real* direction lets a
 violation of the structural-coverage verification requirement go undetected.
 Tool error detection before qualification: **no** for ERR-02, ERR-03 and
 ERR-07; **weak** for ERR-11 and ERR-12 (a lost per-test record on QNX is
-only noticed by comparing against the Linux report). The expected tool confidence level is therefore **TCL LOW**, and the
+only noticed by comparing against the Linux report); ERR-13 under-reports and is
+detected by the reporters' warning and the integration test. The expected tool confidence level is therefore **TCL LOW**, and the
 qualification method of the S-CORE process, validation of the software tool,
 applies. The evaluation itself is recorded in the Tool Verification Report.

docs/requirements/tool_requirements.rst

@@ -585,6 +585,20 @@ Summary and archive
    paths preserved, also when the gate fails; a missing test-logs directory
    shall be an error.
 
+.. tool_req:: Instrumentation filter hint
+   :id: tool_req__coverage_instrumentation_hint
+   :version: 1
+   :implemented: YES
+   :tags: report, ERR-13
+   :safety: ASIL_B
+   :satisfies: stkh_req__coverage__uc_scope_completeness
+
+   When an in-scope file has no test data, no file of its directory has test
+   data, and a ``test`` or ``tests`` subdirectory of that directory has, both
+   reporters shall warn that Bazel's default ``--instrumentation_filter``
+   probably excluded the file and shall name the flag to set. The warning
+   shall list the files and shall not change the report.
+
 Validation
 ----------
 
@@ -600,4 +614,8 @@ Validation
    C++ and Rust units whose line and branch counts are derived by hand: the
    produced LCOV shall match the expected records exactly (``DA``, ``BRDA``,
    ``LF``, ``LH``, ``BRF``, ``BRH`` per file), including exact-0 % records for an
-   untested C++ library and an untested Rust binary.
+   untested C++ library and an untested Rust binary. The fixture shall contain
+   a library in a package without tests that is exercised from a ``test``
+   subpackage, measured on both backends with the explicit instrumentation
+   filter, and the gcov run shall be repeated with Bazel's guessed filter to
+   show the warning of :need:`tool_req__coverage_instrumentation_hint`.

docs/verification/verification_report.rst

@@ -53,10 +53,11 @@ Test inventory
      - 20
      - merge_profraw, merge_no_data, merge_tool_error
    * - ``//score_coverage/tests:reporter_test``
-     - 69
+     - 71
      - report_merged_profile, report_allowlist, report_baseline_zero,
        report_rlib_expansion, report_missing_baseline, report_relative_paths,
-       report_outputs, report_unmapped, scope_transitive
+       report_outputs, report_unmapped, scope_transitive,
+       instrumentation_hint
    * - ``//score_coverage/tests:gcov_reporter_test``
      - 21
      - gcov_merge, gcov_baseline, gcov_html, report_relative_paths,
@@ -79,9 +80,9 @@ Test inventory
    * - ``//score_coverage/tests/starlark:coverage_scope_tests`` (14 analysis tests)
      - 14
      - scope_transitive, scope_excludes, scope_baseline_objects, scope_gcno
-   * - ``integration_tests/run_integration_test.sh`` (22 end-to-end checks)
-     - 22
-     - validation_ground_truth, report_baseline_zero, report_relative_paths,
+   * - ``integration_tests/run_integration_test.sh`` (25 end-to-end checks)
+     - 25
+     - validation_ground_truth, instrumentation_hint, report_baseline_zero, report_relative_paths,
        report_allowlist, report_unmapped, gcov_merge, gcov_baseline, gcov_html,
        gate_exit_codes, gate_no_verdict, just_unknown_id, artifacts,
        summary_first

integration_tests/.bazelrc

@@ -29,6 +29,12 @@ test --test_output=errors
 # ============================================================================
 
 coverage:llvm_cov --nocache_test_results
+# Instrument every package of this workspace. Left unset, Bazel guesses the
+# filter from the packages of the test targets (stripping only a trailing
+# "/tests"), and a library tested from a "test" subpackage, like //lib, is
+# compiled without instrumentation unless one of its deps happens to be
+# instrumented. A module uses its root package, e.g. ^//score[/:].
+coverage:llvm_cov --instrumentation_filter=^//
 coverage:llvm_cov --cxxopt=-O0
 coverage:llvm_cov --combined_report=lcov
 coverage:llvm_cov --experimental_fetch_all_coverage_outputs
@@ -82,6 +88,10 @@ coverage:llvm_cov --@rules_rust//rust/settings:extra_rustc_flag=-Zcoverage-optio
 # ============================================================================
 # Use with: bazel coverage --config=gcov //src/... --build_tests_only
 coverage:gcov --nocache_test_results
+# See coverage:llvm_cov. On this backend Bazel's own collector additionally
+# drops the counters of every library outside the filter, even when it was
+# compiled with instrumentation through an instrumented dependency.
+coverage:gcov --instrumentation_filter=^//
 coverage:gcov --cxxopt=-O0
 coverage:gcov --combined_report=lcov
 coverage:gcov --dynamic_mode=off

integration_tests/expected_lcov.dat

@@ -21,6 +21,12 @@
 #                  module's own lines (37-44) are instrumented and hit once.
 #                  Branches: 17 (value < 0) both ways, 19 (value == 0) true
 #                  only => 3 of 4.
+# lib/cross_pkg.cpp library in a package without tests, exercised from
+#                  //lib/test (the baselibs layout); only measured because the
+#                  config sets --instrumentation_filter explicitly. pick(3)
+#                  once: entry (18), condition (19), `return 0` (23) and the
+#                  closing brace (24) hit; the `return 1` block (20-22) at 0;
+#                  branch 19 false direction only => 1 of 2.
 # src/coverable.cpp classify() called with -5 and 0 => line 24 (`positive`)
 #                  at 0, branch 21 false direction never taken => 3 of 4.
 # rust/main.rs     no test at all => every line 0 via the empty-profile
@@ -90,6 +96,21 @@ BRH:0
 LF:3
 LH:0
 end_of_record
+SF:lib/cross_pkg.cpp
+DA:18,1
+DA:19,1
+DA:20,0
+DA:21,0
+DA:22,0
+DA:23,1
+DA:24,1
+BRDA:19,0,0,0
+BRDA:19,0,1,1
+BRF:2
+BRH:1
+LF:7
+LH:4
+end_of_record
 SF:src/coverable.cpp
 DA:17,2
 DA:18,2

integration_tests/expected_lcov_gcov.dat

@@ -13,12 +13,18 @@
 # Ground truth for the gcov backend (GCC 12.2.0, the collection path QNX/QCC
 # takes on target), derived BY HAND from the fixture sources and
 # coverable_test, which is the only test of `bazel coverage --config=gcov
-# //src/...`. Compared like expected_lcov.dat: function records are ignored,
+# //src/... //lib/...`. Compared like expected_lcov.dat: function records are ignored,
 # one record per file. gcov semantics differ from LLVM's: a line is counted
 # when the compiler emitted code for it (no closing braces, no lines of
 # functions it never emitted), and BRDA lists both directions of every
 # conditional jump.
 #
+# lib/cross_pkg.cpp library in a package without tests, exercised from
+#                  //lib/test (the baselibs layout). Without the explicit
+#                  --instrumentation_filter Bazel's collector drops its
+#                  counters. pick(3) once: entry (17), condition (19) and
+#                  `return 0` (23) hit, `return 1` (21) at 0; no brace lines;
+#                  branch 19 false direction only => 1 of 2.
 # src/coverable.cpp classify() called with -5 and 0: entry (17) and the first
 #                  condition (18) twice; `negative` (19) once; the second
 #                  condition (21) and `zero` (22) once; `positive` (24) never.
@@ -33,13 +39,38 @@
 #                  gcov has no lines for 22-24 (the LLVM backend reports them
 #                  at 0). Reported under the declared path although the test
 #                  compiles it through the vendored_math_internal twin target.
-# (absent)         external/itest_external+/include/vext/vext.h: gcov records
-#                  the header, but Bazel's per-test merger keeps only files of
-#                  its instrumented-files manifest, which never lists sources
-#                  of external repositories; the file is listed as no-data.
+# external/itest_external+/include/vext/vext.h
+#                  header vendored from an external repository by the
+#                  workspace target //third_party:vendored_ext. Bazel's
+#                  per-test merger keeps only files of the instrumented-files
+#                  manifest, and that manifest lists the sources of the
+#                  instrumented targets, external headers included: with the
+#                  explicit --instrumentation_filter the vendoring target is
+#                  instrumented and the header is measured (without the
+#                  filter it was no-data). thrice() (18-19) called once by
+#                  coverable_test; never_used() is an unused inline function
+#                  with no code emitted => 2 of 2 lines, no branches.
 # (absent)         src/empty_unit.cpp (compiled without code), src/unused_api.h
 #                  (no data), rust/*.rs (not instrumentable by gcov),
 #                  extlib.cpp / extlib.h (out of scope), the test source.
+SF:external/itest_external+/include/vext/vext.h
+DA:18,1
+DA:19,1
+LF:2
+LH:2
+end_of_record
+SF:lib/cross_pkg.cpp
+DA:17,1
+DA:19,1
+DA:21,0
+DA:23,1
+BRDA:19,0,0,0
+BRDA:19,0,1,1
+BRF:2
+BRH:1
+LF:4
+LH:3
+end_of_record
 SF:src/coverable.cpp
 DA:17,2
 DA:18,2