Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .bazelrc
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,10 @@ build:time-arm64-qnx --extra_toolchains=@score_qcc_aarch64_toolchain//:aarch64-q
# -------------------------------------------------------------------------------
try-import %workspace%/user.bazelrc

# Coverage configuration for C++
coverage --features=coverage
coverage --combined_report=lcov
coverage --cache_test_results=no
# Coverage configuration — LLVM (Linux) + gcov (QNX).
# All coverage settings live in quality/coverage/coverage.bazelrc.
# Usage: bazel coverage //score/... --build_tests_only
import %workspace%/quality/coverage/coverage.bazelrc

# -------------------------------------------------------------------------------
# Sanitizer configurations — powered by score_cpp_policies
Expand Down
53 changes: 43 additions & 10 deletions .github/workflows/code-coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#
# SPDX-License-Identifier: Apache-2.0
# *******************************************************************************
name: Test / Code Coverage (>85%)
name: Test / Code Coverage Linux

permissions:
contents: read
Expand All @@ -29,12 +29,45 @@ on:
types: [checks_requested]

jobs:
code-coverage:
uses: eclipse-score/cicd-workflows/.github/workflows/cpp-coverage.yml@93aac16ada7d247bbb6ae926509ddea74cf5213a # v0.0.2
with:
bazel-target: "//score/..."
bazel-config: "time-x86_64-linux"
extra-bazel-flags: "--test_output=errors --nocache_test_results"
artifact-name-suffix: "_cpp"
retention-days: 10
min-coverage: 85
coverage-linux:
name: LLVM Coverage (Linux x86_64)
runs-on: ubuntu-24.04
permissions:
contents: read
actions: write

steps:
- uses: eclipse-score/more-disk-space@6a3b48901846bf7f8cc985925157d71a8973e61f # v1
with:
level: 4

- name: Checkout repository
uses: actions/checkout@v6

- name: Setup Bazel with shared caching
uses: bazel-contrib/setup-bazel@0.18.0
with:
disk-cache: ${{ github.workflow }}
repository-cache: true

- name: Run coverage
run: bazel coverage --build_tests_only --test_output=errors //score/...

- name: Check coverage thresholds
if: always()
run: |
output_path="$(bazel info output_path)"
unzip -o "${output_path}/_coverage/_coverage_report.dat" -d coverage_output/
run: python3 quality/coverage/check_coverage.py \
--coverage-dir coverage_output/ \
--min-line 85 \
--min-branch 70

- name: Upload coverage artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: inc_time_coverage_report_${{ github.sha }}
path: coverage_output/
if-no-files-found: ignore
retention-days: 10
7 changes: 6 additions & 1 deletion MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ bazel_dep(name = "score_logging", version = "0.2.1")
### Modules that are used internally within the repository but not exposed as part of the public API

bazel_dep(name = "score_docs_as_code", version = "4.5.0")
# Required by //quality/coverage/llvm_cov:enable_llvm_coverage_for_death_tests (cc_feature).
bazel_dep(name = "rules_cc", version = "0.2.17", dev_dependency = True)

bazel_dep(name = "score_cpp_policies", version = "0.0.1", dev_dependency = True)

Expand Down Expand Up @@ -120,8 +122,11 @@ llvm = use_extension(
)
llvm.toolchain(
llvm_version = "19.1.7",
extra_known_features = [
"//quality/coverage/llvm_cov:enable_llvm_coverage_for_death_tests",
],
)
use_repo(llvm, "llvm_toolchain")
use_repo(llvm, "llvm_toolchain", "llvm_toolchain_llvm")

# grpc-java@1.66.0 has a BCR bug (extension no longer generates
# com_envoyproxy_protoc_gen_validate). Pulled transitively via
Expand Down
30 changes: 29 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,37 @@ artifacts of the module.
### 3️⃣ Run Tests

```sh
bazel test //tests/...
bazel test --config=time-x86_64-linux //score/...
```

### 4️⃣ Run Coverage (Linux — LLVM source-based)

All coverage configuration lives in [`quality/coverage/`](quality/coverage/README.md).
See that README for the full pipeline architecture, scope configuration, and
how to extend coverage to new components.

```sh
bazel coverage --build_tests_only //score/...
```

The reporter generates an HTML report, LCOV data, and a text summary, packaged
in a zip at `$(bazel info output_path)/_coverage/_coverage_report.dat`.
Extract and check thresholds with the bundled script:

```sh
output_path="$(bazel info output_path)"
unzip -o "${output_path}/_coverage/_coverage_report.dat" -d coverage_output/
python3 quality/coverage/check_coverage.py \
--coverage-dir coverage_output/ \
--min-line 85 \
--min-branch 70
# HTML report: coverage_output/html_report/index.html
# LCOV data: coverage_output/lcov_report/lcov.dat
```

**QNX coverage** uses the gcov pipeline. Add `--config=time-x86_64-qnx` to
activate it; the LLVM settings are reset automatically.

---

## 🛠 Tools & Linters
Expand Down
39 changes: 39 additions & 0 deletions quality/coverage/BUILD
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# *******************************************************************************
# Copyright (c) 2026 Contributors to the Eclipse Foundation
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Apache License Version 2.0 which is available at
# https://www.apache.org/licenses/LICENSE-2.0
#
# SPDX-License-Identifier: Apache-2.0
# *******************************************************************************

load("//quality/coverage:coverage_scope.bzl", "coverage_scope")

# Defines the production source file scope for coverage reporting.
# The aspect traverses each dep's cc_library deps transitively —
# add new top-level production libraries here when they are not reachable
# via the deps of an existing entry.
coverage_scope(
name = "time_coverage_scope",
testonly = True,
visibility = ["//quality/coverage:__subpackages__"],
deps = [
# score/time: package-level facades; transitively cover clock_core,
# ptp_types, and all clock interfaces + implementations
"//score/time/vehicle_time:vehicle_time",
"//score/time/steady_time:steady_time",
"//score/time/system_time:system_time",
"//score/time/high_res_steady_time:high_res_steady_time",
# score/time_daemon: binary alias transitively covers svt_handler,
# job_runner, control_flow_divider, msg_broker, ptp_machine/shm,
# verification_machine and their common dep chains, plus the binary's
# own application sources
"//score/time_daemon:time_daemon",
# score/time_slave: binary alias covers application sources + gptp_engine
"//score/time_slave:time_slave",
],
)
184 changes: 184 additions & 0 deletions quality/coverage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Coverage Pipeline

This directory contains the LLVM source-based coverage pipeline for `score/time`,
`score/time_daemon`, and `score/time_slave`.

---

## Why LLVM Source-Based Coverage

GCC/gcov instruments at assembly level and generates **phantom branches** for:

- Exception-unwind edges on every non-`noexcept` call site
- GMock internal bookkeeping branches (all mock files)
- Assertion-abort edges only coverable via death-test subprocesses

These phantom branches permanently force mock files and PTP headers to ~50%
branch coverage regardless of test quality, making the 90% branch goal
unreachable with gcov.

LLVM source-based coverage (`llvm-cov`) tracks **source regions**, not assembly
branches. The result: no phantom branches, accurate metrics, and a 95%+ line
coverage baseline with gaps only from genuinely untested code paths.

---

## File Layout

```
quality/coverage/
├── BUILD # coverage_scope target (production file allowlist)
├── coverage.bazelrc # All Bazel coverage flags — imported by .bazelrc
├── coverage_scope.bzl # Starlark rule + aspect for allowlist generation
├── check_coverage.py # Per-component threshold gating script
└── llvm_cov/
├── BUILD # merger, reporter, reporter_wrapper targets
├── merger.py # Per-test profraw → profdata zip generator
├── reporter.py # Final HTML + LCOV report generator
└── reporter_wrapper.bzl # Bakes allowlist + baseline-objects into the launcher
```

---

## Architecture: Coverage Scope

The **`coverage_scope` rule** uses a Bazel aspect to traverse the `deps` graph of
declared production library targets and emit two generated files:

| File | Contents |
|---|---|
| `time_coverage_scope_allowlist.txt` | One workspace-relative source path per line |
| `time_coverage_scope_objects.txt` | Paths to `.a` archives for baseline coverage |

The **`reporter_wrapper` rule** generates a shell launcher that calls `reporter.py`
with `--coverage_allowlist` and `--baseline_objects` pre-baked. No manual
`--ignore_filename_regex` flags are needed.

### Production scope roots (`quality/coverage/BUILD`)

The scope is rooted at the package-level production library targets. The aspect
traverses their `deps` **transitively**, so adding a new sub-library as a dep of
an existing root automatically includes it.

When a new **top-level** production library is added (not reachable via any
existing root's dep chain), add it to `coverage_scope(deps=[...])` in
[`quality/coverage/BUILD`](BUILD).

### Baseline coverage

Files in the allowlist that compiled but were never exercised by any test appear
at **0% coverage** in the report rather than being silently omitted. This makes
coverage gaps visible without any test having to explicitly import the file.

---

## Pipeline Data Flow

```
bazel coverage --build_tests_only //score/...
├─ [per test] merger.py
│ profraw ──────────────► profdata zip
└─ [once, final] reporter_wrapper.sh
│ calls reporter.py with:
│ --coverage_allowlist=time_coverage_scope_allowlist.txt
│ --baseline_objects=time_coverage_scope_objects.txt
│ <Bazel standard coverage args>
└─ reporter.py
llvm-profdata merge all profdata ──► merged.profdata
llvm-cov export (LCOV) ──► lcov.dat
llvm-cov show (HTML) ──► html_report/index.html
[filtered to allowlist; baseline objects fill 0% gaps]
zip output ──► _coverage_report.dat
```

---

## Running Coverage Locally

```sh
# Run all tests and generate the coverage report
bazel coverage --build_tests_only //score/...

# Unpack the report
output_path="$(bazel info output_path)"
unzip -o "${output_path}/_coverage/_coverage_report.dat" -d coverage_output/

# Open HTML report
xdg-open coverage_output/html_report/index.html

# Check per-component thresholds
python3 quality/coverage/check_coverage.py \
--coverage-dir coverage_output/ \
--min-line 85 \
--min-branch 70
```

**QNX coverage** resets all LLVM flags and uses the gcov pipeline. Add
`--config=time-x86_64-qnx` to any coverage command to activate it.

---

## `check_coverage.py` — Threshold Gating

Parses `coverage_output/lcov_report/lcov.dat`, groups files by component
(`score/<component>/`), and prints a summary table:

```
Component Lines Branches
score/time 95.4% (245/257) 88.2% (90/102)
score/time_daemon 82.1% (...) ...
```

Exits with code 1 if any component falls below `--min-line` or `--min-branch`.
Used in CI (`code-coverage.yml`).

### Arguments

| Flag | Default | Description |
|---|---|---|
| `--coverage-dir` | required | Directory containing `lcov_report/lcov.dat` |
| `--min-line` | `85.0` | Minimum line coverage % per component |
| `--min-branch` | `0.0` | Minimum branch coverage % per component |

---

## Extending the Scope

### Adding a new clock domain

The new domain's production library (e.g. `//score/time/my_clock:my_clock`) is
automatically included if it is a transitive dep of any existing entry in
`coverage_scope(deps=[...])`. Otherwise add it explicitly:

```python
# quality/coverage/BUILD
coverage_scope(
name = "time_coverage_scope",
...
deps = [
...
"//score/time/my_clock:my_clock",
],
)
```

### Adding a new time_daemon sub-component

Same rule: if not reachable from `svt_handler` or `ipc/svt/receiver:factory`,
add the production library target to `coverage_scope(deps=[...])`.

---

## Bazel Feature: Death-Test Coverage

`enable_llvm_coverage_for_death_tests` (defined in [`llvm_cov/BUILD`](llvm_cov/BUILD))
adds `-mllvm -runtime-counter-relocation` compiler flags. This enables
`LLVM_PROFILE_CONTINUOUS_MODE=1` so that `ASSERT_DEATH` subprocesses write
`.profraw` files before the process exits, making death-test branches coverable.

The feature is wired via `llvm.toolchain(extra_known_features=[...])` in
`MODULE.bazel` and activated automatically for all coverage builds via
`coverage --features=enable_llvm_coverage_for_death_tests` in `coverage.bazelrc`.
Loading
Loading