Skip to content

Commit 5e1d80d

Browse files
AlexanderLaninAlexanderLanin
authored
feat: score_pytest (#478)
Moving score_pytest from tooling to Docs-as-Code to reduce dependency to tooling and enable a more self enclosed module
1 parent 1969183 commit 5e1d80d

15 files changed

Lines changed: 511 additions & 12 deletions

File tree

‎.github/workflows/test.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
- name: Run test targets
3737
run: |
3838
bazel run --lockfile_mode=error //:ide_support
39-
bazel test --lockfile_mode=error //src/...
39+
bazel test --lockfile_mode=error //src/... //score_pytest/...
4040
4141
- name: Prepare bundled consumer report
4242
if: always()

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,6 @@ styles/
2323
.venv*
2424
__pycache__/
2525
/.coverage
26+
27+
# bug: This file is created in repo root on test discovery.
28+
/consumer_test.log

‎score_pytest.bzl‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# *******************************************************************************
2+
# Copyright (c) 2025 Contributors to the Eclipse Foundation
3+
#
4+
# See the NOTICE file(s) distributed with this work for additional
5+
# information regarding copyright ownership.
6+
#
7+
# This program and the accompanying materials are made available under the
8+
# terms of the Apache License Version 2.0 which is available at
9+
# https://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# SPDX-License-Identifier: Apache-2.0
12+
# *******************************************************************************
13+
"""Bazel interface for running pytest"""
14+
15+
load("@docs_as_code_hub_env//:requirements.bzl", "requirement")
16+
load("@rules_python//python:defs.bzl", "py_test")
17+
18+
def score_pytest(name, srcs, args = [], data = [], deps = [], env = {}, plugins = [], pytest_config = None, **kwargs):
19+
pytest_bootstrap = Label("@score_docs_as_code//score_pytest:main.py")
20+
21+
if not pytest_config:
22+
pytest_config = Label("@score_docs_as_code//score_pytest:pytest.ini")
23+
24+
if not srcs:
25+
fail("No source files provided for %s! (Is your glob empty?)" % name)
26+
27+
plugins = ["-p attribute_plugin"] + ["-p %s" % plugin for plugin in plugins]
28+
29+
docs_pytest = requirement("pytest")
30+
pytest_in_deps = False
31+
for dep in deps:
32+
if "//pytest:pkg" in str(dep):
33+
if str(dep) != str(docs_pytest):
34+
fail("Please do not provide your own pytest version. We want to use the same pytest version everywhere.")
35+
pytest_in_deps = True
36+
if not pytest_in_deps:
37+
deps = deps + [docs_pytest]
38+
39+
py_test(
40+
name = name,
41+
srcs = [
42+
pytest_bootstrap,
43+
] + srcs,
44+
main = pytest_bootstrap,
45+
args = [
46+
"-c $(location %s)" % pytest_config,
47+
"-p no:cacheprovider",
48+
49+
# XML_OUTPUT_FILE: Location to which test actions should write a test
50+
# result XML output file. Otherwise, Bazel generates a default XML
51+
# output file wrapping the test log as part of the test action. The XML
52+
# schema is based on the JUnit test result schema.
53+
"--junitxml=$$XML_OUTPUT_FILE",
54+
] +
55+
args +
56+
plugins +
57+
["$(location %s)" % x for x in srcs],
58+
deps = deps + ["@score_docs_as_code//score_pytest:attribute_plugin"],
59+
data = [
60+
pytest_config,
61+
] + data,
62+
env = env | {
63+
"PYTHONDONOTWRITEBYTECODE": "1",
64+
},
65+
**kwargs
66+
)

‎score_pytest/BUILD‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# *******************************************************************************
2+
# Copyright (c) 2025 Contributors to the Eclipse Foundation
3+
#
4+
# See the NOTICE file(s) distributed with this work for additional
5+
# information regarding copyright ownership.
6+
#
7+
# This program and the accompanying materials are made available under the
8+
# terms of the Apache License Version 2.0 which is available at
9+
# https://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# SPDX-License-Identifier: Apache-2.0
12+
# *******************************************************************************
13+
14+
load("@aspect_rules_py//py:defs.bzl", "py_library")
15+
load("//:score_pytest.bzl", "score_pytest")
16+
17+
exports_files([
18+
"pytest.ini",
19+
"main.py",
20+
])
21+
22+
score_pytest(
23+
name = "test_rules_are_working_correctly",
24+
srcs = glob(["tests/*.py"]),
25+
)
26+
27+
py_library(
28+
name = "attribute_plugin",
29+
srcs = ["attribute_plugin.py"],
30+
imports = ["."],
31+
visibility = ["//visibility:public"],
32+
)

‎score_pytest/README.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
<!--
2+
Copyright (c) 2026 Contributors to the Eclipse Foundation
3+
4+
SPDX-License-Identifier: Apache-2.0
5+
-->
6+
7+
# Score pytest wrapper & plugin
8+
9+
This module provides support for running [pytest](https://docs.pytest.org/en/latest/contents.html)-based tests with Bazel, and includes a **pytest plugin** that adds structured metadata to JUnit XML reports, improving traceability, test classification, and documentation.
10+
11+
---
12+
13+
## Features
14+
15+
- **Test Classification**: Categorize tests by type and derivation technique
16+
- **Requirements Traceability**: Link tests to requirement IDs
17+
- **Automatic File/Line Attribution**: Annotates tests with file path and line number
18+
- **JUnit XML Integration**: Exports metadata as `<properties>` in test reports
19+
- **Bazel Integration**: Run tests with `score_pytest` Bazel rule
20+
21+
---
22+
23+
## Usage
24+
25+
### In `MODULE.bazel`
26+
27+
Add a dependency to `score_docs_as_code` to use the `score_pytest` module.
28+
The module will determine the appropriate `pytest` version. It is not possible to override this.
29+
30+
---
31+
32+
### In `BUILD`
33+
34+
```starlark
35+
load("@score_docs_as_code//:score_pytest.bzl", "score_pytest")
36+
37+
# simple case:
38+
score_pytest(
39+
name = "test_my_first_check",
40+
srcs = ["test_my_first_check.py"],
41+
42+
# Optional custom pyproject.toml or pytest.ini
43+
# Recommended if you have one.
44+
# This will align CLI, IDE and bazel test behavior.
45+
pytest_config = "//:pyproject.toml",
46+
)
47+
48+
# all options:
49+
score_pytest(
50+
name = "test_my_first_check",
51+
srcs = ["test_my_first_check.py"],
52+
plugins = [
53+
# Optionally specify additional pytest plugins
54+
],
55+
args = [
56+
"--basetemp=/tmp/pytest", # Optional args
57+
],
58+
env = {
59+
"LD_LIBRARY_PATH": "/path/to/dynamic/lib", # Optional environment
60+
},
61+
pytest_config = "//:pytest_ini", # Optional custom pytest.ini
62+
tags = ["integration"] # Optional tags for test grouping
63+
)
64+
```
65+
66+
---
67+
68+
## Using the Test Properties Plugin
69+
70+
You can use the provided `add_test_properties` decorator to enhance your tests with metadata:
71+
72+
### Example
73+
74+
```python
75+
from your_module import add_test_properties
76+
77+
@add_test_properties(
78+
partially_verifies=["REQ-001", "REQ-002"],
79+
test_type="interface-test",
80+
derivation_technique="boundary-values"
81+
)
82+
def test_user_login():
83+
"""Test user login with valid credentials."""
84+
...
85+
```
86+
87+
### Required Parameters
88+
89+
- `test_type`: Type of test being executed
90+
- `derivation_technique`: Method used to derive the test
91+
- **Either** `partially_verifies` or `fully_verifies`: List of requirement IDs
92+
93+
---
94+
95+
### Accepted Values
96+
97+
#### Test Types
98+
99+
- `fault-injection`
100+
- `interface-test`
101+
- `requirements-based`
102+
- `resource-usage`
103+
104+
#### Derivation Techniques
105+
106+
- `requirements-analysis`
107+
- `design-analysis`
108+
- `boundary-values`
109+
- `equivalence-classes`
110+
- `fuzz-testing`
111+
- `error-guessing`
112+
- `explorative-testing`
113+
114+
---
115+
116+
### More Examples
117+
118+
```python
119+
@add_test_properties(
120+
fully_verifies=["REQ-AUTH-001"],
121+
test_type="requirements-based",
122+
derivation_technique="requirements-analysis"
123+
)
124+
def test_authentication_flow():
125+
"""Complete authentication flow test."""
126+
...
127+
128+
@add_test_properties(
129+
partially_verifies=["REQ-UI-001", "REQ-UI-002"],
130+
test_type="interface-test",
131+
derivation_technique="boundary-values"
132+
)
133+
def test_form_validation():
134+
"""Test form input validation boundaries."""
135+
...
136+
```
137+
138+
---
139+
140+
## Output in JUnit XML
141+
142+
When using `--junit-xml=report.xml`, the plugin augments each test case with:
143+
144+
- **Properties**:
145+
- `PartiallyVerifies` or `FullyVerifies`
146+
- `TestType`
147+
- `DerivationTechnique`
148+
- **File** and **Line**: Relative source path and line number of the test function
149+
150+
### Example Output
151+
152+
```xml
153+
<testsuites>
154+
<testsuite name="TestInterfaceValidation" tests="2" failures="0" errors="0" time="0.123">
155+
<testcase name="test_api_response_format" file="src/testfile_1.py" line="10" time="0.056">
156+
<properties>
157+
<property name="PartiallyVerifies" value="TREQ_ID_2, TREQ_ID_3"/>
158+
<property name="TestType" value="interface-test"/>
159+
<property name="DerivationTechnique" value="design-analysis"/>
160+
</properties>
161+
</testcase>
162+
<testcase name="test_error_handling" file="src/testfile_1.py" line="38" time="0.067">
163+
<properties>
164+
<property name="PartiallyVerifies" value="TREQ_ID_2, TREQ_ID_3"/>
165+
<property name="TestType" value="interface-test"/>
166+
<property name="DerivationTechnique" value="design-analysis"/>
167+
</properties>
168+
</testcase>
169+
</testsuite>
170+
</testsuites>
171+
```

0 commit comments

Comments
 (0)