You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CONTRIBUTE.md
+37-9Lines changed: 37 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,30 +1,42 @@
1
1
# DVE Contributing guidelines
2
2
3
-
# Developer information
3
+
__If you're planning to contribute to the DVE, please follow all the guidance below. Failure to follow the guidance in this document may result in your contributions being automatically rejected.__
4
4
5
5
## Getting started
6
6
7
-
To begin with I would recommend that users read all the documentation available within the [docs](./docs/). It gives an overview of how the DVE works and how to work with the dischema json document.
7
+
I would recommend that you read all the documentation available within the [docs](https://nhsdigital.github.io/data-validation-engine/). It gives an overview of how the DVE works and how to work with the dischema json document.
8
8
9
9
## General requirements
10
-
To start contributing to the DVE project you will need the following tooling available:
10
+
11
+
To start contributing to the DVE project you will need the tooling listed within the `.tool-versions` or `mise.toml`. The following tools are required because...
12
+
11
13
| Tool | Version | Reason |
12
14
| ---- | ------- | ------ |
13
-
|`Python`| 3.7.17|Currently supported version of `Python` for the DVE. |
14
-
|`Poetry`|1.4.2| Build and venv tool used for the DVE. |
15
+
|`Python`| 3.11|Latest version of Python supported by the DVE. |
16
+
|`Poetry`|2.2.1| Build and package manager tool used for the DVE. |
15
17
|`Java`| java liberica-1.8.0 |`Java` version required for `PySpark`. |
16
-
|`pre-commit`|2.21.0 | Currently installed as part of the `poetry` venv but seperate installation is fine. |
17
-
|`commitizen`|3.9.1 | Like `pre-commit`, installed as part of the `poetry` venv but seperate installation is fine. This is used to manage commits and automated semantic versioning. |
18
+
|`pre-commit`|4.3.0 | Currently installed as part of the `poetry` venv but seperate installation is fine. |
19
+
|`commitizen`|4.9.1 | Like `pre-commit`, installed as part of the `poetry` venv but seperate installation is fine. This is used to manage commits and automated semantic versioning. |
18
20
|`git-secrets`| Latest | Utilised as part of the `pre-commit` to ensure that no secrets are commited to the repository. There is a helper installation script within [scripts](/scripts/git-secrets/). |
19
21
20
-
Additionally, we have created a [asdf support](.tool-versions) and [mise-en-toml](.mise.toml) for those utilising `asdf` or `mise-en-toml` software.
22
+
You can install all the developer requirements with the following command:
23
+
24
+
```bash
25
+
poetry install --with dev
26
+
```
21
27
22
28
## Testing Requirements
23
29
24
30
Testing requirements are given in [pyproject.toml](./poetry.toml#48) under `tool.poetry.group.test.dependencies`. These are always pinned versions for consistency, but should be updated regularly if new versions are released. The following core packages are used for testing:
25
31
-[pytest](https://docs.pytest.org/en/stable/): Used for Python unit tests, and some small e2e tests which check coverage.
26
32
-[behave](https://github.com/behave/behave): Used for full, business-driven end-to-end tests.
27
-
-[coverage](https://coverage.readthedocs.io/en/7.10.7/): Used to get coverage for `pytest` tests.
33
+
-[coverage](https://coverage.readthedocs.io/en/): Used to get coverage for `pytest` tests.
34
+
35
+
You can install these requirements with the following command:
36
+
37
+
```bash
38
+
poetry install --with test
39
+
```
28
40
29
41
## Linting/Formatting/Type Checking Requirements
30
42
@@ -38,6 +50,12 @@ This mostly breaks down to:
38
50
39
51
We use these tools to ensure that code quality is not excessively compromised, even when working at pace.
40
52
53
+
You can install these requirements with the following command:
54
+
55
+
```bash
56
+
poetry install --with lint
57
+
```
58
+
41
59
## Installation for Development
42
60
43
61
We are utilising Poetry for build dependency management and packaging. If you're on a system that has `Make` available, you can simply run `make install` to setup a local virtual environment with all the dependencies installed (this won't install Poetry for you).
@@ -50,6 +68,16 @@ Tests should be run after installing the package for development as outlined abo
50
68
- To check the coverage run `poetry run coverage report -m`
51
69
- To run the behave tests, run `poetry run behave tests/features` (these are not included in coverage calculations)
52
70
71
+
## Committing
72
+
73
+
We use [commitizen](https://github.com/commitizen-tools/commitizen) to commit new changes. This ensures...
74
+
75
+
1. A consistent standard for the commit messages
76
+
2. Generation of changelog from the commit messages
77
+
3. Allows for automatic bumping of the version based on the changes
78
+
79
+
Please use `poetry run cz c` or `cz c` (if already in the venv).
80
+
53
81
## Submitting a pull request
54
82
55
83
If you want to contribute to the DVE then please follow the steps below:
@@ -25,11 +25,11 @@ If you'd like more detailed documentation around these services the please read
25
25
26
26
The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for `Spark` and `DuckDB`. So, if you need a `MySQL` backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on [requesting new features and raising bug reports here](#requesting-new-features-and-raising-bug-reports).
27
27
28
-
Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing][#Contributing] section.
28
+
Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing](#Contributing) section.
29
29
30
30
## Installation and usage
31
31
32
-
The DVE is a Python package and can be installed using `pip`. As of release v0.6.1 we currently support Python 3.10 & 3.11, with Spark version 3.4 and DuckDB version of 1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
32
+
The DVE is a Python package and can be installed using package managers such as [pip](https://pypi.org/project/pip/). As of the latest release we support Python 3.10 & 3.11, with Spark v3.4 and DuckDB v1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
33
33
34
34
If you're planning to use the Spark backend implementation, you will also need OpenJDK 11 installed.
35
35
@@ -38,12 +38,12 @@ Python dependencies are listed in `pyproject.toml`.
38
38
To install the DVE package you can simply install using a package manager such as [pip](https://pypi.org/project/pip/).
Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
44
+
*Note - Only versions >=0.6.2 are available on PyPi. For older versions please install directly from the git repo or build from source.*
45
45
46
-
Please note - The long term aim is to make the DVE available via PyPi and Conda but we are not quite there yet. Once available this documentation will be updated to contain the new installation options.
46
+
Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
47
47
48
48
Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. __Use at your own risk__.
49
49
@@ -60,10 +60,10 @@ Below is a list of features that we would like to implement or have been request
60
60
| ------- | --------------- | --------- |
61
61
| Open source release | 0.1.0 | Yes |
62
62
| Uplift to Python 3.11 | 0.2.0 | Yes |
63
-
| Upgrade to Pydantic 2.0 |Not yet confirmed| No |
63
+
| Upgrade to Pydantic 2.0 |Before 1.0 release| No |
64
64
| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No |
65
65
66
-
Beyond the Python upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
66
+
Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
0 commit comments