Skip to content

Commit d0bc7a5

Browse files
georgeRobertsongeorgeRobertson
committed
ci: ci conda upload
1 parent ebea1ee commit d0bc7a5

8 files changed

Lines changed: 122 additions & 115 deletions

File tree

‎.github/workflows/ci_conda_publish.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: CI Conda Publish
2+
3+
on: workflow_dispatch
4+
5+
jobs:
6+
build:
7+
publish-to-conda:
8+
name: Publish to Conda
9+
# if: startsWith(github.ref, 'refs/tags/')
10+
# needs:
11+
# - publish-to-pypi
12+
runs-on: conda/miniconda3:latest
13+
environment: conda
14+
steps:
15+
- name: Install conda requirements
16+
run: |
17+
conda install anaconda-client conda-build
18+
19+
- name: Install Grayskull
20+
run: |
21+
conda install -c conda-forge grayskull=3.1.0
22+
23+
- name: Build conda metadata
24+
run: |
25+
conda grayskull pypi data-validation-engine
26+
27+
- name: Replace incorrect values
28+
run: |
29+
sed -i 's/- AddYourGitHubIdHere/- georgeRobertson\n - stevenhsd/' ./data-validation-engine/meta.yaml
30+
sed -i 's/- data_validation_engine/- dve/' ./data-validation-engine/meta.yaml
31+
32+
- name: Build Conda packages
33+
run: |
34+
conda config --set anaconda_upload no
35+
make cdist
36+
37+
- name: upload dist
38+
env:
39+
ANACONDA_TOKEN = ${{ secrets.ANACONDA_TOKEN }}
40+
ANACONDA_USER = ${{ secrets.ANACONDA_USER }}
41+
run: |
42+
anaconda upload conda_dist/noarch/*.conda --token "$ANACONDA_TOKEN" --user "$ANACONDA_USER"
43+
44+
- name: cleanup
45+
run: conda build purge

.github/workflows/ci_publish.yml renamed to .github/workflows/ci_pypi_publish.yml

Lines changed: 1 addition & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: CI Publish
1+
name: CI PyPi Publish
22

33
on: workflow_dispatch
44

@@ -55,35 +55,3 @@ jobs:
5555

5656
- name: publish
5757
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
58-
59-
publish-to-conda:
60-
name: Publish to Conda
61-
if: startsWith(github.ref, 'refs/tags/')
62-
needs:
63-
- publish-to-pypi
64-
runs-on: conda/miniconda3:latest
65-
environment: conda
66-
steps:
67-
- name: Install conda requirements
68-
run: |
69-
conda install anaconda-client conda-build
70-
71-
- name: Install Grayskull
72-
run: |
73-
conda install -c conda-forge grayskull=3.1.0
74-
75-
- name: Build conda metadata
76-
run: |
77-
conda grayskull pypi data-validation-engine
78-
79-
- name: Build Conda packages
80-
run: |
81-
conda config --set anaconda_upload no
82-
conda build . -c conda-forge --output-folder ./conda_dist/
83-
84-
- name: upload dist
85-
env:
86-
ANACONDA_TOKEN = ${{ secrets.ANACONDA_TOKEN }}
87-
run: |
88-
anaconda upload conda_dist/*.tar.bz2 --token "$ANACONDA_TOKEN" --user "$ANACONDA_USER"
89-
anaconda upload conda_dist/*.conda --token "$ANACONDA_TOKEN" --user "$ANACONDA_USER"

‎.mise.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
[tools]
22
python="3.11"
3-
poetry="2.2.1"
3+
poetry="2.3.3"
44
java="liberica-1.8.0"

‎.tool-versions‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
python 3.11.14
2-
poetry 2.2.1
2+
poetry 2.3.3
33
java liberica-1.8.0

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,9 @@ wheel:
1212

1313
dist: wheel
1414

15+
cdist:
16+
conda build . -c conda-forge --output-folder ./conda_dist/
17+
1518
# testing
1619
behave:
1720
${activate} behave

‎README.md‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010

1111
The Data Validation Engine (DVE) is a configuration driven data validation library built and utilised by NHS England. Currently the package has been reverted from v1.0.0 release to a 0.x as we feel the package is not yet mature enough to be considered a 1.0.0 release. So please bear this in mind if reading through the commits and references to a v1+ release when on v0.x.
1212

13-
As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` file and example files can be accessed [here](./tests/testdata/). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here](./docs/).
13+
As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` file and example files can be accessed [here](https://github.com/NHSDigital/data-validation-engine/tree/main/tests/testdata). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here](https://nhsdigital.github.io/data-validation-engine/).
1414

1515
Once a dischema file has been defined, you are ready to use the DVE. The DVE is typically orchestrated based on four key "services". These are...
1616

@@ -21,7 +21,7 @@ Once a dischema file has been defined, you are ready to use the DVE. The DVE is
2121
| 3. | Business Rules | The business rules service will perform more complex validations such as comparisons between fields and tables, aggregations, filters etc to generate new entities. |
2222
| 4. | Error Reports | The error reports service will take all the errors raised in previous services and surface them into a readable format for a downstream users/service. Currently, this implemented to be an excel spreadsheet but could be reconfigured to meet other requirements/use cases. |
2323

24-
If you'd like more detailed documentation around these services the please read the extended documentation [here](./docs/).
24+
If you'd like more detailed documentation around these services the please read the extended documentation [here](https://nhsdigital.github.io/data-validation-engine/).
2525

2626
The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for `Spark` and `DuckDB`. So, if you need a `MySQL` backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on [requesting new features and raising bug reports here](#requesting-new-features-and-raising-bug-reports).
2727

@@ -43,7 +43,7 @@ pip install data-validation-engine
4343

4444
*Note - Only versions >=0.6.2 are available on PyPi. For older versions please install directly from the git repo or build from source.*
4545

46-
Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
46+
Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](https://nhsdigital.github.io/data-validation-engine/).
4747

4848
Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. __Use at your own risk__.
4949

@@ -56,17 +56,20 @@ If you have feature request then please follow the same process whilst using the
5656

5757
## Upcoming features
5858
Below is a list of features that we would like to implement or have been requested.
59-
| Feature | Release Version | Released? |
60-
| ------- | --------------- | --------- |
61-
| Open source release | 0.1.0 | Yes |
62-
| Uplift to Python 3.11 | 0.2.0 | Yes |
63-
| Upgrade to Pydantic 2.0 | Before 1.0 release | No |
64-
| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No |
65-
66-
Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
59+
| Feature | Release Version | Released? |
60+
| ------------------------------------------------------------------------------- | ----------------- | --------- |
61+
| Open source release | 0.1.0 | Yes |
62+
| Uplift to Python 3.11 | 0.2.0 | Yes |
63+
| Uplift Pyspark to 3.5 | TBA | No |
64+
| Allow DVE to run on Python 3.12+ | TBA | No |
65+
| Upgrade to Pydantic 2.0 | TBA | No |
66+
| Uplify Pyspark to 4.0+ | TBA | No |
67+
| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No |
68+
69+
Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#Contributing) section and get involved.
6770

6871
## Contributing
69-
Please see guidance [here](./CONTRIBUTE.md).
72+
Please see guidance [here](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).
7073

7174
## Legal
7275
This codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

0 commit comments

Comments
 (0)