Repository navigation
Question: testing dbt through this package #68
Description
Feel free to disregard if this is out of scope, but I'm curious how the developer or other users of this package are testing their mwaa/dbt projects so that they can deploy with confidence. My current process is basically:
- develop/run dbt locally
- create a dag for the project
- push to CI
- ci runs dbt from the cli with a config that should more closely match production
- deploy dbt project to s3
- mwaa picks up dag/code
- user manually clicks through mwaa dashboard to trigger a run
The interface with mwaa, dbt and this package is where I'm having the hardest time testing code since errors only come up when I'm deep into the process. Is there a way that I could be running dbt locally through this package, or in CI (basically without getting all the way to mwaa) that would allow for faster development?
I've looked through the tests for this package and am thinking about going that route, but it seems to miss some of the airflow issues that could come up. I was also thinking of installing airflow and running the dags from the airflow cli in CI, but again not sure if I'm missing something.
Any advice, even just high level, would be much appreciated.
Activity
tomasfarias commentedon Jun 11, 2022
We are using a combination of unit tests and integration tests:
Unit tests
Our unit tests follow Airflow's recommendations in Best Practices and are mainly concerned with testing whether our DAGs can load, have a correct structure (we do much more besides just running
Dbt*operators), and are parametrized correctly.Integration tests
For a complete integration testing, ideally I'd want a second MWAA environment, where to deploy my DAGs first, and I'd run my DAGs in a testing cluster that contains a subset of your production data. You can then make assertions on your data, for example using
DbtTestOperator, or regulardbt test.Of course, this has a significant added cost, as we would need to duplicate our entire production environment. To save some costs, instead of an entirely different cluster, one could use a different database or schema in the same production cluster. Moreover, the second MWAA environment may also be skipped, with some name-mangling of DAGs and usage of variables to parametrize the dbt target to the testing environment, or even triggering the DAGs in CI/CD (for example, our DAGs test is running a sample DAG).
Naturally, using the same systems as in production will stress production resources, so good tracking of production performance is required, especially if your SLAs are tight.
All of our DAGs are constantly running
DbtTestOperatorto ensure correctness in both testing and production environments. We do not deploy to production until our entire pipeline has successfully run in our testing environment. This does take quite a lot of time though and has slowed down our release cadence, so we are actually looking into fast tracking some changes by scoping down some test runs.About this package's tests
airflow-dbt-pythonis built on top of dbt and Airflow, so we do not test dbt or Airflow themselves. Moreover, covering all possible testing scenarios would be impossible due to how complex a dbt project can be. We do try to cover basic dbt features with some mock models, and I do want to cover all the extra featuresairflow-dbt-pythonprovides.For this reason, I'd be interested to know which Airflow issues that could come up are you referencing? It may be valuable to add some testing coverage here, depending on what they are.
Thank you.
tomasfarias commentedon Jun 22, 2022
I'll be converting this one to a discussion for the time being, as there are no issues to tackle.
If we arrive at the need to increase test coverage, we can create specific issues for that task.