dbt integration broken

[sdebruyn]

The dbt integration seems to be broken (all current latest versions from pypi.cloud.soda.io, not using the core versions). I run it from Python like this:

1. Create a scan object
2. Make sure this scan object has a configuration for Soda Cloud and for the data source we are using
3. Set the scan definition name and the data source name on the scan object
4. Create a DbtCloud object and pass it the scan object
5. Run .ingest on the dbt ingestor

  File "/tmp/ipykernel_40/3283421021.py", line 7, in <module>
    exit_value = ingestor.ingest()
                 ^^^^^^^^^^^^^^^^^
  File "/home/trusted-service-user/jupyter-env/python3.11/lib/python3.11/site-packages/soda/cloud/dbt.py", line 135, in ingest
    self.flush_test_results(
  File "/home/trusted-service-user/jupyter-env/python3.11/lib/python3.11/site-packages/soda/cloud/dbt.py", line 143, in flush_test_results
    scan_results = self.build_scan_results(checks)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trusted-service-user/jupyter-env/python3.11/lib/python3.11/site-packages/soda/cloud/dbt.py", line 151, in build_scan_results
    check_dicts = [check.get_cloud_dict() for check in checks]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trusted-service-user/jupyter-env/python3.11/lib/python3.11/site-packages/soda/cloud/dbt.py", line 151, in <listcomp>
    check_dicts = [check.get_cloud_dict() for check in checks]
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trusted-service-user/jupyter-env/python3.11/lib/python3.11/site-packages/soda/execution/check/dbt_check.py", line 36, in get_cloud_dict
    "name": self.name,
            ^^^^^^^^^
  File "/home/trusted-service-user/jupyter-env/python3.11/lib/python3.11/site-packages/soda/execution/check/check.py", line 235, in name
    jinja_resolve = self.data_source_scan.scan.jinja_resolve
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'scan'

So the error is because the check objects created by the ingestor don't have a data source set and the check.name uses the data source to build its name.

[tools-soda]

CLOUD-9200

[sdebruyn]

Also, please fix your version for Core. The Cloud version has some newer code.

[adkinsty]

@sdebruyn this is a Soda cloud capability and is not intended to be used in soda core, so any changes to soda-dbt will happen to our commercial packages. Secondly, I don't think you are using the connector as intended. The recommended usage is via a CLI command like this:

soda ingest dbt -d my_datasource_name  -c /path/to/configuration.yml  --dbt-manifest path/to/manifest.json --dbt-run-results path/to/run_results.json

as described in our docs here.

Could you please try the command like this after installing soda-dbt from the Soda Pypi server into a new python venv and lmk if you are still facing issues?

[sdebruyn]

@adkinsty What is the use case for this and this? Last update 2 days ago.

[adkinsty]

The update you're referring to above is an automated increment of the version number that happens to all of the sub-modules of soda core. However, nonetheless, there is no use case for the dbt connector aside from uploading dbt test results to soda cloud.

As for the history of soda-core and why the dbt connector was originally included there, I am not sure tbh. I'd need to ask one of the original developers.

Anyways, as mentioned above, you should use soda-dbt installed from the Soda PyPi registry, since that is the version that is being maintained, and you should use it with the CLI as described in our docs.

[sreyas-lankala]

Hi @sodadata team! I'm a Data Engineering student currently using dbt and Snowflake. I've noticed this integration issue and would like to help investigate and fix it. Can I take this on?