Skip to content

Commit d7bfcce

Browse files
committed
refactor project directory structure (#170)
* Rename `infrastructure/terraform-gcp/` to `infrastructure/gcp/` * Rename `assets/` to `.assets`/ * Rename `visualization/` to `dataviz/` (metabase and superset modules) * Update Terraform - Google provider version from `6.x` to `7.19.0` * Update main README to reflect the new directory structure
1 parent 31d457b commit d7bfcce

23 files changed

Lines changed: 12 additions & 7 deletions

README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Refer to the modules below for covered topics and tools.
1111
* Rust data ingestion
1212
* [ELT Ingestion with Airbyte](./module1-data-ingestion/airbyte/)
1313
* [data load tool (dlt)](./module1-data-ingestion/data-load-tool/)
14-
* [IaC with Terraform (Google Cloud Platform)](./infrastructure/terraform-gcp/)
14+
* [IaC with Terraform (Google Cloud Platform)](./infrastructure/gcp/)
1515

1616
### [Module 2: Workflow orchestration](./module2-workflow-orchestration/)
1717
* Workflow orchestration with Airflow 3.x
@@ -24,21 +24,26 @@ Refer to the modules below for covered topics and tools.
2424

2525
### [Module 4: Analytics engineering](./module4-analytics-engineering/)
2626
* [BigQuery and dbt](./module4-analytics-engineering/bigquery/)
27+
* Snowflake and dbt
2728
* Databricks and dbt
2829
* [Redshift and dbt](./module4-analytics-engineering/redshift/)
2930
* [ClickHouse and dbt](./module4-analytics-engineering/clickhouse/)
3031
* [PostgreSQL and dbt](./module4-analytics-engineering/postgres/)
3132
* [DuckDB and dbt](./module4-analytics-engineering/duckdb/)
32-
* [Data visualization with Superset/Metabase](./module4-analytics-engineering/visualization/)
33+
* [Data visualization with Metabase](./module4-analytics-engineering/dataviz/metabase/)
34+
* [Data visualization with Superset](./module4-analytics-engineering/dataviz/superset/)
3335

3436
### [Module 5: Batch processing](./module5-batch-processing/)
35-
* [PySpark 4.x](./module5-batch-processing/pyspark-4.x/)
37+
* [PySpark 4.x + Spark Connect](./module5-batch-processing/pyspark-4.x/)
38+
* [PySpark 3.x + Spark Connnect](./module5-batch-processing/pyspark-3.x/)
3639
* Spark + Kotlin API
3740
* Spark (Scala)
3841

3942
### [Module 6: Stream processing](./module6-stream-processing/)
43+
* PyFlink
4044
* [Stream processing with Kafka, ksqlDB and Kotlin](./module6-stream-processing/kotlin/)
4145
* [Kafka Streams with ksqlDB](./module6-stream-processing/ksqldb/)
4246

4347
### Extras
4448
* [LakeHouse with Delta, Iceberg, Hive](https://github.com/iobruno/lakehouse-labs/)
49+
* [Data Catalog with DataHub](https://github.com/iobruno/data-catalog-labs)
Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ On the GCP Console, create a new `Service Account` with the roles of:
1616

1717
Next, access the created service_account, and create a 'New Key' with `Key type: JSON`, and save it somewhere safe on your workstation
1818

19-
![terraform-service-account](../../assets/terraform-service-account-pt1.png)
20-
![terraform-service-account](../../assets/terraform-service-account-pt2.png)
19+
![terraform-service-account](../../.assets/terraform-service-account-pt1.png)
20+
![terraform-service-account](../../.assets/terraform-service-account-pt2.png)
2121

2222

2323
Now, export the environment variables `GOOGLE_APPLICATION_CREDENTIALS` pointing to the full path where the .json credentials file was downloaded/saved:
@@ -33,7 +33,7 @@ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcp-credentials.json
3333

3434
In [Google Cloud Storage](https://console.cloud.google.com/storage/browser?hl=en&project=iobruno-gcp-labs), create a bucket that Terraform will use as its backend to save state:
3535

36-
![terraform-state-bucket](../../assets/terraform-tfstate-bucket.png)
36+
![terraform-state-bucket](../../.assets/terraform-tfstate-bucket.png)
3737

3838

3939
**2.** Configure Terraform backend for GCS:
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 commit comments

Comments
 (0)