Baraah A.

[thebaraah]

Summary

• Databricks Job Run URL: <paste-your-job-run-url-here>
• Task 1 (PySpark): Notebook in task-1/ with aggregated borough and payment type queries.
• Task 2 (dbt Incremental): Ported dbt project in task-2/ with materialized='incremental', merge strategy, timing comparison, and DESCRIBE HISTORY proof in WRITEUP.md.
• Task 3 (Job Scheduling): Scheduled Databricks Job from GitHub fork on hyf-dbt-warehouse with screenshots and orchestration comparison in task-3/SCHEDULING.md.

How to review

• Open task-2/WRITEUP.md to check initial vs incremental build times and DESCRIBE HISTORY output.
• Open task-3/SCHEDULING.md to verify the Databricks Job Run URL, screenshots, and Airflow comparison.
• Open AI_ASSIST.md for documented LLM interactions.

Secrets hygiene checklist

• No .env or profiles.yml files committed.
• No Databricks personal access tokens (dapi...) hardcoded in any notebook or SQL file.
• profiles.yml.example and .env.example templates present.

[github-actions]

📝 HackYourFuture auto grade

Assignment Score: 90 / 100 ✅

Status: ✅ Passed
Minimum score to pass: 60
🧪 The auto grade is experimental and still being improved

Test Details

=== Databricks Lab Autograder ===
  ✓ PASS  task-1 notebook: pyspark_exploration.py.ipynb
  ✓ PASS  task-2/dbt_project.yml
  ✓ PASS  task-2/WRITEUP.md
  ✓ PASS  task-3/SCHEDULING.md
  ✓ PASS  AI_ASSIST.md
  ✓ PASS  Level 1: required files (15/15 pts)
  ✓ PASS  notebook uses show()
  ✓ PASS  notebook addresses borough and/or payment_type
  ✓ PASS  Level 2: PySpark notebook (25/25 pts)
  ✓ PASS  fct_trips incremental materialization
  ✓ PASS  incremental merge config present
  ✗ FAIL  WRITEUP should explain is_incremental()
  ✗ FAIL  WRITEUP should reference {{ this }} or the incremental filter
  ✓ PASS  Level 3: dbt incremental + WRITEUP (20/30 pts)
  ✓ PASS  task-3/screenshots/ contains 4 screenshot(s)
  ✓ PASS  SCHEDULING.md contains Databricks Job Run URL
  ✓ PASS  SCHEDULING.md contains orchestration comparison
  ✓ PASS  Level 4: Task 3 Job Scheduling (15/15 pts)
  ✓ PASS  AI_ASSIST.md contains documented AI interaction
  ✓ PASS  .gitignore excludes .env and profiles.yml
  ✓ PASS  profiles.yml.example present
  ✓ PASS  Level 5: AI_ASSIST & secrets hygiene (15/15 pts)

Score: 90 / 100  (passing: 60)  pass=true
Total: 90/100 — pass=true (passing threshold: 60)

[qiraahmad]

Goob job, 13/15.

The notebook join and aggregations look right, the Databricks job is Git-backed (main / task-2), ran green on hyf-dbt-warehouse, and the schedule is paused.

The one thing to change: task-2/WRITEUP.md is still the empty template. You already have delta_history.png (CREATE OR REPLACE then MERGE). Add both dbt build times and a short explanation that names is_incremental() and {{ this }}.

Also delete leftover stg_test.sql / costumer.sql, and fix the duplicate version: 2 in _sources.yml plus the orphaned tests in schema.yml.