Hannah N

[hannahwn]

Summary

• Databricks Job Run URL: [<paste-your-job-run-url-here>](https://adb-7405619530719547.7.azuredatabricks.net/jobs/727982853732525/runs/959594307168856?o=7405619530719547)
• Task 1 (PySpark): Notebook in task-1/ with aggregated borough and payment type queries.
• Task 2 (dbt Incremental): Ported dbt project in task-2/ with materialized='incremental', merge strategy, timing comparison, and DESCRIBE HISTORY proof in WRITEUP.md.
• Task 3 (Job Scheduling): Scheduled Databricks Job from GitHub fork on hyf-dbt-warehouse with screenshots and orchestration comparison in task-3/SCHEDULING.md.

How to review

• Open task-2/WRITEUP.md to check initial vs incremental build times and DESCRIBE HISTORY output.
• Open task-3/SCHEDULING.md to verify the Databricks Job Run URL, screenshots, and Airflow comparison.
• Open AI_ASSIST.md for documented LLM interactions.

Secrets hygiene checklist

• No .env or profiles.yml files committed.
• No Databricks personal access tokens (dapi...) hardcoded in any notebook or SQL file.
• profiles.yml.example and .env.example templates present.

[github-actions]

📝 HackYourFuture auto grade

Assignment Score: 90 / 100 ✅

Status: ✅ Passed
Minimum score to pass: 60
🧪 The auto grade is experimental and still being improved

Test Details

=== Databricks Lab Autograder ===
  ✓ PASS  task-1 notebook: pyspark_exploration.ipynb.ipynb
  ✓ PASS  task-2/dbt_project.yml
  ✓ PASS  task-2/WRITEUP.md
  ✓ PASS  task-3/SCHEDULING.md
  ✓ PASS  AI_ASSIST.md
  ✓ PASS  Level 1: required files (15/15 pts)
  ✓ PASS  notebook uses show()
  ✓ PASS  notebook addresses borough and/or payment_type
  ✓ PASS  Level 2: PySpark notebook (25/25 pts)
  ✓ PASS  fct_trips incremental materialization
  ✓ PASS  incremental merge config present
  ✓ PASS  WRITEUP explains incremental behavior
  ✗ FAIL  WRITEUP should reference {{ this }} or the incremental filter
  ✓ PASS  Level 3: dbt incremental + WRITEUP (25/30 pts)
  ✓ PASS  task-3/screenshots/ contains 3 screenshot(s)
  ✓ PASS  SCHEDULING.md contains Databricks Job Run URL
  ✓ PASS  SCHEDULING.md contains orchestration comparison
  ✓ PASS  Level 4: Task 3 Job Scheduling (15/15 pts)
  ⚠ WARN  AI_ASSIST.md needs your documented prompt and rationale
  ✓ PASS  .gitignore excludes .env and profiles.yml
  ✓ PASS  profiles.yml.example present
  ✓ PASS  Level 5: AI_ASSIST & secrets hygiene (10/15 pts)

Score: 90 / 100  (passing: 60)  pass=true
Total: 90/100 — pass=true (passing threshold: 60)

[qiraahmad]

Strong technical work; documentation needs one more pass. I'd give it 13/15.

You configured incremental fct_trips correctly and captured real build timings, and your Delta history shows CREATE OR REPLACE then MERGE.

To finish the submission, expand WRITEUP.md so the second build is explained in your own words about how the incremental config is functioning. Point to history_fct_trips.png as your DESCRIBE HISTORY proof rather than leaving the template placeholders.

AI_ASSIST.md still needs one real LLM interaction where it has the prompt you sent, what came back, and what you kept, changed, or discarded.

On Task 1, your aggregations are fine, but avoid trips.count() on the full ~128M-row table and the extra collect() calls after each aggregation. One show() on the small aggregated result is enough. Finally, align profiles.yml.example with dbt_project.yml, as both should use the profile name nyc_taxi_borough_daily.