Halyna R.

AI_ASSIST.md

@@ -1,12 +1,18 @@
 # AI assistance log
 
 <!-- Document at least one point where you used an LLM on this assignment.
-     Never paste connection strings, passwords, or real data. Fill in each field. -->
+     Never paste connection strings, passwords, or real data. Replace TODO. -->
 
 ## Use 1
 
-**Prompt I sent:** _Replace this section._
+**Prompt I sent:**
 
-**What the model answered:** _Replace this section._
+Docker Desktop was stuck on “Starting the Docker Engine”, and Windows showed that the paging file was too small. I asked what safe steps I could try without deleting my project data.
 
-**What I kept, changed, or discarded, and why:** _Replace this section._
+**What the model answered:** 
+
+The model suggested restarting Windows, shutting down WSL, stopping unused Docker containers, and starting only one Astro project. It also warned me not to reset Docker to factory settings or remove Docker volumes because this could delete local Airflow data and settings.
+
+**What I kept, changed, or discarded, and why:** 
+
+I followed the safe steps: I restarted the computer, opened only the current project, and stopped containers from older projects. I did not uninstall Docker or delete any Docker volumes because I wanted to keep my local Airflow history and connection settings. I checked the result myself by running `docker ps` and reopening the Airflow interface. No passwords, connection strings, or private data were shared with the model.

ASSIGNMENT_REPORT.md

@@ -1,31 +1,89 @@
 # Assignment report
 
-<!-- Fill in every section below. Keep it short: a few sentences each. -->
+<!-- Replace every TODO. Keep it short: a few sentences per section. -->
 
 ## Schedule choice and reason
 
-_Replace this section._
+I chose `@monthly` because TLC publishes the taxi parquet files by month and the ingest task owns one monthly partition per Airflow run. This prevents a daily schedule from repeatedly loading the same monthly file. Normal operation uses `catchup=False`; historical data is loaded deliberately with `backfill create`.
 
 ## Task dependency graph
 
-_Replace: describe ingest -> dbt_run -> dbt_test and why order matters._
+The strict dependency chain is:
+
+```text
+ingest_taxi_month → dbt_run → dbt_test
+```
+
+The ingest task must finish before dbt builds the models, and `dbt_test` must run only after a successful build. If ingestion fails, Airflow blocks both downstream tasks instead of transforming stale or incomplete data.
 
 ## dbt project used
 
-_Replace: your Week 10 project or the class reference?_
+I used the class reference Week 10 dbt project copied into `include/dbt_project/`. Airflow runs dbt through `uvx --python 3.11` because the Astro runtime uses Python 3.14 and stable dbt-core is not compatible with that interpreter.
+
+## Logical-date parameterization and idempotency
+
+The ingest task reads the current Airflow run context through `_partition_date_from_context()`. Scheduled and backfill runs use `dag_run.logical_date`; a manual run may use an explicitly supplied past logical date or `target_date` configuration. The date selects both the TLC parquet URL and the monthly database partition.
+
+Before appending a month, the task deletes only rows whose `lpep_pickup_datetime` belongs to that `YYYY-MM`. Therefore rerunning the same logical month ends with one copy of that month rather than duplicate rows.
+
+## Retry configuration
+
+The DAG configures two retries with a two-minute delay. Retries are useful for transient network or database failures. They do not solve deterministic failures such as a 403 for an unpublished file or invalid SQL.
 
 ## One debugging case I resolved
 
-_Replace: what failed, how you found the cause in the logs, and the fix._
+A manual run requested `green_tripdata_2026-07.parquet` and failed with HTTP 403. Reading the ingest log showed that a manual Airflow 3 run had no past logical date and fell back to the current date. I fixed the workflow by requiring a past logical date for manual runs and by using explicit historical dates for backfills.
+
+## Backfill and idempotency evidence
+
+Command used for the required seven monthly runs:
+
+```powershell
+astro dev run backfill create --dag-id taxi_pipeline --from-date 2024-01-01 --to-date 2024-07-31 --max-active-runs 1
+```
+
+Command used to repeat completed runs:
+
+```powershell
+astro dev run backfill create --dag-id taxi_pipeline --from-date 2024-01-01 --to-date 2024-07-31 --max-active-runs 1 --reprocess-behavior completed
+```
+
+SQL used to record row counts:
+
+```sql
+SELECT
+    to_char(lpep_pickup_datetime, 'YYYY-MM') AS month,
+    count(*) AS row_count
+FROM airflow_<my_role>.raw_trips
+WHERE lpep_pickup_datetime >= TIMESTAMP '2024-01-01'
+  AND lpep_pickup_datetime < TIMESTAMP '2024-08-01'
+GROUP BY 1
+ORDER BY 1;
+```
+
+### Results to add after running
+
+| Month | First backfill count | Repeated backfill count |
+|---|---:|---:|
+| 2024-01 | 56549 | 56549 |
+| 2024-02 | 53571 | 53571 |
+| 2024-03 | 57447 | 57447 |
+| 2024-04 | 56467 | 56467 |
+| 2024-05 | 60994 | 60994 |
+| 2024-06 | 54735 | 54735 |
+| 2024-07 | 51811 | 51811 |
 
-## Parameterized runs and backfill
+The two columns must be identical before submission.
 
-_Replace: how {{ ds }} / logical date drives the partition; the exact backfill create command you ran (with --max-active-runs 1)._
+## Shared Airflow deployment proof
 
-## Idempotency row counts (before / after re-run)
+- Namespaced DAG ID: halyna_taxi_pipeline
+- Student tag:  student:halyna
+- Merged shared-deploy PR: https://github.com/HackYourAssignment/c55-data-week-12/pull/7
+- Shared UI screenshot: `evidence/local_green_run.png`
 
-_Replace: paste monthly counts before the re-run, then after. They must match._
+If the shared VM was unavailable, state that explicitly here and include the local green-run evidence instead.
 
-## Shared Airflow deploy proof (if VM online)
 
-_Replace: merged c55-shared-airflow PR URL + path to your shared-UI screenshot in this repo._
+<!-- Target tier: also document your {{ ds }} parameter usage and the
+     backfill command(s) you ran, with before/after row counts. -->

RUNBOOK.md

@@ -1,22 +1,44 @@
 # RUNBOOK
 
-<!-- Fill in every section below. Another student should be able to
+<!-- Replace every TODO with real content. Another student should be able to
      operate your DAG from this file alone, without reading your Python. -->
 
 ## How to trigger the DAG manually
 
-_Replace this section._
+1. Start the local stack from the project root with `astro dev start`.
+2. Open the Airflow UI and confirm that the `azure_pg` connection exists.
+3. Open `taxi_pipeline`, unpause it, and click **Trigger**.
+4. In Trigger Options, pass a real past logical date such as `2024-01-01`.
+   If the UI does not expose a logical-date field, use this run configuration:
+   `{"target_date": "2024-01-01"}`.
+5. Open the new run and confirm the order `ingest_taxi_month → dbt_run → dbt_test`.
+
+Do not trigger the DAG for a future or unpublished TLC month. Such a run returns HTTP 403 because the parquet file does not exist yet.
 
 ## How to run a backfill
 
-_Replace this section._
+The DAG uses a monthly schedule, so seven assignment runs require seven months. In PowerShell run this as one line, without Bash backslashes:
+
+```powershell
+astro dev run backfill create --dag-id taxi_pipeline --from-date 2024-01-01 --to-date 2024-07-31 --max-active-runs 1
+```
+
+If the DAG is paused, create the backfill first and then unpause it in the UI. Wait until all runs finish before repeating the range. To repeat completed dates for the idempotency proof, use:
+
+```powershell
+astro dev run backfill create --dag-id taxi_pipeline --from-date 2024-01-01 --to-date 2024-07-31 --max-active-runs 1 --reprocess-behavior completed
+```
 
 ## How to inspect task logs
 
-_Replace this section._
+Open `taxi_pipeline → Runs → the failed run → the red task → Logs`. Read upward from the final generic message until the first concrete exception, such as `HTTPError`, `DatabaseError`, `Compilation Error`, or `Env var required but not provided`. Also check **Rendered Templates** for `dbt_run` and `dbt_test` to confirm the resolved database host, user, database, and schema. Passwords must remain redacted.
 
 ## Top 3 likely failures and first response
 
-1. _Replace: symptom, first check, fix_
-2. _Replace this section._
-3. _Replace this section._
+1. **`ingest_taxi_month` fails with HTTP 403/404.** Check the requested URL and logical date in the log. A future month or mistyped path is deterministic, so retries will not fix it. Trigger/backfill a published month and correct the URL or date range.
+2. **PostgreSQL connection or permission failure.** Confirm `AIRFLOW_STUDENT` in `.env`, then verify `azure_pg` in Admin → Connections. The login and schema name must match, for example role `halyna` writes to `airflow_halyna`. Do not commit credentials.
+3. **`dbt_run` fails.** Read the first dbt `Runtime Error` or `Compilation Error`, not only the final Bash exit code. Confirm `include/dbt_project/profiles.yml` exists, required `PG_*` variables are rendered, and `dbt deps` runs before `dbt run`.
+
+## Safe recovery and escalation
+
+Clear and retry one task only for a transient network or database interruption. Use a backfill after a code or business-logic fix that affects several partitions. If the shared scheduler, shared `azure_pg` connection, or shared VM is broken, do not edit shared settings; report the issue to the teacher.

dags/exampledag.py

@@ -0,0 +1,98 @@
+"""
+## Astronaut ETL example DAG
+
+This DAG queries the list of astronauts currently in space from the
+Open Notify API and prints each astronaut's name and flying craft.
+
+There are two tasks, one to get the data from the API and save the results,
+and another to print the results. Both tasks are written in Python using
+Airflow's TaskFlow API, which allows you to easily turn Python functions into
+Airflow tasks, and automatically infer dependencies and pass data.
+
+The second task uses dynamic task mapping to create a copy of the task for
+each Astronaut in the list retrieved from the API. This list will change
+depending on how many Astronauts are in space, and the DAG will adjust
+accordingly each time it runs.
+
+For more explanation and getting started instructions, see our Write your
+first DAG tutorial: https://www.astronomer.io/docs/learn/get-started-with-airflow
+
+![Picture of the ISS](https://www.esa.int/var/esa/storage/images/esa_multimedia/images/2010/02/space_station_over_earth/10293696-3-eng-GB/Space_Station_over_Earth_card_full.jpg)
+"""
+
+from airflow.sdk import Asset, dag, task
+from pendulum import datetime
+import requests
+
+
+# Define the basic parameters of the DAG, like schedule and start_date
+@dag(
+    start_date=datetime(2025, 4, 22),
+    schedule="@daily",
+    doc_md=__doc__,
+    default_args={"owner": "Astro", "retries": 3},
+    tags=["example"],
+)
+def example_astronauts():
+    # Define tasks
+    @task(
+        # Define an asset outlet for the task. This can be used to schedule downstream DAGs when this task has run.
+        outlets=[Asset("current_astronauts")]
+    )  # Define that this task updates the `current_astronauts` Asset
+    def get_astronauts(**context) -> list[dict]:
+        """
+        This task uses the requests library to retrieve a list of Astronauts
+        currently in space. The results are pushed to XCom with a specific key
+        so they can be used in a downstream pipeline. The task returns a list
+        of Astronauts to be used in the next task.
+        """
+        try:
+            r = requests.get("http://api.open-notify.org/astros.json")
+            r.raise_for_status()
+            number_of_people_in_space = r.json()["number"]
+            list_of_people_in_space = r.json()["people"]
+        except Exception:
+            print("API currently not available, using hardcoded data instead.")
+            number_of_people_in_space = 12
+            list_of_people_in_space = [
+                {"craft": "ISS", "name": "Oleg Kononenko"},
+                {"craft": "ISS", "name": "Nikolai Chub"},
+                {"craft": "ISS", "name": "Tracy Caldwell Dyson"},
+                {"craft": "ISS", "name": "Matthew Dominick"},
+                {"craft": "ISS", "name": "Michael Barratt"},
+                {"craft": "ISS", "name": "Jeanette Epps"},
+                {"craft": "ISS", "name": "Alexander Grebenkin"},
+                {"craft": "ISS", "name": "Butch Wilmore"},
+                {"craft": "ISS", "name": "Sunita Williams"},
+                {"craft": "Tiangong", "name": "Li Guangsu"},
+                {"craft": "Tiangong", "name": "Li Cong"},
+                {"craft": "Tiangong", "name": "Ye Guangfu"},
+            ]
+
+        context["ti"].xcom_push(
+            key="number_of_people_in_space", value=number_of_people_in_space
+        )
+        return list_of_people_in_space
+
+    @task
+    def print_astronaut_craft(greeting: str, person_in_space: dict) -> None:
+        """
+        This task creates a print statement with the name of an
+        Astronaut in space and the craft they are flying on from
+        the API request results of the previous task, along with a
+        greeting which is hard-coded in this example.
+        """
+        craft = person_in_space["craft"]
+        name = person_in_space["name"]
+
+        print(f"{name} is currently in space flying on the {craft}! {greeting}")
+
+    # Use dynamic task mapping to run the print_astronaut_craft task for each
+    # Astronaut in space
+    print_astronaut_craft.partial(greeting="Hello! :)").expand(
+        person_in_space=get_astronauts()  # Define dependencies using TaskFlow API syntax
+    )
+
+
+# Instantiate the DAG
+example_astronauts()