Skip to content

feat: add PROPERTY dbt models + fix parse_mixed_date ordering - #11

Open
benjamin-shi-tql wants to merge 7 commits into
mainfrom
feat/property-dbt-models
Open

feat: add PROPERTY dbt models + fix parse_mixed_date ordering#11
benjamin-shi-tql wants to merge 7 commits into
mainfrom
feat/property-dbt-models

Conversation

@benjamin-shi-tql

Copy link
Copy Markdown
Collaborator

Summary

Models the Yardi RAW.PROPERTY object into staging -> intermediate -> marts, following the same conventions as the existing transaction/unit models, and fixes a latent date-parsing bug in the shared parse_mixed_date macro.

New models

  • models/staging/stg_yardi__properties.sql (+ yml)
  • models/intermediate/int_yardi__property_details.sql (+ yml)
  • models/marts/dim_properties.sql (+ yml)

Data-quality remediations (validated against live Snowflake, n=20)

  1. Leading-zero ZIP loss: SZIPCODE NUMBER -> LPAD(...,5,'0') (Boston 2101 -> '02101').
  2. Magic-number ITYPE -> property_type (Residential/Commercial/Mixed Use).
  3. Mixed-format TEXT dates DACQUIRED/DDISPOSED -> DATE via macro (100% parse coverage).
  4. BACTIVE/BTAXEXEMPT 0/1 -> boolean.
  5. Sensitive owner tax id masked to last 4.
  6. Zero-value owner FK HOWNER -> NULLIF(...,0).
  7. Region conformance (major): raw SREGION disagrees with property state for 14/20 rows (70%) (e.g. New York NY tagged "Southeast"). Conformed region derived from state; raw kept as region_reported + _int_region_mismatch flag.
  8. Unit reconciliation: ITOTALUNITS kept authoritative; actual RAW.UNIT count surfaced as _int_unit_count_mismatch (observability only — UNIT is a partial sample).
  9. _int_active_disposed_conflict reconciling BACTIVE vs disposal date.

Macro bug fix (broader impact)

MM/DD/YYYY was tried before MM/DD/YY; Snowflake TRY_TO_DATE greedily accepts a 2-digit year into the 4-digit slot, so 01/21/08 parsed to year 0008. This silently corrupted 426/2000 TRANS.DTEFFECTIVE and 35/240 UNIT.DTLASTINSPECTION rows to years < 1900. Reordering MM/DD/YY first fixes it (2-digit format returns NULL for genuine 4-digit years). Post-fix: 0 parse failures, 0 sub-1900 years across all four TEXT date columns. This also repairs the existing fct_transactions and dim_units models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant