tz-aware vs naive datetimes crash contracts, semantic checks, domain checks and cdc_profile #233

Summary

These functions compare or subtract datetime values without normalising time zones first. When one side is tz-aware (e.g. ISO strings with an offset such as FHIR dateTime) and the other is naive, or offsets are mixed, pandas raises TypeError: Cannot compare tz-naive and tz-aware or ValueError on mixed offsets.

Affected functions

• Semantic consistency checks crash fd.clean on a tz-aware/naive date column pair (src/freshdata/semantic/consistency.py:110)
• Healthcare domain checks crash on FHIR dateTime values with a UTC offset (src/freshdata/domains/healthcare/validator.py:417)
• validate_fields crashes on offset-aware dates when a date FieldSpec has min/max bounds (src/freshdata/fieldcheck.py:433)
• cdc_profile mixes timezones: default now is local time; naive/aware or mixed offsets crash (src/freshdata/cdc.py:217)
• compare_to_baseline raises TypeError when baseline and current timestamps differ in tz (src/freshdata/enterprise/contracts.py:1160-1167)

Environment

freshdata 2.0.0; found at 55a8044 and re-checked on main c87efbd (2026-09-15). Reproduces on Python 3.9.6 / pandas 1.5.3 / numpy 1.26.4 and Python 3.12.14 / pandas 2.3.3 / numpy 2.5.3.

Suggested fix

• Parse with pd.to_datetime(..., utc=True) (or convert both sides to UTC / both to naive-UTC) before comparisons, in one shared helper.
• Add tests with an offset-aware column, a naive column, and mixed offsets.

Details

1. Semantic consistency checks crash fd.clean on a tz-aware/naive date column pair

_check_date_pair_ordering compares end_parsed > start_parsed without normalizing time zones, so a tz-aware start column with a naive end column raises TypeError. _check_future_start_dates compares a tz-aware column with a naive reference Timestamp in the same way. These checks are documented as review warnings only, yet they abort the clean.

Reproduction

import sys, warnings; warnings.simplefilter("ignore")
import pandas as pd
import freshdata as fd
n = 10
df = pd.DataFrame({"start_date": pd.date_range("2024-01-01", periods=n, tz="UTC"),
                   "end_date": pd.date_range("2024-02-01", periods=n)})
fd.clean(df.assign(start_date=df.start_date.dt.tz_localize(None)), semantic_mode="auto", verbose=False)
try:
    fd.clean(df, semantic_mode="auto", verbose=False); err = None
except Exception as e:
    err = f"{type(e).__name__}: {e}"
print("EXPECTED: clean completes")
print("ACTUAL:", err or "ok")

Expected

The clean completes.

Actual

EXPECTED: clean completes
ACTUAL: TypeError: Invalid comparison between dtype=datetime64[ns] and DatetimeArray

Where

• src/freshdata/semantic/consistency.py:110: ordered = (end_parsed > start_parsed) & both
• src/freshdata/semantic/consistency.py:167: future = parsed.notna() & (parsed > reference_ts)

2. Healthcare domain checks crash on FHIR dateTime values with a UTC offset

fd.parse_domain(resource, format="fhir") keeps deceasedDateTime as given (2015-02-14T13:42:00+10:00, a normal FHIR dateTime) and birthDate as a date. to_datetime_safe returns a tz-aware series for the first and a naive series for the second, and _check_deceased_after_birth compares them, so fd.clean(frame, domain="healthcare") raises TypeError. _check_age_range (naive Timestamp.now() minus a tz-aware birth_date) and _check_duration (mixed-offset period_start/period_end) fail in the same way. _check_not_future_finished already passes utc=True.

Reproduction

import sys, warnings; warnings.simplefilter("ignore")
import pandas as pd
import freshdata as fd
res = {"resourceType": "Patient", "id": "1", "birthDate": "1970-01-01", "gender": "male",
       "deceasedDateTime": "2015-02-14T13:42:00+10:00"}
frame = fd.parse_domain(res, format="fhir").frames["patient"]
try:
    fd.clean(frame, domain="healthcare", verbose=False); err = None
except Exception as e:
    err = f"{type(e).__name__}: {e}"
print("EXPECTED: parsed FHIR Patient with an offset dateTime validates")
print("ACTUAL:", err or "ok")

Expected

The parsed Patient validates without error.

Actual

Python 3.12:

EXPECTED: parsed FHIR Patient with an offset dateTime validates
ACTUAL: TypeError: Invalid comparison between dtype=datetime64[ns, UTC+10:00] and DatetimeArray

Python 3.9:

EXPECTED: parsed FHIR Patient with an offset dateTime validates
ACTUAL: TypeError: Invalid comparison between dtype=datetime64[ns, pytz.FixedOffset(600)] and DatetimeArray

Where

• src/freshdata/domains/healthcare/validator.py:417: return df.index[both & (deceased_date < birth_date)].tolist()
• src/freshdata/domains/healthcare/validator.py:423: age_years = (pd.Timestamp.now() - birth_date).dt.days / _DAYS_PER_YEAR
• src/freshdata/domains/healthcare/validator.py:481: return df.index[both & ((end - start).dt.days >= _MAX_ENCOUNTER_DAYS)].tolist()

3. validate_fields crashes on offset-aware dates when a date FieldSpec has min/max bounds

_check_value parses 2024-01-01T10:00:00+05:30 to a tz-aware Timestamp and compares it with the naive _date_bound("1900-01-01"), which raises TypeError. The vectorized prescreen hits the same error, but its try/except swallows it and sends the cell to this slow path, which has no guard.

Reproduction

import sys, warnings; warnings.simplefilter("ignore")
import pandas as pd
import freshdata as fd
df = pd.DataFrame({"ts": ["2024-01-01T10:00:00+05:30", "2023-05-05"]})
try:
    fd.validate_fields(df, {"ts": fd.FieldSpec(semantic_type="date", min_value="1900-01-01")}); err = None
except Exception as e:
    err = f"{type(e).__name__}: {e}"
print("EXPECTED: a FieldValidationReport")
print("ACTUAL:", err or "ok")

Expected

A FieldValidationReport.

Actual

EXPECTED: a FieldValidationReport
ACTUAL: TypeError: Cannot compare tz-naive and tz-aware timestamps

Where

• src/freshdata/fieldcheck.py:433: if lo is not None and ts < lo:
• src/freshdata/fieldcheck.py:338: return None if pd.isna(ts) else ts

4. cdc_profile mixes timezones: default now is local time; naive/aware or mixed offsets crash

cdc_profile never normalises timestamps to a single timezone, which causes three problems:

1. Default now is local time. The docstring says "now defaults to the current UTC time". For tz-naive event times the code uses pd.Timestamp.now(tz=None), which is local wall-clock time, so freshness is off by the host's UTC offset. West of UTC, recent data looks like it comes from the future, and stale is not raised until the data is older than stale_after plus the offset.
2. Naive now or watermark crashes on tz-aware events. Passing a naive now= or watermark= string with tz-aware event times raises TypeError.
3. Mixed UTC offsets crash. Event-time strings with different UTC offsets, for example either side of a DST change, parse to an object column, and cummax raises TypeError.

Reproduction

import os, time
os.environ["TZ"] = "America/New_York"; time.tzset()   # any non-UTC host
import pandas as pd, freshdata as fd

# 1) naive UTC event that is 3h old
ev = pd.Timestamp.now(tz="UTC").tz_localize(None) - pd.Timedelta("3h")
r = fd.cdc_profile(pd.DataFrame({"ts": [ev]}), event_time="ts", stale_after="1h")
print(r.freshness_seconds, [d.kind for d in r.defects])

# 2) naive now / watermark with tz-aware events
aware = pd.DataFrame({"ts": pd.to_datetime(["2024-01-01 00:10", "2024-01-01 00:00"]).tz_localize("UTC")})
fd.cdc_profile(aware, event_time="ts", now="2024-01-02")               # TypeError
fd.cdc_profile(aware, event_time="ts", watermark="2024-01-01 00:05")   # TypeError

# 3) mixed offsets
mixed = pd.DataFrame({"ts": ["2024-03-10T01:30:00-05:00", "2024-03-10T03:30:00-04:00"], "k": ["a", "a"]})
fd.cdc_profile(mixed, event_time="ts", key="k", now=pd.Timestamp("2024-03-11", tz="UTC"))  # TypeError

Expected

1. freshness_seconds is about 10800 and a stale defect is raised.
2. A naive now or watermark is interpreted in the event-time zone (or UTC), or rejected with a clear ValueError.
3. Offsets are normalised and a report is returned.

Actual

1. -3600.0 [] in September on a New York host.
2. TypeError: Cannot subtract tz-naive and tz-aware datetime-like objects. and TypeError: Invalid comparison between dtype=datetime64[ns, UTC] and Timestamp.
3. TypeError: cummax is not supported for object dtype.

Where

• src/freshdata/cdc.py:217: pd.Timestamp.now(tz=max_ts.tz), where tz=None gives local time.
• cdc.py:295 and cdc.py:318: pd.Timestamp(now) and pd.Timestamp(watermark) are never localised or converted.
• cdc.py:298: pd.to_datetime(raw, errors="coerce") without utc=True.

5. compare_to_baseline raises TypeError when baseline and current timestamps differ in tz

If a baseline was built from a tz-aware datetime column and the current frame has the same column as tz-naive (or the other way round), compare_to_baseline crashes instead of returning a drift report. Both dtypes normalise to the datetime family, so no dtype finding short-circuits the range comparison. Upstream systems dropping or adding a timezone is exactly the kind of drift this check exists to report.

Reproduction

import pandas as pd
import freshdata as fd

base = fd.build_baseline(pd.DataFrame({"ts": pd.date_range("2024-01-01", periods=40, tz="UTC")}), name="b")
cur = pd.DataFrame({"ts": pd.date_range("2024-01-01", periods=40)})  # tz dropped upstream
fd.compare_to_baseline(cur, base)

Expected

A DriftReport, ideally with a finding that the timezone changed.

Actual

TypeError: Cannot compare tz-naive and tz-aware timestamps

Where

src/freshdata/enterprise/contracts.py:1160-1167 (_check_datetime_range):

try:
    b_min = pd.Timestamp(base.min_timestamp) if base.min_timestamp else None
    ...
except (ValueError, TypeError):  # pragma: no cover - defensive
    return
if b_min is not None and c_min is not None and c_min < b_min:

The comparison happens outside the try block.