perf(PERF-002): skip discarded parquet reads with footer stats

[DoRmAmMu1997]

Ticket scope (PERF-002)

This PR centralizes Parquet footer date-statistics reading and uses those bounds as an advisory miss optimization in the daily-data loader.

What remains optimized

• backend/parquet_stats.py reads optional timestamp min/max statistics without decompressing data pages.
• get_daily_history() can skip a full read when the footer already proves the cached file cannot cover the requested range; that frame would otherwise be read and immediately discarded before fetching fresh data.
• Missing, corrupt, mistyped, null, or statless footer metadata returns (None, None) and falls back to the established full-read behavior.

Safety behavior after review

• A prefetch never declares a cache fresh from footer statistics alone. It reads and validates the actual frame, so a readable footer cannot hide corrupt data pages.
• When footer bounds appear to cover a requested range, the loader recomputes bounds from the frame it actually read before returning a cache hit. This closes file-replacement/TOCTOU mismatches between the metadata check and data read.
• No cache format, sidecar, public API, or Parquet write behavior changes.

Benchmark interpretation

The original synthetic microbenchmark measured footer-bound extraction at roughly 12x faster than loading a whole multi-year frame solely to inspect two dates. That measurement still explains the discarded-read miss-path optimization, but it is not a blanket warm-prefetch or app-relaunch speedup: correctness requires warm prefetch to read the frame before reporting it fresh.

Verification

• Focused Parquet/daily-loader suite: 50 passed
• Full suite: 1,403 passed, 1 skipped; coverage 87.95%
• compileall, Ruff, mypy, Bandit, pinned pip-audit, pre-commit hooks, and git diff --check: passed
• Security diff review: no findings

The change remains intentionally limited to footer-statistics helpers, loader decision logic, and focused regressions for corrupt data pages and changed files.

[DoRmAmMu1997]

Self-review (posted as COMMENT — the bot runs as the PR author and cannot APPROVE). Verdict: Approve.

This diff changes cache-decision logic, so the review focused on "can any file be treated differently than before, other than faster?":

1. Fail-safe shape verified branch-by-branch — timestamp_bounds returns (None, None) for: missing file, corrupt file, no timestamp column, zero row groups, ANY row group without min/max stats, all-null column, non-date-like statistic. Every caller maps (None, None) to the pre-PERF-002 full-read logic. So the set of files whose handling changed = exactly the set the footer proves fresh/covered — and for those, the old code computed the same two dates from the same data and made the same decision.
2. Fresh-condition equivalence — _covers_full_window uses first ≤ start and last ≥ today, cross-checked against ensure_daily_history: its "fresh" verdict requires NOT first > start (:471) and last ≥ today (:487) → identical. The checked_through sidecar path only matters when last < today, which the fast path never claims.
3. One subtle correctness point in the helper — a single statless row group aborts to (None, None) even if other groups have stats, because that group's rows could extend past every other group's bounds. Locked by test_bounds_span_multiple_row_groups + test_writer_without_statistics_returns_none_pair.
4. Semantics of footer min/max vs _date_bounds — footer statistics ignore nulls; _date_bounds drops NaT via errors="coerce". Same nulls-excluded bounds. Mixed valid+NaT columns therefore agree; all-NaT columns have no stats → fallback (test-locked).
5. The "no pandas I/O" claim is proven, not asserted — two integration tests monkeypatch daily_data_loader.pd.read_parquet to raise AssertionError; the prefetch fresh path and the get_daily_history miss decision both pass under that trap.
6. No former hit becomes a miss — test_get_daily_history_covered_range_without_statistics_is_still_a_hit uses a client that fails the test if Dhan is reached, on a statless (fallback-only) file.
7. health.py left alone, deliberately — rewiring it onto a never-raise helper would silently stop counting corrupt files as unreadable on the health page. Documented in the PR body as the reason the "reuse if trivial" option was declined.
8. Benchmark honesty — synthetic files are written by the same to_parquet(index=False) call production uses (default statistics), sized ~2,600 rows × 100 files; both passes reported; the 12× is the warm steady state the relaunch prefetch actually lives in.

Gates: 1,401 passed + 1 skipped, coverage 88.19% (floor 87); pre-existing loader suite (33 tests) unmodified; ruff/mypy(120)/bandit/pip-audit/pre-commit/compileall clean; constraints/pyproject diff vs main empty. Security review: no findings (local trusted cache files; no new error-message surface; no network/writes).

[DoRmAmMu1997]

Codex review plan before fixes:

• Treat Parquet footer statistics as advisory, not proof that an unreadable cache is fresh.
• Keep full-frame validation in the prefetch freshness path.
• Recompute bounds on the frame actually read before returning a cache hit, closing replacement/TOCTOU gaps.
• Add corrupt-data-page and changed-file regressions.
• Update the PR title/body to describe only the discarded-read optimization that remains.

I’ll implement this with TDD, run the full quality/security/Docker gates, and push the follow-up here.

[DoRmAmMu1997]

Codex review findings for PERF-002. Footer metadata is useful for avoiding a discarded read, but it is advisory rather than proof that the data pages remain readable or unchanged.

[DoRmAmMu1997]

Completion verdict

Complete — no remaining reportable findings. This is a completion comment, not an approval.

Commit 4224ac1 made Parquet footer statistics advisory, retained full-read prefetch validation, rechecked the actual frame bounds, added corrupt/changed-file regressions, and corrected the PR title and body. Verification completed with 1,403 tests passed, 1 skipped, 87.95% coverage, and green hosted CI.

A formal diff-security review found no remaining reportable findings.