chore(QUAL-007): check the whole test tree with mypy [stacked on #105]

docs/architecture/README.md

@@ -77,7 +77,7 @@ testing · extension points.
 - **[rank-002-handoff.md](rank-002-handoff.md)** — RANK-002 implemented build brief for the `backend/scoring/` scorer (pure components + config + the `run_scan` call + UI sort/components + tests).
 - **[auth-003-role-model.md](auth-003-role-model.md)** — AUTH-003 role model: hierarchical viewer/analyst/admin, the capability→min-role map, the database-driven `user_roles` store with an `ADMIN_EMAILS` bootstrap floor, resolution precedence, defense-in-depth enforcement, and denial logging/audit.
 - **[auth-003-handoff.md](auth-003-handoff.md)** — AUTH-003 build brief for the `backend/auth/roles.py` policy + `user_roles` table/migration + repository + `require_capability` enforcement + the admin Roles page + tests.
-- **[audit-2026-06.md](audit-2026-06.md)** — June 2026 codebase audit & hardening register (QUAL-001/002/003, REF-001, PERF-001, DOC-001): what was found, fixed, and deferred.
+- **[audit-2026-06.md](audit-2026-06.md)** — June–July 2026 codebase audit and hardening register through PR #107: what was found, fixed, rejected, and deferred across both review waves.
 
 ## Conventions
 

docs/architecture/audit-2026-06.md

@@ -198,3 +198,80 @@ removal of one redundant flush, and the stronger regression guard.
 | Fundamentals eligibility tooltip UX | Users can't tell why "Check Fundamentals" mode differs by symbol; small UI improvement, needs product wording. |
 | `_parse_company_page(session=None)` + a page with an HTMX peers URL would crash in the unmocked path | Production-unreachable (`fetch_company_data` always builds a real Session); the test seam relies on mocking `_fetch_html`. Documented at the call site; tidy when the scraper next changes. |
 | Postgres-backed deployment guide with a worked example | docs/operations.md covers the switch; a full worked deploy (host, service file, reverse proxy) is its own doc when a real deployment exists. |
+
+## July 2026 — whole-app review, two waves (appended 2026-07-11)
+
+Two independent review passes ran this month and their outputs were reconciled
+against main @ `0c87b65` before the second wave started, so nothing below was
+built twice. **Wave 1** (another Claude session, PRs
+[#88](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/88)–[#97](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/97))
+and **wave 2** (this session, PRs
+[#98](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/98)–[#107](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/107),
+one ticket per PR) together close most of the June deferral list; the status of
+every June item is updated in the table at the end of this section.
+
+### Wave 1 (PRs #88–#97, other session)
+
+DOC-002 AGENTS.md IPO map (#88) · TEST-004 golden coverage for the five
+untested deterministic screeners (#89) · UI-001 chart-cache
+`screener_version` key, SEC-001 role-email shape check, QUAL-005 coverage
+floor raised to 87 (#90–#92) · AI-005 shared sync bridge, landed as
+REFACTOR-003 with ADR (#93) · TEST-005 screener.in HTML snapshot tests (#94)
+· UI-002 fundamentals freshness caption + eligibility captions (#95) ·
+QUAL-004 mypy on tests/ phase 1 (#96) · REF-002 app.py extraction of
+`ui/scan_view.py` + `ui/fundamentals_panel.py` (#97).
+
+### Wave 2 (PRs #98–#107, this session)
+
+| PR | Ticket | What landed |
+|---|---|---|
+| [#98](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/98) | TEST-007 | Render-path tests for the #97 extractions; locks the AUTH-003 export gate, OBS-003 export audit, symbol-stable chart selection after row reordering, secret-safe cached/agent failures, and force-refresh reruns. |
+| [#99](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/99) | REF-003 | app.py 1,127→822 lines: status panel + parameter controls → `ui/`, 100% tested; `cache_summary` default no longer bound at import time. |
+| [#100](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/100) | AI-006 | One shared verdict-JSON extractor in `backend/ai_runtime.py`; rejects non-finite constants and parser failures before validation/cache signing, with `allow_inf_nan=False` as model-level defense in depth. |
+| [#101](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/101) | IPO-006 | One `canonical_sebi_url` behind both SSRF gates; rejects encoded traversal/separators, preserves PDF-path policy across redirects, resolves wrappers from the final URL, and normalizes malformed URL/stream failures into safe domain errors. |
+| [#102](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/102) | TEST-006 | IPO pipeline end-to-end scenario (ingest→download→extract→ratios→verdict) with a provenance-digest chain across every stage hand-off. |
+| [#103](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/103) | PERF-002 | Parquet footer-statistics cache decisions (see the sidecar supersession note below); benchmark-gated at ~12×. |
+| [#104](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/104) | DEPLOY-004 | Password-safe env-file/prompt Postgres workflow, URL-encoding guidance, SQLite→Postgres migration recipe, Alembic encoded-password safety, and the correct `audit_logs` table; `pool_pre_ping` remains server-only. |
+| [#105](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/105) | QUAL-006 | pandas-stubs adopted (see the estimate correction below). The wave's one sanctioned constraints/pyproject change. |
+| [#106](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/106) | QUAL-007 | mypy checks the whole test tree by default; 29 modules of pre-typing debt are enumerated in a mechanically enforced shrink-only override. |
+| (this PR) | DOC-003 | This register section. |
+
+### Findings verified FALSE in this review (do not re-flag)
+
+1. **"`cpr_yearly` computes with floats, violating the Decimal-for-money
+   rule."** False as a violation: float math is the framework-wide convention
+   for *indicator* calculations across every screener and `backend/indicators.py`;
+   the Decimal rule governs money at persistence/contract boundaries. Changing
+   one screener would create inconsistency, not correctness.
+2. **"`cpr_yearly` is untested."** False — four dedicated tests plus a golden
+   snapshot existed at review time.
+3. **"The technical agent lacks a prompt-injection quarantine."** Not a gap:
+   its structured-candle-only posture (no untrusted-text tool) is the locked
+   TEST-003 decision above, with regression tests pinning it.
+
+### Considered and REJECTED in wave 1 (recorded so they are not re-proposed)
+
+- **Session-state registry abstraction** for the UI — indirection without a
+  demonstrated bug class.
+- **Indicators decorator dedup** — the repetition is shallow; a decorator
+  would obscure the per-indicator math it wraps.
+- **`sectors` iterrows micro-optimization** — not a measured hotspot.
+- **DNS-rebinding hardening for the fixed SEBI listing URLs** — the listing
+  URLs are hardcoded constants, and the byte-fetching surface (the PDF
+  downloader) already resolves the host and rejects non-public answers.
+- **`run_scan` length refactor** — long but linear and heavily commented;
+  splitting it would scatter one coherent lifecycle.
+
+### June deferral list — status after July
+
+| June item | July status |
+|---|---|
+| Agent SDK boilerplate dedup | **Landed in two steps**: the sync bridge (REFACTOR-003, #93) and the verdict-JSON extractor (AI-006, #100). Options-construction/retry/error-taxonomy blocks stay per-agent **by ADR decision** — that is the recorded end state, not remaining debt. |
+| mypy on tests/ | **Landed in two phases**: whitelist (#96), then checked-by-default with a 29-module shrink-only debt list (#106). |
+| pandas-stubs | **Landed** (#105). Correction for future estimators: June predicted "hundreds of errors"; with stubs version-matched to the pinned pandas minor (2.3.3), the real count was **22**, all mechanical. |
+| Parquet metadata sidecar | **Superseded, honoring the June objection.** PERF-002 (#103) needs **no second cache format**: Parquet footers already carry row-group min/max statistics (health.py had read them since OBS-002). Benchmark-gated per the objection (~12× on the bounds question). Review hardening on the PR: footer bounds are treated as an advisory index — the prefetch still validates the actual frame before a "fresh" verdict, because a valid footer can coexist with corrupt data pages. |
+| screener.in HTML snapshots | **Landed** (#94). |
+| Fundamentals eligibility tooltip | **Landed** (#95). |
+| Postgres worked deployment guide | **Landed** (#104), including the SQLite→Postgres data-migration recipe and pool guidance. |
+| Redaction single-pass regex | Still deferred — remains a micro-optimization. |
+| `_parse_company_page(session=None)` crash path | Still deferred — production-unreachable; tidy when the scraper next changes. |

pyproject.toml

@@ -38,18 +38,17 @@ external = ["BLE"]
 # CI runs 3.11; local development may run newer. Pinning the target version
 # keeps local runs honest about what CI will accept.
 python_version = "3.11"
-# tests/ adopts mypy module-by-module (QUAL-004): the shared fixtures and the
-# two policy/guard tests go first because they encode repo invariants. Extend
-# this list as further test modules come clean; do not remove entries.
+# QUAL-004 started tests/ adoption module-by-module (conftest + the two
+# policy/guard tests); QUAL-007 finished it — the whole test tree is checked,
+# so a fixture or fake that drifts from the typed application API fails here
+# before it fails at runtime.
 files = [
     "app.py",
     "backend",
     "screeners",
     "ui",
     "Dependencies",
-    "tests/conftest.py",
-    "tests/test_supply_chain_policy.py",
-    "tests/test_repository_layer_boundary.py",
+    "tests",
 ]
 # First-adoption strictness: check function bodies everywhere (including
 # untyped ones) without yet *requiring* annotations on every def. Stricter
@@ -62,6 +61,49 @@ warn_redundant_casts = true
 # import/assignment lines also suppress the environment-dependent unused-ignore.
 warn_unused_ignores = true
 
+[[tool.mypy.overrides]]
+# QUAL-007's remaining debt: these test modules still carry pre-typing idioms
+# (fake clients passed where the real client class is annotated, records-dict
+# rows, unannotated accumulators) — ~385 mechanical errors at adoption time.
+# Every OTHER test module is fully checked, and a NEW test file is checked by
+# default. Shrink this list as modules come clean; NEVER add to it — a new
+# entry means new untyped debt, which is exactly what this gate exists to
+# prevent.
+# NOTE: tests/ is not a package (no __init__.py), so mypy sees these as
+# top-level modules — no "tests." prefix.
+module = [
+    "test_app_comparison_page",
+    "test_app_validation_page",
+    "test_auth_session",
+    "test_daily_data_loader",
+    "test_daily_scan_job",
+    "test_dhan_client",
+    "test_forward_return_service",
+    "test_indicators",
+    "test_ipo_document_downloader",
+    "test_ipo_models",
+    "test_ipo_ratio_engine",
+    "test_ipo_repository",
+    "test_ipo_scorecard",
+    "test_notifications_channels",
+    "test_notifications_report",
+    "test_notifications_service",
+    "test_pdf_reader",
+    "test_real_screeners",
+    "test_result_contract",
+    "test_scan_run_integration",
+    "test_scan_service",
+    "test_scan_storage_repository",
+    "test_scanner_base",
+    "test_scoring_model",
+    "test_screener_in_client",
+    "test_screener_registry",
+    "test_sixty_seven_agent",
+    "test_sixty_seven_search_client",
+    "test_technical_analysis_agent",
+]
+ignore_errors = true
+
 [[tool.mypy.overrides]]
 # Third-party packages that ship no type stubs (and optional accelerators
 # that are not installed on CI). pandas graduated OUT of this list in

tests/test_admin_roles_service.py

@@ -22,9 +22,11 @@ def _audit_events(file_session_factory) -> list[str]:
 def _role_changed_rows(file_session_factory) -> list[dict]:
     with file_session_factory() as session:
         return [
+            # The None filter narrows the Optional ORM column; role_changed
+            # rows always carry metadata, so it filters nothing at runtime.
             row.metadata_json
             for row in get_recent_audit_logs(session)
-            if row.event == "role_changed"
+            if row.event == "role_changed" and row.metadata_json is not None
         ]
 
 

tests/test_app_audit_page.py

@@ -9,12 +9,14 @@
 import datetime as dt
 from contextlib import contextmanager
 from types import SimpleNamespace
+from typing import cast
 
 from sqlalchemy.exc import OperationalError
 
 from backend.auth.roles import Role
 from backend.auth.session import AuthenticatedUser
 from backend.security import MASK
+from backend.storage import AuditLog
 from ui import audit_page
 
 
@@ -84,7 +86,9 @@ def test_audit_row_renders_system_for_missing_user():
         user_email=None,
         metadata_json=None,
     )
-    row = audit_page._audit_row(entry)
+    # The formatter only reads the four attributes the namespace fakes, so the
+    # cast documents an intentional duck-typed AuditLog stand-in.
+    row = audit_page._audit_row(cast(AuditLog, entry))
     assert row["Event"] == "data_refresh_started"
     assert row["User"] == "system"
     assert row["Details"] == ""

tests/test_app_health_page.py

@@ -4,6 +4,8 @@
 
 import datetime as dt
 
+import pandas as pd
+
 import app
 from backend.auth.roles import Role
 from backend.auth.session import AuthenticatedUser
@@ -41,7 +43,8 @@ def __init__(self):
         self.warnings: list[str] = []
         self.captions: list[str] = []
         self.metrics: list[tuple[str, object]] = []
-        self.dataframes: list[object] = []
+        # DataFrame-typed so assertions may call .to_dict on captured tables.
+        self.dataframes: list[pd.DataFrame] = []
 
     def subheader(self, *_args, **_kwargs):
         pass

tests/test_app_history_page.py

@@ -458,10 +458,14 @@ def fake_session_scope():
         lambda *, client: loader if client is None else None,
         raising=False,
     )
+    def _record_chart_render(**kwargs: object) -> object:
+        rendered.append(kwargs)
+        return kwargs["chart_symbol"]
+
     monkeypatch.setattr(
         history_page,
         "_render_cached_symbol_chart",
-        lambda **kwargs: rendered.append(kwargs) or kwargs["chart_symbol"],
+        _record_chart_render,
         raising=False,
     )
 

tests/test_app_ipo_manual_page.py

@@ -12,6 +12,7 @@
 import datetime as dt
 from decimal import Decimal
 
+import pandas as pd
 import pytest
 
 from backend.auth.roles import Role
@@ -252,7 +253,7 @@ def NumberColumn(self, *_args, **_kwargs) -> None:
 
     def __init__(self) -> None:
         """Prepare the capture slot and the column-config factory."""
-        self.captured_frame: object | None = None
+        self.captured_frame: pd.DataFrame | None = None
         self.column_config = self._ColumnConfig()
 
     def markdown(self, *_args, **_kwargs) -> None:

tests/test_app_orchestration.py

@@ -10,6 +10,7 @@
 import inspect
 import os
 import time
+from collections.abc import Callable
 from datetime import date as real_date
 from pathlib import Path
 from types import SimpleNamespace
@@ -88,7 +89,7 @@ def __init__(self, _client):
 
 def test_capability_flags_are_required_at_every_render_boundary():
     """A forgotten call-site argument must fail closed instead of enabling actions."""
-    boundaries = [
+    boundaries: list[tuple[Callable[..., object], str]] = [
         (app._render_sidebar, "can_run"),
         (app._render_scan_output, "can_export"),
         (app._render_history_page, "can_export"),
@@ -122,10 +123,14 @@ def test_shared_cached_chart_renderer_uses_mapped_security_id(monkeypatch):
     payload = SimpleNamespace(html="<div>chart</div>", height=640)
     monkeypatch.setattr(chart_cache, "st", fake_st)
     monkeypatch.setattr(chart_cache, "components", fake_components, raising=False)
+    def _record_and_return_payload(*args: object) -> SimpleNamespace:
+        calls.append(args)
+        return payload
+
     monkeypatch.setattr(
         chart_cache,
         "_get_or_build_chart_payload",
-        lambda *args: calls.append(args) or payload,
+        _record_and_return_payload,
     )
     selected = SimpleNamespace(key="demo", universe="nifty_500")
     universe = pd.DataFrame([{"symbol": "TCS", "security_id": "11536"}])
@@ -324,10 +329,12 @@ def record_audit(**kwargs):
         calls.append("audit")
         raise StopAfterAudit()
 
+    def _record_schema_ready() -> bool:
+        calls.append("schema")
+        return True
+
     monkeypatch.setattr(app, "ensure_project_dirs", lambda: calls.append("dirs"))
-    monkeypatch.setattr(
-        app, "ensure_database_schema", lambda: calls.append("schema") or True
-    )
+    monkeypatch.setattr(app, "ensure_database_schema", _record_schema_ready)
     monkeypatch.setattr(app, "record_audit_event", record_audit)
 
     with pytest.raises(StopAfterAudit):

tests/test_audit_recorder.py

@@ -54,8 +54,11 @@ def test_record_audit_event_redacts_metadata_before_storage(file_session_factory
 
     with file_session_factory() as session:
         rows = get_recent_audit_logs(session)
-        assert rows[0].metadata_json["access_token"] == MASK
-        assert rows[0].metadata_json["setting"] == "LOG_LEVEL"
+        # The column is Optional in the ORM; this row just wrote metadata.
+        metadata = rows[0].metadata_json
+        assert metadata is not None
+        assert metadata["access_token"] == MASK
+        assert metadata["setting"] == "LOG_LEVEL"
 
 
 def test_record_audit_event_records_system_event_without_user(file_session_factory):

tests/test_daily_data_loader_footer_stats.py

@@ -16,6 +16,7 @@
 from __future__ import annotations
 
 from datetime import date, timedelta
+from typing import cast
 
 import pandas as pd
 import pyarrow as pa
@@ -24,6 +25,7 @@
 
 from backend import daily_data_loader
 from backend.daily_data_loader import DailyDataLoader, history_start_date
+from backend.dhan_client import DhanDataClient
 
 TODAY = date(2026, 7, 10)
 YEARS_BACK = 10
@@ -36,9 +38,9 @@ def _instrument() -> dict:
 def _covering_frame() -> pd.DataFrame:
     """Business-day candles spanning the full prefetch window ending today."""
     start = history_start_date(YEARS_BACK, TODAY) - timedelta(days=5)
-    stamps = pd.date_range(start, TODAY, freq="B")
+    stamps = pd.DatetimeIndex(pd.date_range(start, TODAY, freq="B"))
     if stamps[-1].date() != TODAY:
-        stamps = stamps.append(pd.DatetimeIndex([pd.Timestamp(TODAY)]))
+        stamps = pd.DatetimeIndex(stamps.append(pd.DatetimeIndex([pd.Timestamp(TODAY)])))
     return pd.DataFrame(
         {
             "timestamp": stamps,
@@ -188,7 +190,11 @@ def fetch_daily_candles(self, **_kwargs) -> pd.DataFrame:
             return fetched.copy(deep=True)
 
     client = OneShotClient()
-    loader = DailyDataLoader(client, cache_dir=tmp_path, request_delay_seconds=0.0)
+    loader = DailyDataLoader(
+        cast(DhanDataClient, client),
+        cache_dir=tmp_path,
+        request_delay_seconds=0.0,
+    )
     short = fetched.loc[fetched["timestamp"] >= pd.Timestamp(date(2026, 7, 1))]
     _write_cache(loader, short, statistics=True)
     # Forbid reads AFTER writing the cache; to_parquet is unaffected.
@@ -212,7 +218,11 @@ class ForbiddenClient:
         def fetch_daily_candles(self, **_kwargs) -> pd.DataFrame:
             raise AssertionError("a covered cache must not reach Dhan")
 
-    loader = DailyDataLoader(ForbiddenClient(), cache_dir=tmp_path, request_delay_seconds=0.0)
+    loader = DailyDataLoader(
+        cast(DhanDataClient, ForbiddenClient()),
+        cache_dir=tmp_path,
+        request_delay_seconds=0.0,
+    )
     _write_cache(loader, _covering_frame(), statistics=False)
 
     frame, from_cache = loader.get_daily_history(
@@ -249,7 +259,11 @@ def fetch_daily_candles(self, **_kwargs) -> pd.DataFrame:
             return fetched.copy(deep=True)
 
     client = OneShotClient()
-    loader = DailyDataLoader(client, cache_dir=tmp_path, request_delay_seconds=0.0)
+    loader = DailyDataLoader(
+        cast(DhanDataClient, client),
+        cache_dir=tmp_path,
+        request_delay_seconds=0.0,
+    )
     _write_cache(loader, _covering_frame(), statistics=True)
     monkeypatch.setattr(
         daily_data_loader,