diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 5601ace..81d2a90 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -77,7 +77,7 @@ testing · extension points. - **[rank-002-handoff.md](rank-002-handoff.md)** — RANK-002 implemented build brief for the `backend/scoring/` scorer (pure components + config + the `run_scan` call + UI sort/components + tests). - **[auth-003-role-model.md](auth-003-role-model.md)** — AUTH-003 role model: hierarchical viewer/analyst/admin, the capability→min-role map, the database-driven `user_roles` store with an `ADMIN_EMAILS` bootstrap floor, resolution precedence, defense-in-depth enforcement, and denial logging/audit. - **[auth-003-handoff.md](auth-003-handoff.md)** — AUTH-003 build brief for the `backend/auth/roles.py` policy + `user_roles` table/migration + repository + `require_capability` enforcement + the admin Roles page + tests. -- **[audit-2026-06.md](audit-2026-06.md)** — June 2026 codebase audit & hardening register (QUAL-001/002/003, REF-001, PERF-001, DOC-001): what was found, fixed, and deferred. +- **[audit-2026-06.md](audit-2026-06.md)** — June–July 2026 codebase audit and hardening register through PR #107: what was found, fixed, rejected, and deferred across both review waves. ## Conventions diff --git a/docs/architecture/audit-2026-06.md b/docs/architecture/audit-2026-06.md index c3192bd..287e44e 100644 --- a/docs/architecture/audit-2026-06.md +++ b/docs/architecture/audit-2026-06.md @@ -198,3 +198,80 @@ removal of one redundant flush, and the stronger regression guard. | Fundamentals eligibility tooltip UX | Users can't tell why "Check Fundamentals" mode differs by symbol; small UI improvement, needs product wording. | | `_parse_company_page(session=None)` + a page with an HTMX peers URL would crash in the unmocked path | Production-unreachable (`fetch_company_data` always builds a real Session); the test seam relies on mocking `_fetch_html`. Documented at the call site; tidy when the scraper next changes. | | Postgres-backed deployment guide with a worked example | docs/operations.md covers the switch; a full worked deploy (host, service file, reverse proxy) is its own doc when a real deployment exists. | + +## July 2026 — whole-app review, two waves (appended 2026-07-11) + +Two independent review passes ran this month and their outputs were reconciled +against main @ `0c87b65` before the second wave started, so nothing below was +built twice. **Wave 1** (another Claude session, PRs +[#88](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/88)–[#97](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/97)) +and **wave 2** (this session, PRs +[#98](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/98)–[#107](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/107), +one ticket per PR) together close most of the June deferral list; the status of +every June item is updated in the table at the end of this section. + +### Wave 1 (PRs #88–#97, other session) + +DOC-002 AGENTS.md IPO map (#88) · TEST-004 golden coverage for the five +untested deterministic screeners (#89) · UI-001 chart-cache +`screener_version` key, SEC-001 role-email shape check, QUAL-005 coverage +floor raised to 87 (#90–#92) · AI-005 shared sync bridge, landed as +REFACTOR-003 with ADR (#93) · TEST-005 screener.in HTML snapshot tests (#94) +· UI-002 fundamentals freshness caption + eligibility captions (#95) · +QUAL-004 mypy on tests/ phase 1 (#96) · REF-002 app.py extraction of +`ui/scan_view.py` + `ui/fundamentals_panel.py` (#97). + +### Wave 2 (PRs #98–#107, this session) + +| PR | Ticket | What landed | +|---|---|---| +| [#98](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/98) | TEST-007 | Render-path tests for the #97 extractions; locks the AUTH-003 export gate, OBS-003 export audit, symbol-stable chart selection after row reordering, secret-safe cached/agent failures, and force-refresh reruns. | +| [#99](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/99) | REF-003 | app.py 1,127→822 lines: status panel + parameter controls → `ui/`, 100% tested; `cache_summary` default no longer bound at import time. | +| [#100](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/100) | AI-006 | One shared verdict-JSON extractor in `backend/ai_runtime.py`; rejects non-finite constants and parser failures before validation/cache signing, with `allow_inf_nan=False` as model-level defense in depth. | +| [#101](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/101) | IPO-006 | One `canonical_sebi_url` behind both SSRF gates; rejects encoded traversal/separators, preserves PDF-path policy across redirects, resolves wrappers from the final URL, and normalizes malformed URL/stream failures into safe domain errors. | +| [#102](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/102) | TEST-006 | IPO pipeline end-to-end scenario (ingest→download→extract→ratios→verdict) with a provenance-digest chain across every stage hand-off. | +| [#103](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/103) | PERF-002 | Parquet footer-statistics cache decisions (see the sidecar supersession note below); benchmark-gated at ~12×. | +| [#104](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/104) | DEPLOY-004 | Password-safe env-file/prompt Postgres workflow, URL-encoding guidance, SQLite→Postgres migration recipe, Alembic encoded-password safety, and the correct `audit_logs` table; `pool_pre_ping` remains server-only. | +| [#105](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/105) | QUAL-006 | pandas-stubs adopted (see the estimate correction below). The wave's one sanctioned constraints/pyproject change. | +| [#106](https://github.com/DoRmAmMu1997/Streamlit-Scanner-App/pull/106) | QUAL-007 | mypy checks the whole test tree by default; 29 modules of pre-typing debt are enumerated in a mechanically enforced shrink-only override. | +| (this PR) | DOC-003 | This register section. | + +### Findings verified FALSE in this review (do not re-flag) + +1. **"`cpr_yearly` computes with floats, violating the Decimal-for-money + rule."** False as a violation: float math is the framework-wide convention + for *indicator* calculations across every screener and `backend/indicators.py`; + the Decimal rule governs money at persistence/contract boundaries. Changing + one screener would create inconsistency, not correctness. +2. **"`cpr_yearly` is untested."** False — four dedicated tests plus a golden + snapshot existed at review time. +3. **"The technical agent lacks a prompt-injection quarantine."** Not a gap: + its structured-candle-only posture (no untrusted-text tool) is the locked + TEST-003 decision above, with regression tests pinning it. + +### Considered and REJECTED in wave 1 (recorded so they are not re-proposed) + +- **Session-state registry abstraction** for the UI — indirection without a + demonstrated bug class. +- **Indicators decorator dedup** — the repetition is shallow; a decorator + would obscure the per-indicator math it wraps. +- **`sectors` iterrows micro-optimization** — not a measured hotspot. +- **DNS-rebinding hardening for the fixed SEBI listing URLs** — the listing + URLs are hardcoded constants, and the byte-fetching surface (the PDF + downloader) already resolves the host and rejects non-public answers. +- **`run_scan` length refactor** — long but linear and heavily commented; + splitting it would scatter one coherent lifecycle. + +### June deferral list — status after July + +| June item | July status | +|---|---| +| Agent SDK boilerplate dedup | **Landed in two steps**: the sync bridge (REFACTOR-003, #93) and the verdict-JSON extractor (AI-006, #100). Options-construction/retry/error-taxonomy blocks stay per-agent **by ADR decision** — that is the recorded end state, not remaining debt. | +| mypy on tests/ | **Landed in two phases**: whitelist (#96), then checked-by-default with a 29-module shrink-only debt list (#106). | +| pandas-stubs | **Landed** (#105). Correction for future estimators: June predicted "hundreds of errors"; with stubs version-matched to the pinned pandas minor (2.3.3), the real count was **22**, all mechanical. | +| Parquet metadata sidecar | **Superseded, honoring the June objection.** PERF-002 (#103) needs **no second cache format**: Parquet footers already carry row-group min/max statistics (health.py had read them since OBS-002). Benchmark-gated per the objection (~12× on the bounds question). Review hardening on the PR: footer bounds are treated as an advisory index — the prefetch still validates the actual frame before a "fresh" verdict, because a valid footer can coexist with corrupt data pages. | +| screener.in HTML snapshots | **Landed** (#94). | +| Fundamentals eligibility tooltip | **Landed** (#95). | +| Postgres worked deployment guide | **Landed** (#104), including the SQLite→Postgres data-migration recipe and pool guidance. | +| Redaction single-pass regex | Still deferred — remains a micro-optimization. | +| `_parse_company_page(session=None)` crash path | Still deferred — production-unreachable; tidy when the scraper next changes. | diff --git a/pyproject.toml b/pyproject.toml index fb76dff..afc0f46 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,18 +38,17 @@ external = ["BLE"] # CI runs 3.11; local development may run newer. Pinning the target version # keeps local runs honest about what CI will accept. python_version = "3.11" -# tests/ adopts mypy module-by-module (QUAL-004): the shared fixtures and the -# two policy/guard tests go first because they encode repo invariants. Extend -# this list as further test modules come clean; do not remove entries. +# QUAL-004 started tests/ adoption module-by-module (conftest + the two +# policy/guard tests); QUAL-007 finished it — the whole test tree is checked, +# so a fixture or fake that drifts from the typed application API fails here +# before it fails at runtime. files = [ "app.py", "backend", "screeners", "ui", "Dependencies", - "tests/conftest.py", - "tests/test_supply_chain_policy.py", - "tests/test_repository_layer_boundary.py", + "tests", ] # First-adoption strictness: check function bodies everywhere (including # untyped ones) without yet *requiring* annotations on every def. Stricter @@ -62,6 +61,49 @@ warn_redundant_casts = true # import/assignment lines also suppress the environment-dependent unused-ignore. warn_unused_ignores = true +[[tool.mypy.overrides]] +# QUAL-007's remaining debt: these test modules still carry pre-typing idioms +# (fake clients passed where the real client class is annotated, records-dict +# rows, unannotated accumulators) — ~385 mechanical errors at adoption time. +# Every OTHER test module is fully checked, and a NEW test file is checked by +# default. Shrink this list as modules come clean; NEVER add to it — a new +# entry means new untyped debt, which is exactly what this gate exists to +# prevent. +# NOTE: tests/ is not a package (no __init__.py), so mypy sees these as +# top-level modules — no "tests." prefix. +module = [ + "test_app_comparison_page", + "test_app_validation_page", + "test_auth_session", + "test_daily_data_loader", + "test_daily_scan_job", + "test_dhan_client", + "test_forward_return_service", + "test_indicators", + "test_ipo_document_downloader", + "test_ipo_models", + "test_ipo_ratio_engine", + "test_ipo_repository", + "test_ipo_scorecard", + "test_notifications_channels", + "test_notifications_report", + "test_notifications_service", + "test_pdf_reader", + "test_real_screeners", + "test_result_contract", + "test_scan_run_integration", + "test_scan_service", + "test_scan_storage_repository", + "test_scanner_base", + "test_scoring_model", + "test_screener_in_client", + "test_screener_registry", + "test_sixty_seven_agent", + "test_sixty_seven_search_client", + "test_technical_analysis_agent", +] +ignore_errors = true + [[tool.mypy.overrides]] # Third-party packages that ship no type stubs (and optional accelerators # that are not installed on CI). pandas graduated OUT of this list in diff --git a/tests/test_admin_roles_service.py b/tests/test_admin_roles_service.py index 899c4d9..923d7cd 100644 --- a/tests/test_admin_roles_service.py +++ b/tests/test_admin_roles_service.py @@ -22,9 +22,11 @@ def _audit_events(file_session_factory) -> list[str]: def _role_changed_rows(file_session_factory) -> list[dict]: with file_session_factory() as session: return [ + # The None filter narrows the Optional ORM column; role_changed + # rows always carry metadata, so it filters nothing at runtime. row.metadata_json for row in get_recent_audit_logs(session) - if row.event == "role_changed" + if row.event == "role_changed" and row.metadata_json is not None ] diff --git a/tests/test_app_audit_page.py b/tests/test_app_audit_page.py index 5f261d7..4d446e8 100644 --- a/tests/test_app_audit_page.py +++ b/tests/test_app_audit_page.py @@ -9,12 +9,14 @@ import datetime as dt from contextlib import contextmanager from types import SimpleNamespace +from typing import cast from sqlalchemy.exc import OperationalError from backend.auth.roles import Role from backend.auth.session import AuthenticatedUser from backend.security import MASK +from backend.storage import AuditLog from ui import audit_page @@ -84,7 +86,9 @@ def test_audit_row_renders_system_for_missing_user(): user_email=None, metadata_json=None, ) - row = audit_page._audit_row(entry) + # The formatter only reads the four attributes the namespace fakes, so the + # cast documents an intentional duck-typed AuditLog stand-in. + row = audit_page._audit_row(cast(AuditLog, entry)) assert row["Event"] == "data_refresh_started" assert row["User"] == "system" assert row["Details"] == "" diff --git a/tests/test_app_health_page.py b/tests/test_app_health_page.py index d75268e..9ca52e2 100644 --- a/tests/test_app_health_page.py +++ b/tests/test_app_health_page.py @@ -4,6 +4,8 @@ import datetime as dt +import pandas as pd + import app from backend.auth.roles import Role from backend.auth.session import AuthenticatedUser @@ -41,7 +43,8 @@ def __init__(self): self.warnings: list[str] = [] self.captions: list[str] = [] self.metrics: list[tuple[str, object]] = [] - self.dataframes: list[object] = [] + # DataFrame-typed so assertions may call .to_dict on captured tables. + self.dataframes: list[pd.DataFrame] = [] def subheader(self, *_args, **_kwargs): pass diff --git a/tests/test_app_history_page.py b/tests/test_app_history_page.py index aa3077b..558e08f 100644 --- a/tests/test_app_history_page.py +++ b/tests/test_app_history_page.py @@ -458,10 +458,14 @@ def fake_session_scope(): lambda *, client: loader if client is None else None, raising=False, ) + def _record_chart_render(**kwargs: object) -> object: + rendered.append(kwargs) + return kwargs["chart_symbol"] + monkeypatch.setattr( history_page, "_render_cached_symbol_chart", - lambda **kwargs: rendered.append(kwargs) or kwargs["chart_symbol"], + _record_chart_render, raising=False, ) diff --git a/tests/test_app_ipo_manual_page.py b/tests/test_app_ipo_manual_page.py index afb5642..913d176 100644 --- a/tests/test_app_ipo_manual_page.py +++ b/tests/test_app_ipo_manual_page.py @@ -12,6 +12,7 @@ import datetime as dt from decimal import Decimal +import pandas as pd import pytest from backend.auth.roles import Role @@ -252,7 +253,7 @@ def NumberColumn(self, *_args, **_kwargs) -> None: def __init__(self) -> None: """Prepare the capture slot and the column-config factory.""" - self.captured_frame: object | None = None + self.captured_frame: pd.DataFrame | None = None self.column_config = self._ColumnConfig() def markdown(self, *_args, **_kwargs) -> None: diff --git a/tests/test_app_orchestration.py b/tests/test_app_orchestration.py index ff64a1c..04e0236 100644 --- a/tests/test_app_orchestration.py +++ b/tests/test_app_orchestration.py @@ -10,6 +10,7 @@ import inspect import os import time +from collections.abc import Callable from datetime import date as real_date from pathlib import Path from types import SimpleNamespace @@ -88,7 +89,7 @@ def __init__(self, _client): def test_capability_flags_are_required_at_every_render_boundary(): """A forgotten call-site argument must fail closed instead of enabling actions.""" - boundaries = [ + boundaries: list[tuple[Callable[..., object], str]] = [ (app._render_sidebar, "can_run"), (app._render_scan_output, "can_export"), (app._render_history_page, "can_export"), @@ -122,10 +123,14 @@ def test_shared_cached_chart_renderer_uses_mapped_security_id(monkeypatch): payload = SimpleNamespace(html="
chart
", height=640) monkeypatch.setattr(chart_cache, "st", fake_st) monkeypatch.setattr(chart_cache, "components", fake_components, raising=False) + def _record_and_return_payload(*args: object) -> SimpleNamespace: + calls.append(args) + return payload + monkeypatch.setattr( chart_cache, "_get_or_build_chart_payload", - lambda *args: calls.append(args) or payload, + _record_and_return_payload, ) selected = SimpleNamespace(key="demo", universe="nifty_500") universe = pd.DataFrame([{"symbol": "TCS", "security_id": "11536"}]) @@ -324,10 +329,12 @@ def record_audit(**kwargs): calls.append("audit") raise StopAfterAudit() + def _record_schema_ready() -> bool: + calls.append("schema") + return True + monkeypatch.setattr(app, "ensure_project_dirs", lambda: calls.append("dirs")) - monkeypatch.setattr( - app, "ensure_database_schema", lambda: calls.append("schema") or True - ) + monkeypatch.setattr(app, "ensure_database_schema", _record_schema_ready) monkeypatch.setattr(app, "record_audit_event", record_audit) with pytest.raises(StopAfterAudit): diff --git a/tests/test_audit_recorder.py b/tests/test_audit_recorder.py index d5db0e4..d9ed7b4 100644 --- a/tests/test_audit_recorder.py +++ b/tests/test_audit_recorder.py @@ -54,8 +54,11 @@ def test_record_audit_event_redacts_metadata_before_storage(file_session_factory with file_session_factory() as session: rows = get_recent_audit_logs(session) - assert rows[0].metadata_json["access_token"] == MASK - assert rows[0].metadata_json["setting"] == "LOG_LEVEL" + # The column is Optional in the ORM; this row just wrote metadata. + metadata = rows[0].metadata_json + assert metadata is not None + assert metadata["access_token"] == MASK + assert metadata["setting"] == "LOG_LEVEL" def test_record_audit_event_records_system_event_without_user(file_session_factory): diff --git a/tests/test_daily_data_loader_footer_stats.py b/tests/test_daily_data_loader_footer_stats.py index f309afc..4fca235 100644 --- a/tests/test_daily_data_loader_footer_stats.py +++ b/tests/test_daily_data_loader_footer_stats.py @@ -16,6 +16,7 @@ from __future__ import annotations from datetime import date, timedelta +from typing import cast import pandas as pd import pyarrow as pa @@ -24,6 +25,7 @@ from backend import daily_data_loader from backend.daily_data_loader import DailyDataLoader, history_start_date +from backend.dhan_client import DhanDataClient TODAY = date(2026, 7, 10) YEARS_BACK = 10 @@ -36,9 +38,9 @@ def _instrument() -> dict: def _covering_frame() -> pd.DataFrame: """Business-day candles spanning the full prefetch window ending today.""" start = history_start_date(YEARS_BACK, TODAY) - timedelta(days=5) - stamps = pd.date_range(start, TODAY, freq="B") + stamps = pd.DatetimeIndex(pd.date_range(start, TODAY, freq="B")) if stamps[-1].date() != TODAY: - stamps = stamps.append(pd.DatetimeIndex([pd.Timestamp(TODAY)])) + stamps = pd.DatetimeIndex(stamps.append(pd.DatetimeIndex([pd.Timestamp(TODAY)]))) return pd.DataFrame( { "timestamp": stamps, @@ -188,7 +190,11 @@ def fetch_daily_candles(self, **_kwargs) -> pd.DataFrame: return fetched.copy(deep=True) client = OneShotClient() - loader = DailyDataLoader(client, cache_dir=tmp_path, request_delay_seconds=0.0) + loader = DailyDataLoader( + cast(DhanDataClient, client), + cache_dir=tmp_path, + request_delay_seconds=0.0, + ) short = fetched.loc[fetched["timestamp"] >= pd.Timestamp(date(2026, 7, 1))] _write_cache(loader, short, statistics=True) # Forbid reads AFTER writing the cache; to_parquet is unaffected. @@ -212,7 +218,11 @@ class ForbiddenClient: def fetch_daily_candles(self, **_kwargs) -> pd.DataFrame: raise AssertionError("a covered cache must not reach Dhan") - loader = DailyDataLoader(ForbiddenClient(), cache_dir=tmp_path, request_delay_seconds=0.0) + loader = DailyDataLoader( + cast(DhanDataClient, ForbiddenClient()), + cache_dir=tmp_path, + request_delay_seconds=0.0, + ) _write_cache(loader, _covering_frame(), statistics=False) frame, from_cache = loader.get_daily_history( @@ -249,7 +259,11 @@ def fetch_daily_candles(self, **_kwargs) -> pd.DataFrame: return fetched.copy(deep=True) client = OneShotClient() - loader = DailyDataLoader(client, cache_dir=tmp_path, request_delay_seconds=0.0) + loader = DailyDataLoader( + cast(DhanDataClient, client), + cache_dir=tmp_path, + request_delay_seconds=0.0, + ) _write_cache(loader, _covering_frame(), statistics=True) monkeypatch.setattr( daily_data_loader, diff --git a/tests/test_fundamental_agent.py b/tests/test_fundamental_agent.py index 38ddde1..d68a80b 100644 --- a/tests/test_fundamental_agent.py +++ b/tests/test_fundamental_agent.py @@ -14,7 +14,7 @@ from datetime import UTC, datetime from pathlib import Path from types import SimpleNamespace -from typing import Any +from typing import Any, cast import pytest @@ -206,7 +206,9 @@ def test_observation_sentiment_must_be_valid(): Observation( topic="Anything", finding="x", - sentiment="bullish", # invalid; only positive/negative/neutral + # Deliberately invalid (only positive/negative/neutral exist); the + # cast lets the type checker allow what the validator must reject. + sentiment=cast(Any, "bullish"), evidence="x", ) diff --git a/tests/test_ipo_contract_policy.py b/tests/test_ipo_contract_policy.py index fc19f55..26234ec 100644 --- a/tests/test_ipo_contract_policy.py +++ b/tests/test_ipo_contract_policy.py @@ -136,10 +136,14 @@ def test_ipo_networking_is_isolated_to_sources_and_all_ipo_code_is_ui_free() -> for path in files: tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) for node in ast.walk(tree): + # Skipping non-import nodes up front also narrows the type so + # ``node.lineno`` below is known to exist (QUAL-007). + if not isinstance(node, (ast.Import, ast.ImportFrom)): + continue modules: list[str] = [] if isinstance(node, ast.Import): modules.extend(alias.name for alias in node.names) - elif isinstance(node, ast.ImportFrom) and node.module: + elif node.module: modules.append(node.module) for module in modules: is_network = any( diff --git a/tests/test_ipo_manual_repository.py b/tests/test_ipo_manual_repository.py index ddac967..644aa4d 100644 --- a/tests/test_ipo_manual_repository.py +++ b/tests/test_ipo_manual_repository.py @@ -13,6 +13,7 @@ import hashlib from decimal import Decimal from pathlib import Path +from typing import Any import pytest @@ -186,7 +187,14 @@ def test_submit_manual_extraction_persists_complete_detached_revision( silently omit one part of the immutable revision. """ issue, document, digest, _path = _cached_document(file_session_factory, tmp_path) - events: list[dict[str, object]] = [] + # dict[str, Any] so nested payload fields (e.g. metadata) stay inspectable + # in the assertions below without per-field casts (QUAL-007). + events: list[dict[str, Any]] = [] + + def _record_audit(**kwargs: Any) -> bool: + """Capture the audit payload and report success like the real sink.""" + events.append(kwargs) + return True created = submit_manual_extraction( issue.id, @@ -194,7 +202,7 @@ def test_submit_manual_extraction_persists_complete_detached_revision( entered_by_email=" ADMIN@Example.com ", data_dir=tmp_path, now=lambda: dt.datetime(2026, 7, 1, 9, tzinfo=dt.UTC), - audit_recorder=lambda **kwargs: events.append(kwargs) or True, + audit_recorder=_record_audit, session_factory=file_session_factory, ) diff --git a/tests/test_ipo_pipeline_e2e.py b/tests/test_ipo_pipeline_e2e.py index 6460065..ee7bf2d 100644 --- a/tests/test_ipo_pipeline_e2e.py +++ b/tests/test_ipo_pipeline_e2e.py @@ -199,7 +199,7 @@ def _score_input(*, source_document_url: str) -> IpoScoreInput: def factor(score: int, reason: str) -> FactorAssessment: """Wrap one factor score with a short evidence-style reason.""" - return FactorAssessment(score=score, reason=reason) + return FactorAssessment(score=Decimal(score), reason=reason) return IpoScoreInput( company_name="Example Limited", diff --git a/tests/test_ipo_sebi_models.py b/tests/test_ipo_sebi_models.py index 8588b01..894944f 100644 --- a/tests/test_ipo_sebi_models.py +++ b/tests/test_ipo_sebi_models.py @@ -4,6 +4,7 @@ import datetime as dt from dataclasses import FrozenInstanceError +from typing import cast import pytest @@ -24,7 +25,9 @@ def test_unknown_issue_type_is_supported_by_the_domain() -> None: """Pin unknown issue type is supported by the domain as an executable IPO regression contract.""" issue = IpoIssueData( company_name="Example Limited", - issue_type="UNKNOWN", + # Deliberately a STRING: the point of this test is that __post_init__ + # coerces raw enum values, so the cast only quiets the type checker. + issue_type=cast(IpoIssueType, "UNKNOWN"), status=IpoStatus.DRHP_FILED, source_confidence=Confidence.HIGH, ) diff --git a/tests/test_ipo_url_canonical.py b/tests/test_ipo_url_canonical.py index 13a4a93..93e534d 100644 --- a/tests/test_ipo_url_canonical.py +++ b/tests/test_ipo_url_canonical.py @@ -9,6 +9,9 @@ from __future__ import annotations +from collections.abc import Callable +from typing import Any + import pytest from backend.ipo.url_canonical import canonical_sebi_url @@ -21,20 +24,29 @@ class _Rejected(Exception): """Caller-supplied error type; the canonicalizer must raise exactly this.""" -def _canonical(value: str, **overrides): +def _canonical( + value: str, + *, + base_url: str = _BASE, + allowed_hosts: frozenset[str] = _ALLOWED, + error: Callable[[], Exception] = _Rejected, + resolver: Callable[..., Any] | None = None, + require_pdf_path: bool = False, +) -> str: """Call the canonicalizer with this suite's defaults, overriding per test. Bundling ``base_url``/``allowed_hosts``/``error`` here keeps each test focused on the one knob it varies, the way the two production wrappers bind their own fixed configuration. """ - kwargs = { - "base_url": _BASE, - "allowed_hosts": _ALLOWED, - "error": _Rejected, - } - kwargs.update(overrides) - return canonical_sebi_url(value, **kwargs) + return canonical_sebi_url( + value, + base_url=base_url, + allowed_hosts=allowed_hosts, + error=error, + resolver=resolver, + require_pdf_path=require_pdf_path, + ) def _resolver_answers(*addresses: str): diff --git a/tests/test_parquet_stats.py b/tests/test_parquet_stats.py index 3b000cc..b025ee3 100644 --- a/tests/test_parquet_stats.py +++ b/tests/test_parquet_stats.py @@ -75,7 +75,10 @@ def test_all_null_timestamps_return_none_pair(tmp_path): """A column of NaT has no min/max statistics — must fall back, not guess.""" path = tmp_path / "DEMO_1.parquet" pd.DataFrame( - {"timestamp": pd.to_datetime([None, None]), "close": [1.0, 2.0]} + { + "timestamp": pd.Series([pd.NaT, pd.NaT], dtype="datetime64[ns]"), + "close": [1.0, 2.0], + } ).to_parquet(path, index=False) assert timestamp_bounds(path) == (None, None) diff --git a/tests/test_scan_ipo_filings_job.py b/tests/test_scan_ipo_filings_job.py index 7bfae28..717bc94 100644 --- a/tests/test_scan_ipo_filings_job.py +++ b/tests/test_scan_ipo_filings_job.py @@ -5,6 +5,7 @@ import datetime as dt import importlib import io +from typing import Any from sqlalchemy import select @@ -113,7 +114,9 @@ def test_failed_category_is_audited_redacted_and_does_not_block_others( job = importlib.import_module("backend.jobs.scan_ipo_filings") output = io.StringIO() persisted: list[SebiFilingCategory] = [] - audits: list[dict[str, object]] = [] + # dict[str, Any] so the nested metadata payload stays inspectable in the + # assertions below without per-field casts (QUAL-007). + audits: list[dict[str, Any]] = [] def fetcher(category, _from_date, _to_date): """Fail only RHP with secret-shaped hostile text to test isolation.""" diff --git a/tests/test_scan_storage_database.py b/tests/test_scan_storage_database.py index d8d4d89..268dba5 100644 --- a/tests/test_scan_storage_database.py +++ b/tests/test_scan_storage_database.py @@ -119,7 +119,8 @@ def test_make_engine_applies_sqlite_concurrency_pragmas(tmp_path: Path): with engine.connect() as connection: assert connection.exec_driver_sql("PRAGMA foreign_keys").scalar() == 1 assert connection.exec_driver_sql("PRAGMA busy_timeout").scalar() == 5000 - assert connection.exec_driver_sql("PRAGMA journal_mode").scalar().lower() == "wal" + # str() narrows scalar()'s Optional return; PRAGMA always answers. + assert str(connection.exec_driver_sql("PRAGMA journal_mode").scalar()).lower() == "wal" finally: engine.dispose() diff --git a/tests/test_scoring_components.py b/tests/test_scoring_components.py index 0444747..38902b0 100644 --- a/tests/test_scoring_components.py +++ b/tests/test_scoring_components.py @@ -9,6 +9,7 @@ from __future__ import annotations import math +from typing import cast import numpy as np import pandas as pd @@ -85,7 +86,11 @@ def test_risk_score_absolute_uses_trailing_log_return_volatility(): "close": [100, 102, 101, 103], } ) - returns = np.log(pd.Series([100, 102, 101, 103]) / pd.Series([100, 102, 101, 103]).shift(1)).dropna() + # Same narrowing as the production component: np.log(Series) is a Series + # at runtime even though numpy's stubs say ndarray (QUAL-007). + returns = cast( + pd.Series, np.log(pd.Series([100, 102, 101, 103]) / pd.Series([100, 102, 101, 103]).shift(1)) + ).dropna() sigma = float(returns.std(ddof=0)) expected = 100.0 * max(0.0, min(1.0, 1.0 - sigma / 0.05)) diff --git a/tests/test_screener_golden_outputs.py b/tests/test_screener_golden_outputs.py index 5a1c430..1335536 100644 --- a/tests/test_screener_golden_outputs.py +++ b/tests/test_screener_golden_outputs.py @@ -69,7 +69,7 @@ class FakeDataLoader: def __init__(self, frames: dict[str, pd.DataFrame]): self.frames = frames - self.last_failures = [] + self.last_failures: list[dict[str, object]] = [] self.last_cache_hits = 0 self.last_cache_misses = len(frames) diff --git a/tests/test_supply_chain_policy.py b/tests/test_supply_chain_policy.py index 49fd849..b5a66f4 100644 --- a/tests/test_supply_chain_policy.py +++ b/tests/test_supply_chain_policy.py @@ -8,12 +8,48 @@ from __future__ import annotations +import copy import re +import tomllib from pathlib import Path +import pytest import yaml ROOT = Path(__file__).resolve().parents[1] +QUAL_007_IGNORE_ERRORS_BASELINE = frozenset( + { + "test_app_comparison_page", + "test_app_validation_page", + "test_auth_session", + "test_daily_data_loader", + "test_daily_scan_job", + "test_dhan_client", + "test_forward_return_service", + "test_indicators", + "test_ipo_document_downloader", + "test_ipo_models", + "test_ipo_ratio_engine", + "test_ipo_repository", + "test_ipo_scorecard", + "test_notifications_channels", + "test_notifications_report", + "test_notifications_service", + "test_pdf_reader", + "test_real_screeners", + "test_result_contract", + "test_scan_run_integration", + "test_scan_service", + "test_scan_storage_repository", + "test_scanner_base", + "test_scoring_model", + "test_screener_in_client", + "test_screener_registry", + "test_sixty_seven_agent", + "test_sixty_seven_search_client", + "test_technical_analysis_agent", + } +) CI_COMMANDS = ( "python -m pre_commit validate-config .pre-commit-config.yaml", "python -m pytest -q --cov=backend --cov=screeners --cov=ui --cov-fail-under=87", @@ -29,6 +65,55 @@ ) +def _assert_qual_007_ignore_errors_only_shrinks(config: dict) -> None: + """Keep QUAL-007's temporary mypy debt list from silently growing. + + Beginner note: the override is a migration aid, not a permanent escape + hatch. Existing entries may be removed as tests gain types, but adding a + new test module would hide fresh errors from CI and must fail this policy + test. + """ + mypy = config["tool"]["mypy"] + assert "tests" in mypy["files"] + + ignored_overrides = [ + override + for override in mypy.get("overrides", []) + if override.get("ignore_errors") is True + ] + assert len(ignored_overrides) == 1 + + modules = ignored_overrides[0]["module"] + assert len(modules) == len(set(modules)), "ignore_errors modules must be unique" + assert set(modules) <= QUAL_007_IGNORE_ERRORS_BASELINE + for module in modules: + assert (ROOT / "tests" / f"{module}.py").is_file(), module + + +def test_qual_007_mypy_ignore_errors_debt_can_only_shrink(): + """The checked-in mypy override must stay within its reviewed baseline.""" + with (ROOT / "pyproject.toml").open("rb") as handle: + config = tomllib.load(handle) + + _assert_qual_007_ignore_errors_only_shrinks(config) + + +def test_qual_007_mypy_ignore_errors_guard_rejects_new_modules(): + """Prove the policy guard fails if a future edit expands the debt list.""" + with (ROOT / "pyproject.toml").open("rb") as handle: + config = tomllib.load(handle) + expanded = copy.deepcopy(config) + ignored_override = next( + override + for override in expanded["tool"]["mypy"]["overrides"] + if override.get("ignore_errors") is True + ) + ignored_override["module"].append("test_new_untyped_debt") + + with pytest.raises(AssertionError): + _assert_qual_007_ignore_errors_only_shrinks(expanded) + + def test_ci_workflow_runs_quality_and_dependency_security_checks(): """CI should run the same checks maintainers run locally.""" workflow = ROOT / ".github" / "workflows" / "quality-and-security.yml" diff --git a/tests/test_user_roles_repository.py b/tests/test_user_roles_repository.py index 9eece98..5ae671b 100644 --- a/tests/test_user_roles_repository.py +++ b/tests/test_user_roles_repository.py @@ -2,6 +2,8 @@ from __future__ import annotations +from typing import Any + import pytest from sqlalchemy.dialects import postgresql from sqlalchemy.exc import IntegrityError @@ -81,7 +83,9 @@ def test_count_user_role_admins(db_session: Session): def test_admin_lock_query_uses_for_update(): """Postgres role mutations must serialize against the same admin rows.""" - statements: list[object] = [] + # Any (not object) so the captured SQLAlchemy statement's .compile() stays + # callable in the assertion below (QUAL-007). + statements: list[Any] = [] class RecordingSession: def scalars(self, statement):