chore(QUAL-006): adopt pandas-stubs across the checked application code#105
Conversation
Removes "pandas.*" from the pyproject mypy ignore list, so every
DataFrame/Series call site in app.py/backend/screeners/ui is now typed by
pandas-stubs (pinned 2.3.3.260113, tracking the pandas==2.3.3 series;
types-pytz pinned as its declared dependency).
June's estimate was "hundreds of errors"; with stubs matched to the pinned
minor version the real count was 22 errors in 12 files, all mechanical:
- dhan_client: infer_epoch_unit now returns the exact Literal["s","ms","us"]
set pd.to_datetime's unit= accepts (annotation only; returns unchanged).
- to_dict("records") sites (scoring/model, scanning/service, ui/common,
technical_agent, ipo_manual_page): keys type as Hashable; narrowed with
cast(list[dict[str, ...]], ...) where columns are known-string.
- .loc[scalar] / Series.name / itertuples fields (indicators, two
screeners, technical_agent): cast() narrowing per house convention.
- resample .agg mapping keyed as dict[Hashable, str] (stubs are invariant
in the mapping key); np.log(Series) result narrowed back from numpy's
ndarray typing; one var annotation (cpr_yearly weekly_closes); pd.notna
boundary helper takes cast(Any, ...).
Zero behavior change: every edit is an annotation or cast; goldens pass
byte-identical (no regeneration), full suite 1,386 passed at main's
baseline count.
This is the wave's sanctioned deliberate constraints.txt/pyproject.toml
change: pandas-stubs + types-pytz pinned, required_names extended in
tests/test_supply_chain_policy.py, requirements-dev.txt updated.
Gates: 1,386 passed, coverage 88.13% (floor 87); mypy CLEAN at full scope
(119 files, pandas typed); pre-commit validate, compileall, ruff, bandit,
pip-audit all clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DoRmAmMu1997
left a comment
There was a problem hiding this comment.
Self-review (posted as COMMENT — the bot runs as the PR author and cannot APPROVE). Verdict: Approve.
The review question for a typing big-bang is "did any cast paper over a real bug?" — each was checked against runtime reality:
cast(list[dict[str, Any]], frame.to_dict("records"))(5 sites) — sound where column names are strings by construction: screener result frames go through the result contract's normalizer (which itself enforces string keys downstream), candle frames come from the loader's fixed boundary columns, and the peer editor's columns are configured string names in the same function. No frame with non-string columns can reach these sites..loc[scalar-int]casts (indicators, 2 sites) — the labels come frompivot_rows.indexcoerced throughint()immediately above; a scalar label on a unique integer index returns a Series. The union type exists because.loccan also take slices/lists — not possible here.Series.namecasts (3 sites) — all three frames arereset_index(drop=True)'d before the rows are taken, so.nameis always the integer positional index. The pre-existingint(...)coercion is retained around every cast (runtime behavior unchanged).infer_epoch_unitLiteral return — the function body returns exactly"s","ms", or"us"; the annotation now says so. This is the one fix that improves an API rather than narrowing at a call site.- itertuples OHLC casts — the loader's
normalize_daily_payloadcoerces open/high/low/close throughpd.to_numericat the boundary, so the numeric assumption is enforced upstream, not hoped for. - No error was suppressed — zero
# type: ignorecomments added; the diff is annotations, casts, one Literal, and comments (73 insertions, 29 deletions, no new statements). - Behavior locks — goldens byte-identical without regeneration; full suite at exactly main's baseline (1,386); the deterministic-screener output contract cannot have drifted.
- Supply-chain hygiene — both new pins are exact, listed in
required_names, present in requirements-dev.txt, and pass pip-audit; the stubs version tracks the pinned pandas 2.3.3 series (not the 3.x stubs line, which would type APIs the pinned runtime doesn't have).
Gates: mypy clean at full scope (119 files, pandas typed); 1,386 passed + 1 skipped, coverage 88.13% (floor 87); ruff/bandit/pip-audit/pre-commit/compileall clean. Security review: no findings (typing-only).
Sequencing note: QUAL-007 (mypy over tests/) waits for this to merge — it needs pandas-stubs to type pandas-using test modules and would conflict on pyproject.toml otherwise. No stacking; it will branch off main after this lands.
|
Codex review plan: I found no actionable standalone issue in QUAL-006. I will retain pandas-stubs and the current casts, rerun the typing and repository gates, and perform the final diff-security review. This PR will be evaluated last in the temporary eight-PR integration tree so its mypy gate covers the preceding changes. |
Flips tests/ typing from an opt-in whitelist (QUAL-004: conftest + two policy tests) to checked-by-default: pyproject mypy `files` now lists "tests", so every NEW test file is type-checked automatically and the ~60 already-clean modules are locked. Full adoption surfaced 409 mechanical errors in 45 modules (fake clients passed where the real client class is annotated, records-dict rows, unannotated accumulators). Per the planned fallback, this PR: - fixes the 16 low-count modules (~24 errors): typed audit-recorder fakes replacing `append(...) or True` lambdas, Optional narrowing on ORM metadata/scalar reads, isinstance-first AST walk in the IPO boundary guard, deliberate-invalid enum/string inputs marked with cast (the validators they exercise are unchanged), DataFrame-typed capture slots in UI fakes, and the same np.log/Series narrowing the production component uses; - enumerates the remaining 29 dirty modules in a shrink-only ignore_errors override (~385 errors of typed-API debt, documented for incremental cleanup) - the comment forbids ever ADDING to the list. The tests/ entries are top-level module names (tests/ has no __init__.py). CI's mypy command is unchanged (config-driven). All 240 tests in the 16 touched modules pass; full suite at baseline. Stacked on QUAL-006 (needs pandas-stubs and edits the same pyproject section). Merge #105 first; GitHub retargets this PR to main when the base branch is deleted. Gates: 1,386 passed, coverage 88.13% (floor 87); mypy clean at 209 files (up from 122); pre-commit validate, compileall, ruff, bandit, pip-audit all clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Appends the "July 2026" section to docs/architecture/audit-2026-06.md so future audits read one durable history instead of re-deriving it: - both waves indexed: #88-#97 (other session) and #98-#107 (this wave, one ticket per PR); - findings verified FALSE this round (cpr_yearly float convention, cpr_yearly test coverage, technical-agent injection posture = locked TEST-003 decision); - wave 1's considered-and-REJECTED list recorded so those ideas are not re-proposed (session-state registry, indicators decorator, sectors iterrows, DNS-rebinding on the fixed listing URLs, run_scan split); - the June deferral table updated item by item: Agent SDK dedup landed in two steps with the per-agent remainder locked by ADR; pandas-stubs estimate corrected (June "hundreds" -> actual 22); the parquet sidecar idea superseded by footer statistics with the June measure-first objection honored and the advisory-index review hardening noted. Stacked on QUAL-007 (#106) -> QUAL-006 (#105); merge those first. Gates: 1,386 passed, coverage 88.13%; ruff clean; mypy clean (209 files). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
QUAL-006 completion review (current-main reconciliation): The original pandas-stubs adoption had no actionable standalone defect. I reconciled this branch with current Fresh verification on the composed head:
Verdict: QUAL-006 remains a runtime-neutral pandas-stubs/type-narrowing change, is current with |
Reconcile the whole-test-tree mypy gate with the newly merged PRs, type the affected fixtures and values without changing runtime behavior, and enforce that QUAL-007's temporary ignore-errors baseline can only shrink. Co-authored-by: Hemant <hemantdhamija@gmail.com> Co-authored-by: Codex <codex@openai.com>
Reconcile the July audit register with the reviewed final PR heads, record the enforced QUAL-007 debt policy, and update the architecture index to describe the durable June-July register through PR #107. Co-authored-by: Hemant <hemantdhamija@gmail.com> Co-authored-by: Codex <codex@openai.com>
…-july docs(DOC-003): record the July 2026 review waves in the audit register [stacked on #106]
chore(QUAL-007): check the whole test tree with mypy [stacked on #105]
Ticket scope (QUAL-006 — pandas-stubs, big-bang per user decision)
Goal: stop treating pandas as
Anyin the checked application code. The June review declined this as "hundreds of errors"; the user chose the big-bang, one-PR route for this wave. With stubs matched to the pinned pandas minor version (pandas-stubs==2.3.3.260113forpandas==2.3.3), the actual surface was 22 errors in 12 files — all mechanical narrowing, fixed here.In scope
"pandas.*"removed from the stub-less override list (dhanhq/pandas_ta/pyarrow/pypdf/talib stay — they genuinely ship no stubs).pandas-stubs==2.3.3.260113,types-pytz==2026.2.0.20260518inconstraints.txt(dev-tools section, same pattern astypes-requests); names added torequirements-dev.txt+ the supply-chain policy lock.cast()(the house narrowing convention, AGENTS.md §5):infer_epoch_unitnow returns the exactLiteral["s","ms","us"]set thatpd.to_datetime(unit=)accepts — annotation only, the returned values were already those literals (2 errors,dhan_client).to_dict("records")keys type asHashable; narrowed todict[str, ...]where columns are known-string:scoring/model(the records list feeding all four component builders — also dissolved two cascade errors),scanning/service,ui/common,technical_agent,ipo_manual_page(9 errors)..loc[scalar]→Series|DataFrame,Series.name→Hashable,itertuplesfields → broad scalar union:cast()at the six sites inindicators,heikin_ashi_supertrend,envelope_knoxville_buy,technical_agent(8 errors)..aggmapping keyeddict[Hashable, str](the stubs' mapping overload is key-invariant),np.log(Series)narrowed back from numpy'sndarraytyping, onevar-annotatedincpr_yearly,pd.notna(cast(Any, ...))in the data-quality boundary helper (3 errors).Proof of zero behavior change
UPDATE_GOLDENregeneration was needed or performed.Literalreturn type, and comments.Out of scope (deliberate)
tests/typing stays at its current QUAL-004 scope — extending mypy to the test tree is QUAL-007, sequenced after this PR merges (it needs these stubs to type pandas-using tests).disallow_untyped_defsescalation; per-module strictness remains a future step.This is ticket 8 of a 10-ticket second improvement wave (TEST-007 #98, REF-003 #99, AI-006 #100, IPO-006 #101, TEST-006 #102, PERF-002 #103, DEPLOY-004 #104, QUAL-006, QUAL-007, DOC-003), each shipped as its own PR off
main.Gates (all green, local, Python 3.13)
mypypytest -q --cov=... --cov-fail-under=87pre_commit validate-config/compileall/ruff/banditpip_audit -r constraints.txt(including the two new pins)Review
/code-review: Approve — every fix was checked against runtime reality before casting (records keys are string column names;.loclabels are scalar ints; OHLC itertuples fields are numeric post-boundary-coercion; theLiteralreturn values were already exactly those strings)./security-review: no findings — typing-only diff; the two new dev-dependencies are exact-pinned and pass pip-audit.🤖 Generated with Claude Code