Skip to content

chore(QUAL-006): adopt pandas-stubs across the checked application code#105

Merged
DoRmAmMu1997 merged 10 commits into
mainfrom
chore/qual-006-pandas-stubs
Jul 13, 2026
Merged

chore(QUAL-006): adopt pandas-stubs across the checked application code#105
DoRmAmMu1997 merged 10 commits into
mainfrom
chore/qual-006-pandas-stubs

Conversation

@DoRmAmMu1997

Copy link
Copy Markdown
Owner

Ticket scope (QUAL-006 — pandas-stubs, big-bang per user decision)

Goal: stop treating pandas as Any in the checked application code. The June review declined this as "hundreds of errors"; the user chose the big-bang, one-PR route for this wave. With stubs matched to the pinned pandas minor version (pandas-stubs==2.3.3.260113 for pandas==2.3.3), the actual surface was 22 errors in 12 files — all mechanical narrowing, fixed here.

⚠️ This PR contains the wave's one sanctioned, deliberate constraints.txt + pyproject.toml change (per AGENTS.md §7's "deliberate, reviewed bump" clause): pandas-stubs and its declared dependency types-pytz are pinned; "pandas.*" leaves the mypy ignore list; required_names in tests/test_supply_chain_policy.py and requirements-dev.txt are co-updated. Every other PR in this wave keeps that diff empty.

In scope

  • pyproject: "pandas.*" removed from the stub-less override list (dhanhq/pandas_ta/pyarrow/pypdf/talib stay — they genuinely ship no stubs).
  • Pins: pandas-stubs==2.3.3.260113, types-pytz==2026.2.0.20260518 in constraints.txt (dev-tools section, same pattern as types-requests); names added to requirements-dev.txt + the supply-chain policy lock.
  • All 22 errors fixed, zero behavior change — every edit is an annotation or a cast() (the house narrowing convention, AGENTS.md §5):
    • infer_epoch_unit now returns the exact Literal["s","ms","us"] set that pd.to_datetime(unit=) accepts — annotation only, the returned values were already those literals (2 errors, dhan_client).
    • to_dict("records") keys type as Hashable; narrowed to dict[str, ...] where columns are known-string: scoring/model (the records list feeding all four component builders — also dissolved two cascade errors), scanning/service, ui/common, technical_agent, ipo_manual_page (9 errors).
    • .loc[scalar]Series|DataFrame, Series.nameHashable, itertuples fields → broad scalar union: cast() at the six sites in indicators, heikin_ashi_supertrend, envelope_knoxville_buy, technical_agent (8 errors).
    • Odds and ends: resample .agg mapping keyed dict[Hashable, str] (the stubs' mapping overload is key-invariant), np.log(Series) narrowed back from numpy's ndarray typing, one var-annotated in cpr_yearly, pd.notna(cast(Any, ...)) in the data-quality boundary helper (3 errors).

Proof of zero behavior change

  • Goldens pass byte-identical — no UPDATE_GOLDEN regeneration was needed or performed.
  • Full suite passes at exactly main's baseline count (1,386 passed, 1 skipped).
  • The diff contains no new statements — only annotations, casts, one Literal return type, and comments.

Out of scope (deliberate)

  • tests/ typing stays at its current QUAL-004 scope — extending mypy to the test tree is QUAL-007, sequenced after this PR merges (it needs these stubs to type pandas-using tests).
  • No disallow_untyped_defs escalation; per-module strictness remains a future step.

This is ticket 8 of a 10-ticket second improvement wave (TEST-007 #98, REF-003 #99, AI-006 #100, IPO-006 #101, TEST-006 #102, PERF-002 #103, DEPLOY-004 #104, QUAL-006, QUAL-007, DOC-003), each shipped as its own PR off main.

Gates (all green, local, Python 3.13)

Gate Result
mypy clean at full scope, pandas typed (119 files)
pytest -q --cov=... --cov-fail-under=87 1,386 passed, 1 skipped; coverage 88.13%
Golden screener outputs 10 passed, byte-identical
pre_commit validate-config / compileall / ruff / bandit clean
pip_audit -r constraints.txt (including the two new pins) no known vulnerabilities

Review

/code-review: Approve — every fix was checked against runtime reality before casting (records keys are string column names; .loc labels are scalar ints; OHLC itertuples fields are numeric post-boundary-coercion; the Literal return values were already exactly those strings). /security-review: no findings — typing-only diff; the two new dev-dependencies are exact-pinned and pass pip-audit.

🤖 Generated with Claude Code

Removes "pandas.*" from the pyproject mypy ignore list, so every
DataFrame/Series call site in app.py/backend/screeners/ui is now typed by
pandas-stubs (pinned 2.3.3.260113, tracking the pandas==2.3.3 series;
types-pytz pinned as its declared dependency).

June's estimate was "hundreds of errors"; with stubs matched to the pinned
minor version the real count was 22 errors in 12 files, all mechanical:

- dhan_client: infer_epoch_unit now returns the exact Literal["s","ms","us"]
  set pd.to_datetime's unit= accepts (annotation only; returns unchanged).
- to_dict("records") sites (scoring/model, scanning/service, ui/common,
  technical_agent, ipo_manual_page): keys type as Hashable; narrowed with
  cast(list[dict[str, ...]], ...) where columns are known-string.
- .loc[scalar] / Series.name / itertuples fields (indicators, two
  screeners, technical_agent): cast() narrowing per house convention.
- resample .agg mapping keyed as dict[Hashable, str] (stubs are invariant
  in the mapping key); np.log(Series) result narrowed back from numpy's
  ndarray typing; one var annotation (cpr_yearly weekly_closes); pd.notna
  boundary helper takes cast(Any, ...).

Zero behavior change: every edit is an annotation or cast; goldens pass
byte-identical (no regeneration), full suite 1,386 passed at main's
baseline count.

This is the wave's sanctioned deliberate constraints.txt/pyproject.toml
change: pandas-stubs + types-pytz pinned, required_names extended in
tests/test_supply_chain_policy.py, requirements-dev.txt updated.

Gates: 1,386 passed, coverage 88.13% (floor 87); mypy CLEAN at full scope
(119 files, pandas typed); pre-commit validate, compileall, ruff, bandit,
pip-audit all clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@DoRmAmMu1997 DoRmAmMu1997 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review (posted as COMMENT — the bot runs as the PR author and cannot APPROVE). Verdict: Approve.

The review question for a typing big-bang is "did any cast paper over a real bug?" — each was checked against runtime reality:

  1. cast(list[dict[str, Any]], frame.to_dict("records")) (5 sites) — sound where column names are strings by construction: screener result frames go through the result contract's normalizer (which itself enforces string keys downstream), candle frames come from the loader's fixed boundary columns, and the peer editor's columns are configured string names in the same function. No frame with non-string columns can reach these sites.
  2. .loc[scalar-int] casts (indicators, 2 sites) — the labels come from pivot_rows.index coerced through int() immediately above; a scalar label on a unique integer index returns a Series. The union type exists because .loc can also take slices/lists — not possible here.
  3. Series.name casts (3 sites) — all three frames are reset_index(drop=True)'d before the rows are taken, so .name is always the integer positional index. The pre-existing int(...) coercion is retained around every cast (runtime behavior unchanged).
  4. infer_epoch_unit Literal return — the function body returns exactly "s", "ms", or "us"; the annotation now says so. This is the one fix that improves an API rather than narrowing at a call site.
  5. itertuples OHLC casts — the loader's normalize_daily_payload coerces open/high/low/close through pd.to_numeric at the boundary, so the numeric assumption is enforced upstream, not hoped for.
  6. No error was suppressed — zero # type: ignore comments added; the diff is annotations, casts, one Literal, and comments (73 insertions, 29 deletions, no new statements).
  7. Behavior locks — goldens byte-identical without regeneration; full suite at exactly main's baseline (1,386); the deterministic-screener output contract cannot have drifted.
  8. Supply-chain hygiene — both new pins are exact, listed in required_names, present in requirements-dev.txt, and pass pip-audit; the stubs version tracks the pinned pandas 2.3.3 series (not the 3.x stubs line, which would type APIs the pinned runtime doesn't have).

Gates: mypy clean at full scope (119 files, pandas typed); 1,386 passed + 1 skipped, coverage 88.13% (floor 87); ruff/bandit/pip-audit/pre-commit/compileall clean. Security review: no findings (typing-only).

Sequencing note: QUAL-007 (mypy over tests/) waits for this to merge — it needs pandas-stubs to type pandas-using test modules and would conflict on pyproject.toml otherwise. No stacking; it will branch off main after this lands.

Copy link
Copy Markdown
Owner Author

Codex review plan:

I found no actionable standalone issue in QUAL-006. I will retain pandas-stubs and the current casts, rerun the typing and repository gates, and perform the final diff-security review. This PR will be evaluated last in the temporary eight-PR integration tree so its mypy gate covers the preceding changes.

Flips tests/ typing from an opt-in whitelist (QUAL-004: conftest + two
policy tests) to checked-by-default: pyproject mypy `files` now lists
"tests", so every NEW test file is type-checked automatically and the ~60
already-clean modules are locked.

Full adoption surfaced 409 mechanical errors in 45 modules (fake clients
passed where the real client class is annotated, records-dict rows,
unannotated accumulators). Per the planned fallback, this PR:

- fixes the 16 low-count modules (~24 errors): typed audit-recorder fakes
  replacing `append(...) or True` lambdas, Optional narrowing on ORM
  metadata/scalar reads, isinstance-first AST walk in the IPO boundary
  guard, deliberate-invalid enum/string inputs marked with cast (the
  validators they exercise are unchanged), DataFrame-typed capture slots
  in UI fakes, and the same np.log/Series narrowing the production
  component uses;
- enumerates the remaining 29 dirty modules in a shrink-only
  ignore_errors override (~385 errors of typed-API debt, documented for
  incremental cleanup) - the comment forbids ever ADDING to the list.

The tests/ entries are top-level module names (tests/ has no __init__.py).
CI's mypy command is unchanged (config-driven). All 240 tests in the 16
touched modules pass; full suite at baseline.

Stacked on QUAL-006 (needs pandas-stubs and edits the same pyproject
section). Merge #105 first; GitHub retargets this PR to main when the base
branch is deleted.

Gates: 1,386 passed, coverage 88.13% (floor 87); mypy clean at 209 files
(up from 122); pre-commit validate, compileall, ruff, bandit, pip-audit
all clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Appends the "July 2026" section to docs/architecture/audit-2026-06.md so
future audits read one durable history instead of re-deriving it:

- both waves indexed: #88-#97 (other session) and #98-#107 (this wave,
  one ticket per PR);
- findings verified FALSE this round (cpr_yearly float convention,
  cpr_yearly test coverage, technical-agent injection posture = locked
  TEST-003 decision);
- wave 1's considered-and-REJECTED list recorded so those ideas are not
  re-proposed (session-state registry, indicators decorator, sectors
  iterrows, DNS-rebinding on the fixed listing URLs, run_scan split);
- the June deferral table updated item by item: Agent SDK dedup landed in
  two steps with the per-agent remainder locked by ADR; pandas-stubs
  estimate corrected (June "hundreds" -> actual 22); the parquet sidecar
  idea superseded by footer statistics with the June measure-first
  objection honored and the advisory-index review hardening noted.

Stacked on QUAL-007 (#106) -> QUAL-006 (#105); merge those first.

Gates: 1,386 passed, coverage 88.13%; ruff clean; mypy clean (209 files).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@DoRmAmMu1997

Copy link
Copy Markdown
Owner Author

QUAL-006 completion review (current-main reconciliation):

The original pandas-stubs adoption had no actionable standalone defect. I reconciled this branch with current main in merge commit 8778b66, so the pandas typing gate is now evaluated against the final #98-#104 runtime and test changes instead of the older stack base. No QUAL-006 runtime behavior or public API was changed.

Fresh verification on the composed head:

  • python -m pytest -q --cov=backend --cov=screeners --cov=ui --cov-fail-under=87 -> 1,515 passed, 1 skipped, 89.31% coverage
  • python -m mypy -> no issues in 123 source files
  • compileall, Ruff, Bandit, pinned pip-audit, pre-commit validation/all hooks, and git diff --check -> passed
  • Codex Security diff scan against current origin/main -> 13/13 files reviewed, 0 candidates, 0 findings, 0 deferred
  • GitHub Actions -> Python 3.11, Python 3.12, and Docker image build all passed

Verdict: QUAL-006 remains a runtime-neutral pandas-stubs/type-narrowing change, is current with main, and is clean and mergeable. Per the batch plan, it should remain the final merge candidate after the other reviewed PRs; no merge was performed here.

DoRmAmMu1997 and others added 5 commits July 13, 2026 08:37
Reconcile the whole-test-tree mypy gate with the newly merged PRs, type the
affected fixtures and values without changing runtime behavior, and enforce
that QUAL-007's temporary ignore-errors baseline can only shrink.

Co-authored-by: Hemant <hemantdhamija@gmail.com>
Co-authored-by: Codex <codex@openai.com>
Reconcile the July audit register with the reviewed final PR heads, record the
enforced QUAL-007 debt policy, and update the architecture index to describe
the durable June-July register through PR #107.

Co-authored-by: Hemant <hemantdhamija@gmail.com>
Co-authored-by: Codex <codex@openai.com>
…-july

docs(DOC-003): record the July 2026 review waves in the audit register [stacked on #106]
chore(QUAL-007): check the whole test tree with mypy [stacked on #105]
@DoRmAmMu1997 DoRmAmMu1997 merged commit de8f199 into main Jul 13, 2026
3 checks passed
@DoRmAmMu1997 DoRmAmMu1997 deleted the chore/qual-006-pandas-stubs branch July 13, 2026 04:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant