refactor(IPO-006): one shared SEBI URL canonicalizer + downloader doc riders

[DoRmAmMu1997]

Ticket scope (IPO-006)

Goal: deduplicate the two drifted _canonical_sebi_url copies guarding every SEBI fetch, without changing either module's public behavior — plus two documentation riders the June review asked for.

The drift being fixed (this is the SSRF gate, so divergence is a security-consistency problem): the downloader's copy (downloader.py:91) had grown a malformed-port guard, an optional DNS-answer check (anti-rebinding), and a require_pdf_path restriction; the scraper's copy (sebi.py:77) had none of these and leaked a bare ValueError on a malformed port.

In scope

• New backend/ipo/url_canonical.py: one canonical_sebi_url(value, *, base_url, allowed_hosts, error, resolver=None, require_pdf_path=False). Every rejection raises the caller's error factory, so the scraper keeps SebiSourceError (fixed message) and the downloader keeps its secret-safe unsafe_url code — the shared module knows neither type. The raiser-helper shape preserves the originals' implicit exception chaining exactly (and satisfies flake8-bugbear B904).
• Both modules keep thin _canonical_sebi_url wrappers with their original signatures — all internal call sites and both existing test suites pass unmodified.
• One declared tightening (the only behavior change): a malformed port (https://host:abc/) in the scraper now raises SebiSourceError instead of leaking a bare ValueError. The URL was rejected either way; it now fails within the module's error taxonomy. Test-locked.
• 16 direct tests in tests/test_ipo_url_canonical.py covering the shared knobs: canonicalization shape (fragment dropped, path defaulted, host casefolded, :443 collapsed), each rejection path via the injected error type, require_pdf_path on/off, and the DNS layer (public passes; private / mixed / empty / failing / malformed answers fail closed; resolver=None provably skips the layer). Written to the IPO subsystem's docstring policy guard.
• Doc riders at the code they describe in downloader.py:
  • the b"%PDF-" check is header-only by design — it rejects HTML error pages served as PDFs; deep validation belongs to the parse stage, and the content-addressed cache re-verifies the SHA-256 on every later read, so corruption is still caught;
  • the os.replace publish sets no explicit file mode by design — these are public SEBI filings, the threat is tampering (covered by digest re-verification), not disclosure; confidentiality of the cache rides on the documented data_dir protection.

Out of scope (deliberate)

• No hardening changes to either caller's configuration: the scraper still runs without the DNS check (its URLs come from fixed official constants and same-host redirects), the downloader's stays always-on. Equalizing them is a security decision for a separate ticket if ever desired.
• The two per-module ALLOWED_HOSTS constants stay (they are part of each module's reviewed surface); the shared function takes the set as a parameter.

This is ticket 4 of a 10-ticket second improvement wave (TEST-007 #98, REF-003 #99, AI-006 #100, IPO-006, TEST-006, PERF-002, DEPLOY-004, QUAL-006, QUAL-007, DOC-003), each shipped as its own PR off main.

Gates (all green, local, Python 3.13)

Gate	Result
pytest -q --cov=backend --cov=screeners --cov=ui --cov-fail-under=87	1,406 passed, 1 skipped; coverage 88.17%
SEBI ingestion + downloader + filings-job suites	pass unmodified
IPO policy guards (docstrings, boundaries)	green
pre_commit validate-config / compileall / ruff / mypy (120 files) / bandit / pip_audit	all clean

constraints.txt / pyproject.toml diff vs main: empty (as required).

Review

/code-review: Approve — wrapper signatures byte-identical; the shared body reproduces the downloader's (stricter) logic with the scraper opting out of downloader-only knobs; exception chaining shape preserved via the raiser helper; dead imports removed (ruff-verified). /security-review: no findings — the SSRF gate's checks are strictly preserved or tightened (the scraper gains the malformed-port fail-closed path), the DNS anti-rebinding layer is unchanged for the downloader and provably skipped-not-nulled for the scraper, and the new tests lock the fail-closed behavior of every rejection path.

🤖 Generated with Claude Code

[DoRmAmMu1997]

Self-review (posted as COMMENT — the bot runs as the PR author and cannot APPROVE). Verdict: Approve.

This diff touches the SSRF gate, so it got the closest read of the wave:

1. Control-by-control comparison — every check in both pre-change copies (HTTPS-only, exact host allowlist, credential rejection, port 443/None, PDF-path restriction, DNS is_global fail-closed, fragment stripping) is preserved unchanged; one control is strictly tightened: a malformed port in the scraper now raises SebiSourceError instead of leaking a bare ValueError (rejected either way — this is the PR's single declared behavior change, test-locked).
2. Check-order fidelity — the only ordering change on the scraper side is the port parse moving ahead of the compound boolean; outcome-identical for every URL class (non-HTTPS/wrong-host still reject on the compound check; the malformed-port-on-allowlisted-host case is the declared rider).
3. No weakening via parameterization — the dangerous knobs (resolver=None, require_pdf_path=False) are bound inside each module's private wrapper; the downloader's always-on DNS layer cannot be disabled from any call site. test_no_resolver_skips_the_dns_check_entirely proves resolver=None skips the layer rather than null-running it (an empty answer set WITH a resolver fails closed — asserted separately).
4. Error taxonomy intact — the shared module raises only the caller's factory; the downloader still emits its secret-safe unsafe_url code, the scraper its fixed message. The _reject helper preserves implicit exception chaining exactly like both originals' raiser helpers (and satisfies B904).
5. Suites unmodified — SEBI ingestion, downloader, and filings-job tests pass untouched; the IPO subsystem's auto-discovering docstring guard and network-boundary guard both pass with the new module (the docstring guard caught my first draft of the test file — fixed by documenting every definition including nested fakes).
6. Micro-note — the scraper's copy called value.strip(); the shared body uses str(value).strip() (the downloader's form). Identical for every str input, which is every call site; a non-str would previously have crashed with AttributeError.

Gates: 1,406 passed + 1 skipped, coverage 88.17% (floor 87); ruff/mypy(120)/bandit/pip-audit/pre-commit/compileall clean; constraints/pyproject diff vs main empty. Security review: no findings, one control tightened.

[DoRmAmMu1997]

Codex review plan before fixes:

• Replace raw PDF-path prefix checks with decoded, normalized segment validation that rejects raw and encoded traversal/separator ambiguity.
• Preserve require_pdf_path=True through every PDF redirect hop.
• Convert malformed URL/wrapper syntax into the downloader’s documented safe error taxonomy.
• Correct the digest explanation so it describes later cache-alteration detection accurately.

I’ll implement these with focused hostile-input regressions, run the complete quality/security/Docker gates, and update this branch.

[DoRmAmMu1997]

Codex review findings for IPO-006. The shared canonicalizer is the right boundary, but it must canonicalize path semantics and propagate the PDF policy across redirects.

[DoRmAmMu1997]

Completion verdict

Complete — no remaining reportable findings. This is a completion comment, not an approval.

Commit 6782cce canonicalized decoded URL segments; rejected encoded traversal and encoded separators; propagated the PDF path across redirects; normalized malformed-URL, wrapper, and stream errors; resolved iframe URLs relative to the final response URL; and corrected the digest comment. Verification completed with 1,432 tests passed, 1 skipped, 87.91% coverage, and green hosted CI.

A formal diff-security review found no remaining reportable findings.