Skip to content

feat(discovery): issue → policy → dataset → effect, with T3 sample and randomization inference - #3

Merged
jsshin2022 merged 2 commits into
mainfrom
feat/issue-to-effect
Sep 26, 2026
Merged

jsshin2022 merged 2 commits into
mainfrom
feat/issue-to-effect

Conversation

@jsshin2022

Copy link
Copy Markdown
Member

Stacked on #2 (merge #1 → #2 → this).

What's added

Issue → policy → dataset → effect. One social-media post or news headline in, a policy-effect analysis out.

Step How LLM role
1. Identify the policy Keyword scoring against catalog/policies.yaml (8 policies with dates, datasets, licenses, pitfalls) Optional: pick among catalog candidates only
2. Recommend datasets Curated datasets from the catalog + optional live search of the data.go.kr catalog (metadata only) None
3. Choose the design From the catalog (did_simultaneous / did_staggered / scm / its) with support status None
4. Estimate Pre-registered plan.yaml → existing six-step flow Narrative only (linted)
  • Sample case cases/t3-land-permit-2025: 2025-03-24 land-transaction-permit expansion (Gangnam, Seocho, Songpa, Yongsan) vs. the other 21 Seoul districts, monthly, 2024-01 to 2025-09. The window ends before the 2025-10-20 citywide designation, and 2025-03 is dropped as a transition month.
    • Runs on simulation data until an API key is set; results are labeled as synthetic everywhere.
    • core/adapters/molit.py implements the MOLIT apartment-transaction API but has not been verified against the live API.
  • App page "이슈 → 데이터 → 효과": example inputs, policy card with sources and pitfalls, dataset table, run/replay the flow.

Statistical fix: few treated units

With 4 of 25 districts treated, cluster-robust (CRV1) inference is unreliable. In simulation with no pre-trend, the joint pre-trend Wald test rejected 50–80% of the time at α = 0.10.

  • core/estimators/ri.py: randomization inference (placebo-in-space over control units, 999 draws) for event-study and DiD whenever treated units < 10 on a balanced, simultaneous-adoption panel.
    • p-values for the post-period effect and the pre-trend.
    • CIs by test inversion.
  • Simulation check (40 seeds): pre-trend rejection falls to 10–20%; 95% CI coverage is 95% for the event study and 90% for DiD.
  • New abstention trigger few_treated_clusters; reports and figures state which inference was used.

Verification

  • pytest: 45 passed; ruff check / ruff format --check: clean
  • New tests cover catalog loading, text-to-policy matching, parsing of search results from a fixture, RI coverage, and the app page (AppTest).

Not yet done

  • Real data for T3 (needs DATA_GO_KR_API_KEY); live validation of the MOLIT adapter
  • Staggered-adoption and SCM estimators (catalog marks those policies as "in preparation")
  • Official data.go.kr catalog API in place of the search-page parser

🤖 Generated with Claude Code

https://claude.ai/code/session_01EMnmZPBD5PQJW1UQAARN7q


Generated by Claude Code

jsshin2022 and others added 2 commits September 26, 2026 16:07
- core/discovery: match social text to catalog policies (keyword score,
  optional LLM pick among candidates), recommend curated datasets plus
  optional live data.go.kr search; design comes from the catalog, not the LLM
- catalog/policies.yaml: 8 policies with dates, datasets, licenses, pitfalls
- cases/t3-land-permit-2025: pre-registered plan (2025.3.24 expansion,
  25 Seoul districts x month, 2024-01..2025-09); simulation until an API key
  is set; MOLIT real-transaction adapter (not yet verified live)
- estimators: randomization inference when treated units < 10. With 4 of 25
  treated, CRV1 pre-trend tests rejected a true null 50-80% of the time;
  RI brings 95% CI coverage to ~95% in simulation
- app: "이슈 → 데이터 → 효과" page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EMnmZPBD5PQJW1UQAARN7q
@jsshin2022
jsshin2022 changed the base branch from feat/sample-flow to main September 26, 2026 07:24
@jsshin2022
jsshin2022 merged commit 3ccf0b7 into main Sep 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant