Implement opportunistic local GA fetch from prod (RFC merged) #1026

Activity

1. skypank-coder commented on Aug 21, 2026

   

   skypank-coder

   on Aug 21, 2026

   Contributor

   More actions

   @northdpole I'd like to take this one. I've read the merged RFC (docs/rfc/local-ga-opportunistic-fetch.md)
   end to end and traced it against the current code, and it lines up well with the auth/#963
   and Workstream F work I've been shipping — I'd scope it Phase-1-first, small test-first PRs.

   Execution read for Phase 1:

   • D1 seam: extract the shared fetch/cache path out of map_analysis() into
     application/utils/ga_upstream_fetch.py so route and REST share one path, keeping the
     cache-only guard for Heroku/prod untouched.
   • Reuse over reinvent: the upstream client mirrors the CRE_UPSTREAM_* / fetch_upstream_json
     retry pattern (feat: retry transient failures during upstream sync #951) under the CRE_GA_UPSTREAM_* scope (D12), and persistence reuses
     add_gap_analysis_result gated on should_persist_primary_gap_analysis_cache so a failed or
     non-material fetch can never overwrite a local material row (D3/D10).
   • Failure/concurrency semantics I want to get exactly right: single in-flight fetch in its own
     ga:upstream:inflight: namespace so it can't collide with the existing ga:inflight: lock (D4);
     no retry on 404 and no silent Neo4j/RQ compute fallback in the request path (D2/D5);
     CRE_GA_NO_UPSTREAM as a clean early-out (D9).

   Proposed Phase-1 slicing: (1) ga_upstream_fetch module + CRE_GA_UPSTREAM_* client + unit tests,
   (2) wire into map_analysis on cache miss (local/non-Heroku only),
   (3) the miss UX (ga_cache_miss 404 payload, in-flight response, the soft "file a ticket" CTA). Phases 2–3
   (pre-warm, TTL, partial streaming) as follow-ups. I'll follow D1–D12 as written. Happy to
   adjust — could you assign me?
2. Abhijeet2409 commented on Aug 26, 2026

   

   Abhijeet2409

   on Aug 26, 2026

   Last edited by Abhijeet2409

   Contributor

   More actions

   @northdpole , @Pa04rth I’d like to work on this one as well. Could you assign it to me ?
3. Abhijeet2409 commented on Aug 28, 2026

   

   Abhijeet2409

   on Aug 28, 2026

   Contributor

   More actions

   Thanks for the detailed RFC that helped a lot — here's how I'm planning to approach Phase 1 before taking the implementation issue.

   • Inspect existing patterns first: Look at the existing CRE_UPSTREAM_* / fetch_upstream_json() implementation, taking reference from the retry pattern introduced in Merged feat: retry transient failures during upstream sync #951, along with the project's configuration patterns.
   • Configuration: Add the new CRE_GA_* variables to .env.example, and check how project currently loads/uses env vars so the new backend functions can consume these at runtime.
   • Tests first: Write tests around the Phase 1 contract using mocked upstream responses — success/upsert, 404 (no retry), retryable failures, same-pair concurrency, disabled upstream, cache hit/no re-fetch, and material-row protection by following existing testing patterns in the project where applicable. The RFC already provides a detailed set of expected test cases, which I'll use as the initial checklist.
   • Implement: As stated in the RFC, extract the relevant shared logic from map_analysis() into the new module, then build fetch_ga_upstream_json() as the HTTP client and resolve_gap_analysis_pair() to coordinate the cache check, lock, fetch, validation, and upsert.
   • Verify: Run the Phase 1 tests and manually verify the implementation matches expected behavior — cache hits/misses, upstream fetching, locking, upsert, and failure cases.

   This is scoped to Phase 1 only (fetch/upsert/lock) — no route wiring or Heroku-guard changes. Implementation will follow the RFC's contract strictly, with no reinterpretation of the maintainer-approved design decisions.