You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is external data being scraped but not being populated in the database.
Refer to #5, #7, #8, #9
Goal
Every ingestion source persists into its Postgres table through a shared repository layer, so downstream model training
and the API read from one source of truth.
Scope
Build out foresight/repository/ with one repository per table (weather, events, competitor_rates,
flight_arrivals, and hotel tables).
Define a single persistence contract: adapter produces normalized records → repository upserts
them.
Add idempotency/upsert support
Wire external adapters to persist through the repository as the reference
implementations.
Consider database best practices (cascading, indexing, cardinality, primary and foreign keys)
Acceptance criteria
External source tables are populated by running their ingestion locally
- weather, events and the three hotel tables: done. flight_arrivals is
wired and tested but needs OpenSky and RapidAPI credentials. competitor_rates has no adapter and no source ticket.
Re-running the same ingestion produces zero duplicate rows (upsert verified)
All DB writes go through repository layer. No adapter touches a SQLAlchemy session or
SQL directly
Activity
[-]Create database and add data[/-][+]Persist Ingested Data Into Postgres[/+]refactor: move the events upsert into the repository layer