Skip to content

Persist Ingested Data Into Postgres #36

Closed

Description

@emmagershman

Problem

There is external data being scraped but not being populated in the database.
Refer to #5, #7, #8, #9

Goal

Every ingestion source persists into its Postgres table through a shared repository layer, so downstream model training
and the API read from one source of truth.

Scope

  • Build out foresight/repository/ with one repository per table (weather, events, competitor_rates,
    flight_arrivals, and hotel tables).
  • Define a single persistence contract: adapter produces normalized records → repository upserts
    them.
  • Add idempotency/upsert support
  • Wire external adapters to persist through the repository as the reference
    implementations.
  • Consider database best practices (cascading, indexing, cardinality, primary and foreign keys)

Acceptance criteria

  • External source tables are populated by running their ingestion locally
    - weather, events and the three hotel tables: done. flight_arrivals is
    wired and tested but needs OpenSky and RapidAPI credentials.
    competitor_rates has no adapter and no source ticket.
  • Re-running the same ingestion produces zero duplicate rows (upsert verified)
  • All DB writes go through repository layer. No adapter touches a SQLAlchemy session or
    SQL directly

Activity

  1. changed the title [-]Create database and add data[/-] [+]Persist Ingested Data Into Postgres[/+] on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

featureIntroduces a new and complete feature

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions