feat: implement EBM (and other things)

.env.example

@@ -3,8 +3,14 @@ APP_NAME=Foresight
 DEBUG=false
 
 # Database
-# Format: postgresql+asyncpg://<user>:<password>@<host>:<port>/<db_name>
+# App traffic. Format: postgresql+asyncpg://<user>:<password>@<host>:<port>/<db_name>
+# On Neon, `neon deploy` / `neon env pull` writes the pooled (-pooler) URL here in
+# libpq form (postgresql://...?sslmode=require&channel_binding=require); config.py
+# normalizes it for asyncpg, so paste it as-is.
 DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/dbname
+# Direct (non-pooled) URL used only by Alembic migrations. Leave empty to reuse
+# DATABASE_URL (fine for local Docker Postgres; required on Neon).
+DATABASE_URL_UNPOOLED=
 
 # External event sources
 PREDICTHQ_TOKEN=

.github/workflows/ci.yml

@@ -150,10 +150,19 @@ jobs:
         run: uv python install
 
       - name: Install dependencies
-        run: uv sync --dev --group ml
+        run: uv sync --dev
+
+      - name: Single migration head
+        run: test "$(uv run alembic heads | wc -l)" -eq 1
 
       - name: Run migrations
         run: uv run alembic upgrade head
 
+      - name: Models match migrations
+        run: uv run alembic check
+
+      - name: Migrations roll back
+        run: uv run alembic downgrade base && uv run alembic upgrade head
+
       - name: Pytest
         run: uv run pytest -q

.gitignore

@@ -53,3 +53,5 @@ backend/data/
 
 # requests-cache store written by the Open-Meteo client
 .cache.sqlite
+.neon
+node_modules/

AGENTS.md

@@ -1,7 +1,9 @@
 # AGENTS.md
 
 Entry point for AI coding agents working in **Foresight**. Read this first, then
-read the per-layer context doc in [`agents/`](agents/) for whatever you're touching.
+[`docs/system-design.md`](docs/system-design.md) for how the backend fits
+together, and the [decision records](docs/decisions/README.md) for the code
+you're touching.
 The goal: keep everyone's software practices aligned when they use AI.
 
 ## What Foresight is
@@ -21,8 +23,10 @@ GitHub Actions · ruff · Conventional Commits.
 routes  →  services  →  repository  →  database
 ```
 
-- **routes** — HTTP layer (FastAPI routers, `*Request`/`*Response` Pydantic schemas).
-- **services** — business logic, orchestration, ML/parsers.
+- **routes** — HTTP layer: FastAPI routers and the `*Request`/`*Response`
+  schemas each one uses.
+- **services** — business logic: `domain/`, `ingest/`, `adapters/`, `ml/`,
+  `job.py`.
 - **repository** — data access; the only layer that talks to the ORM/session.
 - **database** — SQLAlchemy models, migrations, session/engine.
 
@@ -31,19 +35,13 @@ ORM; services never build HTTP responses. Code lives in the installable package
 `backend/foresight/`.
 
 There's a working reference slice to copy — the `Widget` example threads all
-four layers (`example_widget.py` in each). Each doc below points to its file.
-
-Per-layer detail:
-- [`agents/routes.md`](agents/routes.md)
-- [`agents/services.md`](agents/services.md)
-- [`agents/repository.md`](agents/repository.md)
-- [`agents/database.md`](agents/database.md)
-- [`agents/schemas.md`](agents/schemas.md) — Pydantic standard (all layers)
+four layers (`example_widget.py` in each).
 
 ## Commands (run from repo root)
 
 ```bash
-just dev              # db + backend with hot reload
+just dev              # db + backend with hot reload (local Docker Postgres)
+just dev --neon       # same, against Neon; `--neon` works on run/migrate*/test too
 just migrate          # apply migrations
 just migrate-create "msg"   # new Alembic migration after model changes
 just migrate-down     # roll back last migration
@@ -61,14 +59,18 @@ API: http://localhost:8000 · docs: http://localhost:8000/docs
 - **Migrations:** any model change needs an Alembic migration that applies *and*
   rolls back cleanly.
 - **Config:** add new settings to `config.py` and document them in `.env.example`.
-- **Keep docs current:** if you change a layer's patterns, update that layer's
-  file in `agents/`.
+- **Keep docs current:** if you change how the system fits together, update
+  [`docs/system-design.md`](docs/system-design.md).
+- **Decisions:** design decisions live in [`docs/decisions/`](docs/decisions/),
+  one per file. Read the relevant record before changing that code; record new
+  decisions there.
 
 ## Do / don't for agents
 
-- **Do** follow the existing pattern in the layer you're editing — read its
-  `agents/` doc before writing code.
+- **Do** follow the existing pattern in the layer you're editing — see
+  `docs/system-design.md` and the `Widget` slice before writing code.
 - **Do** keep business logic in services and data access in the repository.
 - **Don't** skip layers, add dependencies without updating `pyproject.toml`, or
   change the schema without a migration.
-- **Don't** invent new conventions; this file and `agents/` are the source of truth.
+- **Don't** invent new conventions; this file, `docs/system-design.md` and
+  `docs/decisions/` are the source of truth.

README.md

@@ -27,6 +27,24 @@ just logs
 just destroy
 ```
 
+### Local vs Neon database
+
+Recipes that touch the database take an optional `--neon` flag. Without it
+they use the local Docker Postgres; with it, the Neon `production` branch
+(URLs from `.env`).
+
+```bash
+just dev                  # local Docker Postgres
+just dev --neon           # same stack, pointed at Neon (local db container not started)
+just migrate --neon       # migrations against Neon (direct, non-pooled URL)
+just run --neon           # API on the host, outside Docker
+```
+
+`--neon` works on `dev`, `destroy`, `run`, `migrate`, `migrate-create`,
+`migrate-down` and `test`. Re-run `just dev` / `just dev --neon` to switch;
+Compose recreates the backend. Neon URLs come from `neon env pull`. See
+`docs/decisions/0027-neon-postgres.md`.
+
 The API will be available at [http://localhost:8000](http://localhost:8000)  
 API docs at [http://localhost:8000/docs](http://localhost:8000/docs)
 
@@ -75,29 +93,25 @@ Commit messages must follow [Conventional Commits](https://www.conventionalcommi
 ```
 foresight/
 ├── backend/
-│   ├── foresight/               # Installable package (see pyproject.toml)
-│   │   ├── main.py              # FastAPI app entrypoint (foresight.main:app)
-│   │   ├── config.py            # Settings (pydantic-settings)
-│   │   ├── routes/              # API route handlers
-│   │   ├── schemas/             # Pydantic request/response models
-│   │   ├── services/            # Business logic
-│   │   │   ├── ml/              # ML models (regression, bayesian, ...)
-│   │   │   └── parsers/         # Client data ingestion
-│   │   ├── ingest/              # External-source adapters + CLIs (see docs/)
-│   │   ├── repository/          # Data-access layer
-│   │   └── database/            # SQLAlchemy + Alembic
-│   │       ├── base.py          # Declarative base
-│   │       ├── session.py       # Async session factory
-│   │       ├── models/          # SQLAlchemy models
-│   │       └── migrations/      # Alembic migrations
-│   ├── scripts/                 # Dev tooling (mock data, model comparison)
-│   ├── tests/                   # Pytest suite
-│   ├── alembic.ini
+│   ├── foresight/          # Installable package (see pyproject.toml)
+│   │   ├── main.py         # FastAPI app (foresight.main:app)
+│   │   ├── config.py       # Settings (pydantic-settings)
+│   │   ├── worker.py       # Runs background jobs
+│   │   ├── routes/         # HTTP: routers and their schemas
+│   │   ├── services/       # Business logic: domain, ingest, adapters, ml
+│   │   ├── repository/     # Data access
+│   │   └── database/       # Models, enums, session, Alembic migrations
+│   ├── tests/
 │   ├── pyproject.toml
 │   └── Dockerfile
-├── frontend/                    # TBD
+├── docs/
+│   ├── system-design.md    # How the backend fits together
+│   └── decisions/          # One design decision per file
+├── frontend/               # TBD
 ├── docker-compose.yml
 ├── justfile
-├── .env.example
-└── .pre-commit-config.yaml
+└── .env.example
 ```
+
+See [`docs/system-design.md`](docs/system-design.md) for the layers, data
+model, jobs and API.