European Power Fair Value Forecasting

DE-LU Day-Ahead fair-value forecasting project for European power-market analysis. The project covers public data ingestion, data QA, model validation, prompt curve translation, and a controlled LLM-assisted QA workflow.

1. Project Overview

Quick Run

The full pipeline can be reproduced with:

$env:PYTHONPATH="src"
python -m powerfair.cli run --config configs/de_lu_energy_charts.yaml

Suggested Reading Path

• reports/project_report.md: concise project report with source, QA, model metrics, curve translation, AI workflow, and limitations.
• reports/morning_desk_note.md: short desk note showing how the run would be used before looking at live marks.
• outputs/curve/prompt_curve_view.md: next-week and next-month baseload/peak fair-value table.
• outputs/qa/ai_qa_log.json: LLM prompt, raw response, accepted/rejected QA rules, fallback status, and executed checks.
• outputs/predictions_2025.csv: out-of-sample hourly predictions for the fixed 2025 test window.

What This Builds

• Hourly DE-LU Day-Ahead price dataset.
• Matching hourly fundamentals: load, wind, solar, residual load, and forecast/proxy fields.
• QA reports for missingness, duplicates, outliers, physical-value checks, coverage, and DST.
• Seasonal naive baseline, Ridge linear benchmark, and improved HGB model with blocked time-series validation.
• 2025 out-of-sample predictions_2025.csv.
• Next-week and next-month prompt curve fair-value view.
• Trader-facing morning desk note.
• AI QA prompt/output log.

Generated Artifacts

Generated artifacts include 4 PNG figures (actual_vs_predicted_test_week.png, model_comparison_mae_rmse.png, feature_importance.png, and coverage_missingness_heatmap.png) plus CSV/Markdown tables for QA, model metrics, walk-forward validation, predictions, and curve fair value. outputs/predictions_2025.csv contains 8,760 prediction rows, or 8,761 lines including the header.

2. Pipeline Flow

config
  -> fetch Energy-Charts/SMARD raw JSON
  -> cache raw responses in data/raw/
  -> build hourly processed dataset
  -> run coded QA and AI-assisted QA
  -> build model features
  -> train seasonal naive baseline, Ridge benchmark, and HGB models
  -> validate on walk-forward folds and 2025 OOS
  -> translate hourly forecasts to next-week/month curve views
  -> write project report, desk note, figures, and predictions_2025.csv

The same flow is also available as a supporting artifact in reports/pipeline_flow.md.

3. Setup and Reproduction

Environment Setup

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
Copy-Item .env.example .env

Linux / macOS:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Environment Variables

• GROQ_API_KEY or OPENAI_API_KEY: enables LLM-proposed QA rules. Without a supported key, the same QA hook logs the fallback and executes mandatory coded rules.
• AI_QA_PROVIDER: optional provider selector, one of auto, groq, or openai. auto prefers Groq, then OpenAI.
• GROQ_MODEL: defaults to llama-3.1-8b-instant.
• OPENAI_MODEL: defaults to gpt-4o-mini.

Full Run

Energy-Charts/SMARD DE-LU route:

$env:PYTHONPATH="src"
python -m powerfair.cli run --config configs/de_lu_energy_charts.yaml

Smoke Run

For a quicker run over a shorter date range:

$env:PYTHONPATH="src"
python -m powerfair.cli run --config configs/de_lu_energy_charts.yaml --start 2024-01-01 --end 2026-03-01

AI Provider Selection

Groq route:

# Add GROQ_API_KEY to .env first.
$env:AI_QA_PROVIDER="groq"
$env:PYTHONPATH="src"
python -m powerfair.cli run --config configs/de_lu_energy_charts.yaml

OpenAI route:

# Add OPENAI_API_KEY to .env first.
$env:AI_QA_PROVIDER="openai"
$env:PYTHONPATH="src"
python -m powerfair.cli run --config configs/de_lu_energy_charts.yaml

4. Outputs

Processed Data

• data/processed/de_lu_hourly.csv
• data/processed/de_lu_features.csv

QA Outputs

• outputs/qa/qa_report.md
• outputs/qa/qa_report.json
• outputs/qa/ai_qa_log.json
• outputs/qa/coverage_missingness_heatmap.png

Model Outputs

• outputs/model/model_metrics.csv
• outputs/model/model_training_summary.json
• outputs/model/walk_forward_metrics.csv
• outputs/model/test_predictions.csv
• outputs/model/model_comparison_mae_rmse.png
• outputs/model/actual_vs_predicted_test_week.png
• outputs/model/feature_importance.png

Curve Outputs

• outputs/curve/prompt_curve_view.md
• outputs/curve/prompt_curve_view.csv

Predictions and Reports

• outputs/predictions_2025.csv: included prediction file with one row per 2025 out-of-sample delivery hour. It contains only id as the UTC timestamp and y_pred as the HGB point forecast.
• reports/project_report.md
• reports/morning_desk_note.md
• reports/pipeline_flow.md
• requirements.txt and pyproject.toml

5. Data and Features

Data Sources

• Energy-Charts API: https://api.energy-charts.info/
• Day-Ahead price endpoint: /price?bzn=DE-LU&start=...&end=...
• Public fundamentals endpoint: /public_power?country=de&start=...&end=...
• Renewable forecast endpoint: /public_power_forecast?country=de&production_type=solar|wind_onshore|wind_offshore&forecast_type=current&start=...&end=...
• SMARD data usage: https://www.smard.de/en/datennutzung
• Attribution: Bundesnetzagentur | SMARD.de, CC BY 4.0

Feature Engineering

The model uses timestamp-derived, lagged, and public fundamental features:

• Market calendar: local hour, weekday, month, weekend, German holidays, DST, and peak/off-peak flag.
• Cyclical calendar transforms: hour, weekday, and month sine/cosine features.
• Fundamentals: load, wind, solar, renewable generation, renewable share, and residual-load proxy.
• Price history: 24h, 48h, and 168h price lags plus 24h/168h rolling mean and volatility.
• Leakage guard: feature_as_of_utc is checked to ensure feature timestamps are earlier than target timestamps.

The final feature table is written to data/processed/de_lu_features.csv.

Time Handling

• Canonical timestamps are UTC.
• Europe/Berlin local time is derived only for market-calendar features and DST reporting.
• Energy-Charts fundamentals are 15-minute data and are averaged to hourly in UTC before joining with Day-Ahead prices.
• The QA report explicitly checks 23-hour and 25-hour local DST days.

6. AI QA Purpose

The LLM QA step is not a replacement for deterministic QA and is not required for the model to run. Its purpose is to show a controlled, programmatic use of AI: the pipeline sends schema, sample rows, missingness, summary stats, and metadata to an LLM, asks for candidate QA rules, then validates and filters those rules before execution. The code remains in control, logs the full prompt/response, and falls back to mandatory coded QA checks if no provider is available or the LLM response is unsafe.

7. Source Layout

The source package is divided by project responsibility:

• src/powerfair/data/: config loading, Energy-Charts/SMARD ingestion, caching, and timezone/DST utilities.
• src/powerfair/validation/: deterministic QA checks and report assembly.
• src/powerfair/ai/: LLM QA rule proposal, parsing, filtering, execution, and logging.
• src/powerfair/forecasting/: feature engineering, leakage checks, seasonal naive baseline, Ridge benchmark, gradient boosting model, quantile models, and validation.
• src/powerfair/trading/: DA forecast aggregation into next-week and next-month prompt curve fair-value views.
• src/powerfair/reports/: project report and morning desk note generation.
• src/powerfair/cli.py: orchestration entry point that wires the full workflow together.

8. Notes and Limitations

• Energy-Charts forecast endpoints are public and reproducible, but they are not a full archived forecast-vintage database.
• A production version should maintain point-in-time forecast-vintage tables. Same-hour realised load/wind/solar actuals should be used for QA and ex-post analysis, while model validation should join only forecasts that were available before the target delivery hour.
• For the no-key route, load forecast is an ex-ante seasonal proxy from last week's same-hour actual load; wind and solar use the public forecast endpoint where available.
• The 2025 out-of-sample metrics are calculated before the final prompt-curve model refit; the curve model is refit only after validation using rows before forecast_as_of.
• Negative Day-Ahead prices are valid and are not removed; only extreme outliers are flagged.
• Missing target rows are dropped for modelling and reported in QA rather than imputed.
• The prompt-curve view is a fair-value input. A live trading edge would require comparison with forward market marks.

9. Tests

$env:PYTHONPATH="src"
pytest

See tests/README.md for a short explanation of the targeted DST, leakage, and AI QA guardrail tests.