SokoData 🌾

Open Data Commons for African food markets. Markets, economy, climate, demographics, agriculture, health, education, energy, water, transport, mining, governance, trade, labour, environment, poverty, ICT, finance, tourism, aid, gender and geospatial — one API, built entirely on open data, serving 30+ African countries.

🔴 Live API: sokodata.onrender.com — interactive docs at /docs 🖥️ Live dashboard: alfredshingai.github.io/SokoData — browse markets & prices in your browser, no install

Why

A trader in a market, a farmer in a village, an NGO monitoring food security — none of them can easily answer "what does maize cost today, what is the currency worth, and did it rain where that maize was grown?". African country data is fragmented across WFP, central banks, national statistics offices, regulators, and climate APIs — as raw CSVs, PDFs, and HTML tables: no unified API, no provenance, no product for end users.

SokoData is the commons that turns those scattered sources into a clean, documented, queryable product — one ETL per dataset, one catalog, one API, with honest analytics that survive currency reforms and data revisions. Built for African food markets but country-agnostic by design.

What it gives you

Markets (live, WFP via HDX CC BY-IGO):

$ curl sokodata.onrender.com/v1/insights/movers?window_days=90

Fish (kapenta) at Marula: $6.10 → $10.53/kg (+72%) Oil (vegetable) at Gokwe: $1.70 → $2.74/L (+61%)

Economy (mixed: World Bank API + central bank/regulator scraping):

$ curl sokodata.onrender.com/v1/economy/cpi
$ curl sokodata.onrender.com/v1/economy/rates
$ curl sokodata.onrender.com/v1/economy/fuel

Climate (open API: Open-Meteo + NASA POWER):

$ curl "sokodata.onrender.com/v1/climate/daily?admin1=Harare"
$ curl "sokodata.onrender.com/v1/climate/monthly?admin1=Masvingo"

Demographics, Agriculture, Health + Education, Energy (WDI + FAO + scraping):

$ curl sokodata.onrender.com/v1/demographics/annual
$ curl sokodata.onrender.com/v1/agriculture/annual
$ curl sokodata.onrender.com/v1/health-stats/annual
$ curl sokodata.onrender.com/v1/education/annual
$ curl sokodata.onrender.com/v1/energy/annual

Water, Transport, Mining (WDI + water authority / transport ministry / mining chamber scrape):

$ curl sokodata.onrender.com/v1/water/annual
$ curl sokodata.onrender.com/v1/transport/annual
$ curl sokodata.onrender.com/v1/mining/annual

Governance, Trade, Labour (WDI + electoral commission / trade authority / labour survey scrape):

$ curl sokodata.onrender.com/v1/governance/annual
$ curl sokodata.onrender.com/v1/trade/annual
$ curl sokodata.onrender.com/v1/labour/annual

Environment, Poverty, ICT (WDI + environmental agency / stats office / telecom regulator scrape):

$ curl sokodata.onrender.com/v1/environment/annual
$ curl sokodata.onrender.com/v1/poverty/annual
$ curl sokodata.onrender.com/v1/ict/annual

Finance, Tourism, Aid (WDI + central bank / tourism authority / OECD scrape):

$ curl sokodata.onrender.com/v1/finance/annual
$ curl sokodata.onrender.com/v1/tourism/annual
$ curl sokodata.onrender.com/v1/aid/annual

Gender & Geospatial (WDI + UN Women / HDX COD-AB):

$ curl sokodata.onrender.com/v1/gender/annual
$ curl sokodata.onrender.com/v1/geospatial/boundaries
$ curl sokodata.onrender.com/v1/geospatial/markets/geojson

Cross-dataset Analysis (Anomalies + Forecasting):

$ curl sokodata.onrender.com/v1/analyze/anomalies?table=prices&column=usdprice&threshold=2.0
$ curl sokodata.onrender.com/v1/analyze/anomalies/correlated?table_x=prices&column_x=usdprice&table_y=climate_daily&column_y=tmean_c&join_on=country,date
$ curl sokodata.onrender.com/v1/analyze/anomalies/pattern?table=prices&column=usdprice&window=30
$ curl sokodata.onrender.com/v1/analyze/forecast?table=prices&column=usdprice&horizon=30

Catalog - discover every dataset:

$ curl sokodata.onrender.com/v1/catalog

markets (WFP), economy (WB + scraped), climate (Open-Meteo), demographics (WDI + census), agriculture (WDI + FAO), health (WDI + MoH), education (WDI + MoE), energy (WDI + energy ministry), water (WDI + water authority), transport (WDI + transport ministry), mining (WDI + mining chamber), governance (WDI + electoral commission), trade (WDI + trade authority), labour (WDI + labour survey), environment (WDI + environmental agency), poverty (WDI + stats office), ict (WDI + telecom regulator), finance (WDI + central bank), tourism (WDI + tourism authority), aid (WDI + OECD), gender (WDI + UN Women), geospatial (HDX COD-AB)

API

Endpoint	What it answers
GET /v1/catalog	All datasets, sources, licenses, freshness
GET /v1/markets?q=&admin1=	Market directory (486 markets, with coordinates)
GET /v1/commodities?category=	31 commodities with coverage counts
GET /v1/prices?market_id=&commodity_id=&start=&end=	Raw price series (USD + local)
GET /v1/prices/latest?commodity_id=	Most recent price per market
GET /v1/insights/movers?window_days=90	Largest USD price changes per market/commodity
GET /v1/insights/anomalies?threshold=2	Prices deviating from their own 24-month robust norm
GET /v1/economy/rates	FX rates (central bank scrape + World Bank fallback)
GET /v1/economy/cpi	CPI / inflation (stats office PDF + World Bank)
GET /v1/economy/fuel	Fuel prices by type (energy regulator scrape)
GET /v1/climate/daily?admin1=&start=&end=	Daily precip + temp by admin1 (Open-Meteo)
GET /v1/climate/monthly?admin1=	Monthly aggregates
GET /v1/demographics/annual	Population, growth, urban share (WDI)
GET /v1/demographics/census?admin1=	2022 Census by province (stats office scrape)
GET /v1/agriculture/annual	Cereal yield, agri GDP, food indices (WDI)
GET /v1/agriculture/fao/maize	Maize tonnes (FAO FAOSTAT)
GET /v1/health-stats/annual	Infant/under-5 mortality, immunization (WDI)
GET /v1/education/annual	Enrollment, literacy, completion (WDI)
GET /v1/energy/annual	Electricity access, use, renewables (WDI)
GET /v1/water/annual	Safe/basic water + sanitation (WDI)
GET /v1/transport/annual	Air, rail, road, internet use (WDI)
GET /v1/mining/annual	Mineral rents, ore/metal exports (WDI)
GET /v1/governance/annual	Property rights, transparency, parliament (WDI)
GET /v1/trade/annual	Exports/imports, merchandise values (WDI)
GET /v1/labour/annual	Unemployment, participation, vulnerable work (WDI)
GET /v1/environment/annual	CO2, forest cover, PM2.5 (WDI)
GET /v1/poverty/annual	Extreme poverty, Gini, national poverty (WDI)
GET /v1/ict/annual	Internet, mobile, broadband per 100 (WDI)
GET /v1/finance/annual	Domestic credit, remittances, private credit (WDI)
GET /v1/tourism/annual	Arrivals, receipts (WDI)
GET /v1/aid/annual	Net ODA, ODA % GNI (WDI)
GET /v1/gender/annual	Women parliament, female LFPR, parity (WDI)
GET /v1/geospatial/boundaries	HDX COD-AB metadata
GET /v1/geospatial/markets/geojson	486 markets as GeoJSON
GET /health	Dataset coverage + data provenance
GET /v1/meta/freshness	Per-table last update timestamp + row count
GET /v1/meta/freshness/summary	Total rows, tables with data, oldest/newest dates
GET /v1/auth/me	Current auth context (tier, rate limits)
POST /v1/auth/keys	Create API key (requires tier 2)
GET /v1/auth/keys	List API keys
DELETE /v1/auth/keys/{key_id}	Revoke API key
POST /v1/webhooks	Create webhook subscription
GET /v1/webhooks	List webhook subscriptions
DELETE /v1/webhooks/{webhook_id}	Delete webhook subscription
POST /v1/webhooks/test	Send test webhook
GET /webhooks/whatsapp	WhatsApp Cloud API verification
POST /webhooks/whatsapp	WhatsApp Cloud API incoming messages
GET /v1/export/datasets	List all exportable datasets
GET /v1/export/{dataset_id}	Export dataset (CSV, Parquet, GeoJSON)

Interactive OpenAPI docs ship at /docs when the server runs.

Platform Features (Phase 1)

Endpoint	What it answers
GET /v1/meta/freshness	Per-table last update timestamp + row count
GET /v1/meta/freshness/summary	Total rows, tables with data, oldest/newest dates
GET /v1/auth/me	Current auth context (tier, rate limits)
POST /v1/auth/keys	Create API key (requires tier 2)
GET /v1/auth/keys	List API keys
DELETE /v1/auth/keys/{key_id}	Revoke API key
POST /v1/webhooks	Create webhook subscription
GET /v1/webhooks	List webhook subscriptions
DELETE /v1/webhooks/{webhook_id}	Delete webhook subscription
POST /v1/webhooks/test	Send test webhook
GET /webhooks/whatsapp	WhatsApp Cloud API verification
POST /webhooks/whatsapp	WhatsApp Cloud API incoming messages
GET /v1/export/datasets	List all exportable datasets
GET /v1/export/{dataset_id}	Export dataset (CSV, Parquet, GeoJSON)

Interactive OpenAPI docs ship at /docs when the server runs.

Quickstart

pip install -e ".[dev]"          # add [pdf] for stats office/regulator PDF extraction: pip install -e ".[dev,pdf]"

# build the commons (all 22 datasets)
python -m sokodata.etl_run

# or run one dataset at a time:
python -m sokodata.datasets.markets.etl
python -m sokodata.datasets.economy.etl
python -m sokodata.datasets.climate.etl --admin1 Harare --start 2024-01-01
python -m sokodata.datasets.demographics.etl
python -m sokodata.datasets.agriculture.etl
python -m sokodata.datasets.health.etl
python -m sokodata.datasets.education.etl
python -m sokodata.datasets.energy.etl
python -m sokodata.datasets.water.etl
python -m sokodata.datasets.transport.etl
python -m sokodata.datasets.mining.etl
python -m sokodata.datasets.governance.etl
python -m sokodata.datasets.trade.etl
python -m sokodata.datasets.labour.etl
python -m sokodata.datasets.environment.etl
python -m sokodata.datasets.poverty.etl
python -m sokodata.datasets.ict.etl
python -m sokodata.datasets.finance.etl
python -m sokodata.datasets.tourism.etl
python -m sokodata.datasets.aid.etl
python -m sokodata.datasets.gender.etl
python -m sokodata.datasets.geospatial.etl

# legacy shim still works:
python -m sokodata.etl.run --skip-fetch

# serve the API
uvicorn sokodata.api.main:app --port 8000

To reuse already-downloaded CSVs: python -m sokodata.datasets.markets.etl --skip-fetch (expects them in data/raw/).

Strategy: where there's no API, scrape

The commons principle: open API first, HTML/PDF scraping where no API exists, graceful fallback always.

• Markets - open_api (HDX CSV, weekly) - src/sokodata/datasets/markets/fetch.py
• Economy - mixed: World Bank WDI JSON (CPI/FX, stable fallback) + HTML table scraping for central bank rates (fetch_html_tables + regex) with retries + exponential backoff + dead-letter queue via src/sokodata/core/queue.py. HTML/PDF for energy regulator fuel + PDF extraction for stats office CPI via pdfplumber. See src/sokodata/core/fetch.py for fetch_html_tables, fetch_html_text, extract_pdf_tables, parse_fuel_text. Scrapers log warnings and return [] if the upstream page/PDF changes — they never kill the ETL.
• Climate - open_api (Open-Meteo Archive + NASA POWER, daily, 1981-present) - no scraping needed. Fetch by admin1 centroid, stored as climate_daily/climate_monthly.

Platform reliability (Phase 1):

• Task queue — src/sokodata/core/queue.py: SQLite-backed queue with retries (exponential backoff), dead-letter table, worker pattern in src/sokodata/core/scraper.py.
• Auth & rate limiting — 3 tiers (Anonymous 60/min, Keyed 300/min, Premium 1000/min) with in-memory token bucket, API key management via /v1/auth/keys.
• Webhooks — src/sokodata/core/webhooks.py: subscription management, HMAC signing, multi-channel (HTTP/Telegram/Slack/Email), broadcast + test endpoint.
• Export — src/sokodata/api/routers/export.py: CSV/Parquet/GeoJSON streaming for all 28 tables with filters (country, date, admin1).

Adding a new angle = add src/sokodata/datasets/<name>/ with fetch.py/clean.py/store.py/etl.py and register it in src/sokodata/core/registry.py. The unified runner src/sokodata/etl_run.py and GET /v1/catalog pick it up automatically.

Deploy (free tier)

The repo ships a Render Blueprint — one click deploys the API:

1. Sign in at render.com with GitHub
2. New → Blueprint → pick alfredshingai/SokoData → Apply
3. Done — Render runs python -m sokodata.etl_run (rebuilds all datasets) and starts the API on every boot

Free-tier notes: the service sleeps after ~15 min without traffic (first request after that takes ~1 min while the warehouse rebuilds), and the disk is ephemeral — the SQLite file is rebuilt from source data each boot by design.

Telegram alerts

A GitHub Actions cron job posts a daily price digest (biggest movers + unusual prices) to a Telegram channel at 08:00 Harare time.

Setup, once:

1. In Telegram, message @BotFather → /newbot → choose a name and a ..._bot username → save the token it gives you
2. Create a public channel (e.g. @soko_prices) and add your bot as an administrator
3. Add repository secrets (Settings → Secrets → Actions): TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID (e.g. @soko_prices)
4. Trigger a test send from the Telegram price digest workflow → Run workflow

Anyone who wants alerts just joins the channel. Per-user subscriptions are on the roadmap once the project has persistent storage.

Web dashboard

A zero-dependency, mobile-first dashboard lives in docs/ (vanilla HTML/CSS/JS + Chart.js from CDN, no build step). It ships with the repo and is served by GitHub Pages — data comes straight from the public API, which allows cross-origin browser requests (CORS GET).

Run it locally against your local API: cd docs && python -m http.server 8899 → http://127.0.0.1:8899

WhatsApp bot

Send any message to our WhatsApp number and get the live digest back (greeting first, prices after). Built on the official WhatsApp Cloud API — free because the bot only replies to user-initiated messages.

Setup, once:

1. Create an app at developers.facebook.com → add the WhatsApp product → note the Phone number ID and temporary access token (API Setup page)
2. Add the test number as a recipient (dev mode allows 5 numbers; production needs business verification)
3. Webhooks: callback URL https://sokodata.onrender.com/webhooks/whatsapp, verify token = any string you choose, subscribe to the messages field
4. Set env vars on Render (WHATSAPP_TOKEN, WHATSAPP_PHONE_NUMBER_ID, WHATSAPP_VERIFY_TOKEN) or as GitHub secrets

The dev access token expires every 24h — refresh it on the API Setup page, or generate a permanent token via Business Settings when you verify the business.

Architecture

  Commons                              FastAPI
  datasets/markets ─┐                   ├─ /v1/catalog (22 datasets)
   HDX WFP (CSV)                    ├─ /v1/markets, /v1/commodities, /v1/prices
  datasets/economy/climate ─┤           ├─ /v1/economy/*, /v1/climate/*
  datasets/demographics/agri/health ─┤── SQLite ─┤  /v1/demographics/*, /v1/agriculture/*
  datasets/education/energy/water ─┤   │         ├─ /v1/health-stats/*, /v1/education/*
  datasets/transport/mining/gov/trade/labour ┘   ├─ /v1/energy/*, /v1/water/*, /v1/transport/*
  datasets/environment/poverty/ict/finance/tourism/aid/gender/geospatial ─┘
                                                  ├─ /v1/mining/*, /v1/governance/*, /v1/trade/*, /v1/labour/*
                                                  ├─ /v1/environment/*, /v1/poverty/*, /v1/ict/*
                                                  ├─ /v1/finance/*, /v1/tourism/*, /v1/aid/*, /v1/gender/*
                                                  ├─ /v1/geospatial/*
                                                  ├─ /v1/meta/*, /v1/auth/*, /v1/webhooks/*, /v1/export/*
                                                  └─ /v1/insights/*, /health
                                         core/fetch: open_api | html_scrape | pdf_extract
                                         core/queue: task_queue | retries | dead_letter
                                         core/webhooks: subscriptions | broadcast | HMAC
                                         core/auth: api_keys | tiers | rate_limits

• Zero-infra warehouse — SQLite with enforced natural keys and indexes; swap for Postgres later without touching the API layer. Per-dataset tables (prices, economy_*, climate_*) + dataset_meta.
• Currency-safe analytics — every temporal computation uses WFP's USD conversion, with broken hyperinflation-era conversions detected and nulled at load time (see docs/data_dictionary.md for the full rules and counts).
• Commons registry — src/sokodata/core/registry.py is the catalog of record. GET /v1/catalog and etl_run.py read it; new dataset = new folder + registry entry.
• Graceful scraping — src/sokodata/core/fetch.py atomic downloads, fetch_html_tables / extract_pdf_tables / parse_fuel_text log and return [] on upstream change instead of raising; ETL continues with open-API fallback.
• Task queue — src/sokodata/core/queue.py SQLite-backed queue with retries (exponential backoff), dead-letter table, worker pattern.
• Auth & rate limits — 3 tiers (Anonymous 60/min, Keyed 300/min, Premium 1000/min), API key management, in-memory token bucket.
• Webhooks — src/sokodata/core/webhooks.py subscriptions, HMAC signing, multi-channel, broadcast + test.
• Export — src/sokodata/api/routers/export.py CSV/Parquet/GeoJSON streaming for all 28 tables with filters.

Data credit & license

Data: World Food Programme Price Database via HDX, licensed CC BY-IGO + World Bank WDI CC BY-4.0 + Open-Meteo CC BY-4.0. Each dataset surfaces its own attribution via /v1/catalog/{id} and the existing /health. Code: MIT — see LICENSE.

Roadmap

• Commons foundation: core/registry + core/fetch (open_api / html_scrape / pdf_extract)
• Markets dataset (WFP, 27k obs, 486 markets) + API + dashboard
• Economy dataset (World Bank + central bank/regulator scrapers) + /v1/economy/*
• Climate dataset (Open-Meteo/NASA POWER, daily 1981-present) + /v1/climate/*
• Demographics (WDI + census) + /v1/demographics/*
• Agriculture (WDI + FAO FAOSTAT + agritex scrape) + /v1/agriculture/*
• Health (WDI + health ministry scrape) + /v1/health-stats/*
• Education (WDI + education ministry scrape) + /v1/education/*
• Energy (WDI + utility regulator scrape) + /v1/energy/*
• Water (WDI + water authority scrape) + /v1/water/*
• Transport (WDI + transport ministry scrape) + /v1/transport/*
• Mining (WDI + mining chamber scrape) + /v1/mining/*
• Governance (WDI CPIA + electoral commission scrape) + /v1/governance/*
• Trade (WDI + trade authority scrape) + /v1/trade/*
• Labour (WDI + labour survey scrape) + /v1/labour/*
• Environment (WDI + environmental agency scrape) + /v1/environment/*
• Poverty (WDI + poverty survey scrape) + /v1/poverty/*
• ICT (WDI + telecom regulator scrape) + /v1/ict/*
• Finance (WDI + central bank scrape) + /v1/finance/*
• Tourism (WDI + tourism authority scrape) + /v1/tourism/*
• Aid (WDI + OECD scrape) + /v1/aid/*
• Gender (WDI + UN Women scrape) + /v1/gender/*
• Geospatial (HDX COD-AB + markets GeoJSON) + /v1/geospatial/*
• Unified catalog GET /v1/catalog (22 datasets) + unified ETL python -m sokodata.etl_run
• Telegram digest (channel push, GitHub Actions cron)
• WhatsApp bot (on-demand digest replies via Cloud API)
• Phase 1: Freshness endpoint /v1/meta/freshness + nightly GitHub Actions cron
• Phase 1: Reliability — task queue with retries/dead-letter
• Phase 1: Auth — API keys with tiered rate limits
• Phase 1: Webhooks — subscriptions, broadcast, test endpoint
• Phase 1: Export — CSV/Parquet/GeoJSON streaming for all datasets
• WhatsApp push notifications (paid template messages) + per-user subscriptions
• Extend to all 98 countries in the WFP feed (config-driven, same pipeline)
• Geospatial boundaries + Environment/ICT as next commons pillars
• Seasonal baselines + harvest-cycle forecasting (ML) linking markets ↔ climate ↔ economy ↔ agriculture
• Digital Public Goods (DPG) submission as a commons

Digital Public Goods Compliance

SokoData is designed to meet Digital Public Goods Standard criteria:

Criterion	Status	Evidence
Open License	✅	MIT License (OSI-approved)
Open Data	✅	All 22+ datasets from open sources (WFP HDX CC-BY-IGO, World Bank, Open-Meteo, NASA POWER, FAO, UN, HDX COD-AB)
Open Standards	✅	OpenAPI 3.1, JSON, CSV, Parquet, GeoJSON; REST + streaming endpoints
Open Source	✅	Full source on GitHub, reproducible builds
Platform Independence	✅	Python 3.12+, runs on Linux/macOS/Windows, Docker-ready
Documentation	✅	README, interactive API docs (/docs), dataset catalog (/v1/catalog)
Community	✅	CONTRIBUTING.md, CODE_OF_CONDUCT.md, GOVERNANCE.md, SECURITY.md
Privacy & Safety	✅	No PII collected; API keys hashed; rate limits; security policy

Relevant SDGs: 2 (Zero Hunger), 8 (Decent Work), 9 (Industry/Innovation), 10 (Reduced Inequalities), 12 (Responsible Consumption), 13 (Climate Action), 17 (Partnerships). Primary focus: African food security (SDG 2).

Development

pip install -e ".[dev]"          # 47 tests: cleaning rules, warehouse, API, analytics, commons
# for PDF extraction:
pip install -e ".[dev,pdf]"
ruff check .          # lint
pytest -q

CI runs lint + tests on Python 3.12/3.13/3.14. Pull requests welcome — pick an issue or open one describing the problem first. To add a new dataset: copy src/sokodata/datasets/climate/ as a template, implement fetch/clean/store/etl, register in core/registry.py, and add catalog entry.

---

Built by Alfred Shingai. Not affiliated with WFP or HDX.