A self-hosted world-event monitor that shows its evidence — and publishes the experiments where its own predictions failed.
It collects public data about world events — news, machine-coded event records, disasters, markets, cyber indicators, satellite fire detections, aircraft presence — normalises it into one row shape, stores it locally, and puts it on a map with the provenance attached. It runs on one machine. Nothing leaves it.
basilsuhail.github.io/OSINT — the map, the filters and the cards, drawn from a frozen snapshot of a running console. Hazards, military aircraft and AIS vessels, all clickable. Nothing there is live: it is one moment saved to a file, and the page says so at the top and again beside anything it had to thin out.
Everything below is how to run the real thing, which fetches, stores, scores and refreshes on your own machine.
Skip if you already run Docker and current Node.
Debian, Ubuntu, Raspberry Pi OS — then log out and back in:
sudo apt update && sudo apt install -y git curl ca-certificates && \
curl -fsSL https://get.docker.com | sudo sh && \
sudo usermod -aG docker "$USER" && \
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - && \
sudo apt install -y nodejs && sudo corepack enablemacOS, with Homebrew:
brew install git node && brew install --cask docker && sudo corepack enable && open -a DockerOptional — but without it the Ask panel does not answer. Ask it anything and
it replies The brain is offline right now. The map, the feed, ingestion and
every number still work; the written summaries and the question box do not.
curl -fsSL https://ollama.com/install.sh | sh # Linux
brew install ollama && ollama serve # macOSmake up starts Ollama and pulls what it needs — three models, about 5 GB in
total: one writes the situation summary, one answers in the Ask panel, one
builds the embeddings behind search. Only one is ever loaded at a time. You can
add Ollama later and re-run make up; nothing else needs redoing.
Docker and Node come from their own installers because distribution packages are
usually too old. Do not install pnpm by name — corepack enable fetches the
version packageManager pins, and a different one resolves the lockfile
differently. Details and the swap setting for a single-board host:
§3.1.
git clone https://github.com/BasilSuhail/OSINT.git
cd OSINT
make env # makes your settings file (.env), filled in and ready
make up # postgres · redis · migrations · api · workers · console · ollama if present
make down # stop everything, keep all dataOpen http://localhost:3000. First run takes several minutes — images, packages and the local model download.
Without Ollama everything still runs and the Ask panel answers The brain is offline right now. — see step 1 to add it.
.env is the only file you edit, it is git-ignored, and
env.example is
its template. There is nothing you have to put in it to start: make env
generates the database password and the API token, and points the console at
this machine. Optional source keys and moving the data directory are in
§5.
Three commands look after that file, and you can run any of them as often as you like:
| Command | What it does |
|---|---|
make env |
Makes .env if you have not got one. If you have, it adds the settings you are missing, fills in any that are empty and should not be, and leaves everything you already filled in exactly as it is. |
make env-refresh |
Re-derives the addresses after this machine moves network or changes name. Addresses only — it cannot reach a credential. |
make env-check |
Tells you what is missing, what still needs a value, and what you have spelled wrong. It never prints a value, so it is safe to run while someone is watching your screen. |
To open the console from another device, put the name that device would use in
OSINT_PUBLIC_HOST, run make env-refresh, then make share.
make help lists every command in the Makefile with a line saying what it
does.
Pull the latest:
git checkout main
git pull
make env # adds any settings the update introduced
make upTry a branch — a fix you want to test before it merges, say:
git fetch origin
git checkout <branch-name>
git pull
make env
make upBack to main afterwards with the first block. make down first if the stack
is running and the branch changes how it starts.
make env after every pull is the step people skip. New settings arrive in
env.example over time and this is how they reach your file; miss it and the
feature they switch on stays quietly off, with nothing saying so. It never
touches a value you have already set. make up runs the check and tells you
what it found, then starts anyway.
Full prerequisites: §3. The walkthrough: §1. Anything that goes wrong: §19.
Read §0 before trusting any number on the screen. The predictive claim this project was built to test has been put through every pre-registered protocol built for it and refused every time. That is stated up front, with the tables, rather than buried.
Licence: PolyForm Noncommercial 1.0.0 — source-available, not open source. Security reporting, provider data terms, and attribution are in §25.
4,300 lines. 24 sections. Everything underneath this page.
| The handbook covers | |
|---|---|
| Operate it | Install and first run · read the console · configure · every control · troubleshooting |
| Every formula | Corroboration · divergence · the composite · severity · lead-time gate — each with a worked example and its failure modes |
| What was tested, and what failed | Every pre-registered refusal, with baselines, bootstrap confidence intervals, and the held-out window |
| Whether to trust the data | Bias and provenance, including one country traced end to end with measured counts |
| Rebuild it yourself | Every command that regenerates every number quoted anywhere |
| Reference | Data sources · pipeline · storage · backend · frontend · glossary · code walkthroughs |
Every numbered control above is explained in §2.1 of the handbook, each with a link to the code that produces it.
- 0. What this system is for, and what it is not
- 25. Licence, security, and provider terms
- 26. Repository map
- 27. Documentation index
- 28. SWOT
Section numbers are shared across both files and never reused: 1–24 are in HANDBOOK.md, so a reference to §15 means the same thing wherever you read it.
Read this page first. It states the claim the project makes, the claim it does not make, and what is actually being offered — so nothing later in the handbook has to be inferred.
This is a self-hosted system that collects public data about world events — news, machine-coded event records, disasters, markets, cyber indicators, satellite fire detections, aircraft presence — normalises it into one row shape, stores it locally, and puts it on a map with the evidence attached. It runs on one machine. Nothing leaves it. Every number it shows can be traced back to the row that produced it.
A composite of several open-data signal domains discriminates later instability better than the best single-domain baseline.
That claim was pre-registered, evaluated, and refused. Not once — under every protocol built for it, including on a held-out window reserved specifically for the question:
| Evaluation | Result |
|---|---|
| Incidence, pooled | Composite AUROC ≈ 0.502 against a 0.929 base rate |
| Head-to-head vs single domains | beaten: [] — dominates none of them, in either window |
| Held-out test 2023–24 | Same verdict on data reserved for this question |
| Onset | 0.496 / 0.520 / 0.526 — a coin flip |
| Within-country concordance | 0.531 best, CI [0.474, 0.582], below the declared 0.55 |
The predictive claim failed. This document does not soften that anywhere, and §15 gives every table.
The composite failed. The machinery that caught it did not.
What this project delivers is an apparatus for not fooling yourself, built around data that makes fooling yourself easy. Concretely:
- Protocols are frozen before results exist. Eligibility, target, horizons, contenders, metrics and the decision rule are written down and dated first. The run happens once. Corrections are amendments, never edits (§15.1).
- Verdicts are computed, not narrated. A threshold decides, in code. The
within-country evaluation prints
NEGATIVEbecause_verdict()said so, not because someone read the table and agreed (§15.7). - A trend that fails the rule is reported as a failure. The composite rises monotonically with horizon. The protocol declared in advance that 0.50–0.55 is a negative rather than a promising trend, so it is recorded as a negative (§15.7).
- Forecasts of a constant are refused entry. 1,101 forecasts of the number 0.5 were once recorded as forecasts. The check is now concentration, not exact flatness (§14.4).
- A hindcast cannot enter the journal. A "prediction" whose window overlaps the known past is skipped, because grading it would fake a track record (§15.10).
- Independence must be positively established. A source with no ownership record contributes nothing to a confidence score, because ten anonymous blogs are one claim told ten times, not ten confirmations (§14.6).
- Absence is not imputed as average. A missing domain is excluded and the weights renormalised, rather than entered as "exactly average" (§14.4).
- The bias is measured and published, including the finding that 54 of 55 news feeds are English and that a country the system reports on can have no domestic feed at all (§16).
The evidence that this apparatus works is that it caught its own author's claim and refused it, repeatedly, on the record. A system that only ever confirms the hypothesis of the person who built it has not been tested. This one has.
- It does not forecast instability. See above. Any future claim requires a new pre-registered version and a new evaluation.
- It does not calculate truth. It shows provenance, corroboration, disagreement and sensor agreement. Six independent outlets can be independently wrong.
- Its scores are not probabilities. They are orderings. Nothing here is calibrated, so 0.875 does not mean "correct 87.5% of the time".
- It does not see the whole world. It sees what open, mostly English-language feeds publish, and it says so with numbers rather than a disclaimer.
- It is not a safety-critical alerting service. Upstream feeds fail, arrive late, change format, and withdraw access.
Stated here rather than left to be discovered:
- The running version has never been evaluated. Everything in §15 grades
v1.0. The live system emitsv3.0, which changed the domain structure and the weights (§14.1). - There is no forward evidence yet. 1,695 predictions issued, 0 graded (§15.10).
- Five of six declared robustness tests have not been run (§15.9).
- Two of the five label families were never built.
P4(market crisis) andP5(hazard disruption) do not exist, so a multi-modal index is currently graded only on whether it predicts conflict (§15.2). - The pre-specified case studies were never filled in.
- Reporting delay has never been measured on this system's own data (§16.4).
It is useful to someone who has to answer "is this story actually being independently reported, or is it one wire item repeated?", "does any physical sensor agree with this claim?", "are two countries telling this differently?", and "how much should I trust what I am looking at?" — and who wants each answer to arrive with the evidence attached rather than as a verdict.
It is not useful to someone who wants a risk number to act on. That number was built, tested, and found not to work.
| If you want to | Read |
|---|---|
| Run it | §1 |
| Understand the screen | §2 |
| See how a number is computed | §14 |
| See what was tested and what failed | §15 |
| Judge the data before trusting it | §16 |
| Rebuild every number yourself | §17 |
| Find any file in the repository | §26 |
Find the right document in docs/ |
§27 |
| Know the licence, security policy, and data terms | §25 |
| See a candid read of where this stands | §28 |
PolyForm Noncommercial 1.0.0. Full text in
LICENSE.
This is source-available, not open source. The distinction is not pedantry: every OSI-approved licence permits commercial use, so calling this open source would tell you that you have a right you do not have.
You may use it, run it, study it, fork it, modify it and share your changes, for any noncommercial purpose — personal projects, study and research, and use by charities, educational institutions, public research bodies and government. You may not sell it or use it commercially.
Two reasons, and the second is the real one:
- It is under development and its outputs have been wrong before. The composite is a coin flip in every pre-registered evaluation, and that is written down in §15 rather than hidden. Nothing here is fit to sell.
- It ingests third-party feeds. Several are free for noncommercial or research use and require a separate agreement for anything else. Those terms are not the maintainer's to hand on, so they are not handed on.
The licence covers the code in this repository and nothing else. It does not cover, and cannot cover, the data this software fetches. Those feeds belong to the organisations that publish them, each on its own terms. Nobody here has the right to sub-licence them, so nobody here has granted you anything over them.
Running this software makes you the one fetching the data. Whatever the provider requires — registration, an API key, attribution, a commercial licence, a limit on redistribution — it requires of you, directly, under the agreement you accept when you take the key.
NOTICE.md is the
maintained index of every feed and where its terms live. It is a pointer, not a
legal summary and not legal advice. Terms change. Read the current ones for any
feed you actually enable. Operational guidance on this is in
§9.5.
The bundled gazetteer files under
app/enrichment/data/
are Natural Earth, which is public domain.
Base map tiles, imagery overlays and aircraft presence are fetched by the browser directly from their publishers and are never stored; their attribution is rendered on the map itself.
Report privately — do not open a public issue. The channels, the information
to include, and the automated tooling in place are in
SECURITY.md.
What runs continuously on this repository:
| Control | What it covers |
|---|---|
| Dependabot | pip and pnpm dependencies, weekly, opens update PRs |
| CodeQL | push, pull request, and a weekly scheduled scan |
pip-audit |
backend dependencies, in the backend workflow |
pnpm audit --audit-level high |
frontend dependencies, in the frontend workflow |
| Dependency review | blocks high-severity dependency regressions on PRs |
| Secret scanning + push protection | enabled at repository level |
The repository-level security baseline, what is enabled and what hardening is
still recommended, is logged in
docs/security.md.
Operational security when you run it is a separate matter and is covered in
this handbook: the API token in §5.4,
and network exposure — including the fact that make share adds no password
— in §5.7.
This repository is public and has been forked; forks copy the full history.
Anything committed is beyond recall the moment it is pushed. The rules for what
may and may not go into a commit, an issue or a pull request are in
AGENTS.md, which is
read by both people and coding agents.
The short version: no personal names, no contact details, no credentials, and one issue → one branch → one pull request → one commit. A pre-commit hook screens staged files and blocks on a hit; a block is a prompt to rephrase, not an obstacle to route around.
Issues are welcome. A good one names what you ran, what you expected, and
what happened, with the relevant lines from bash scripts/dev-logs.sh. If the
console is involved, say which browser. §19
covers the failures that already have known causes — worth a look first, because
several of them look like bugs and are configuration.
Security problems do not go in issues. Use the private channel in §25.4.
Pull requests: one issue, one branch, one pull request, one commit. Read
AGENTS.md first —
it is short, and it is binding on what may appear in a commit message, an issue
or a PR description, because this repository is public and has been forked.
Before opening one:
make verify # or: ruff check . && ruff format --check . && pytest
cd osint-frontend && pnpm exec tsc --noEmit && pnpm exec vitest runA pre-commit hook screens staged files and blocks on a hit. A block is a prompt to rephrase, not an obstacle to route around.
What is most useful right now, in order: an evaluation of the running
v3.0 composite (§14.1),
any of the five unrun robustness tests
(§15.9),
a chance-corrected agreement statistic for the severity grader
(§14.5),
and non-English or non-Anglophone-origin feeds
(§16.2).
The shape first, then the route data takes through it.
OSINT/
├── app/ PYTHON BACKEND — ingest · score · serve
│ ├── api.py FastAPI read-API: /events /scores /ingest-health /stream
│ ├── celery_app.py Celery app instance (broker = Redis)
│ ├── tasks.py Celery tasks + beat schedule (cadence + nightly prune)
│ ├── fetcher_registry.py maps source name → fetcher
│ ├── persistence.py upsert events into Postgres (+ Redis "new rows" tick)
│ ├── events_bus.py Redis pub/sub channel powering the live SSE stream
│ ├── housekeeping.py retention policy (see RETENTION_* in .env)
│ ├── db.py / db_models.py SQLAlchemy engine/session + table definitions
│ ├── settings.py ALL config, read from .env
│ ├── models.py canonical Event/Score shapes
│ ├── watchdog.py ingest health monitor
│ ├── sources/ one fetcher per feed + rss_feeds.json registry
│ ├── composite/ aggregation · normalisation · scoring · backfill
│ ├── corroboration/ per-story confidence + sensor cross-checks
│ ├── disagreement/ cross-country telling divergence
│ ├── divergence/ physical-vs-narrative spike and lead-time gate
│ ├── cii/ Country Instability Index
│ ├── journal/ immutable prediction journal
│ ├── onset/ within/ baselines/ the pre-registered evaluations
│ ├── audit/ nightly source-data audit + expectations
│ ├── brain/ local-model narrate · enrich · ask
│ ├── severity/ grading and measured agreement
│ ├── devx/ lan_share — who on the network may reach this
│ └── enrichment/ geocode · NER · sentiment (+ data/ gazetteers)
│
├── osint-frontend/ NEXT.JS CONSOLE — reads app/api.py
│ ├── app/ routes: page.tsx, layout.tsx, providers.tsx, news/
│ ├── lib/ apiClient.ts · queries.ts · realtime.ts · types.ts
│ ├── components/ MapPane · CardDeck · FilterRail · panels/
│ ├── stores/ zustand filter + selection stores
│ └── __tests__/ frontend suite
│
├── data/ ALL LOCAL STORAGE ($OSINT_DATA_DIR, gitignored)
│ ├── postgres/ the actual database files
│ ├── redis/ Redis append-only file
│ ├── private/ licensed/manual inputs — never commit
│ └── exports/ generated reports (evaluations, audits)
├── backups/ snapshot dumps (gitignored)
├── migrations/ Alembic schema migrations
├── scripts/ dev-up.sh · dev-down.sh · snapshot.py · one-off tools
├── tests/ pytest suite (backend)
├── docs/ specifications, protocols, and evaluation records — see §27
│
├── docker-compose.yml Postgres + Redis services
├── Makefile every command in this handbook
├── env.example copy → .env, then fill in
└── .env YOUR config and secrets (gitignored — never commit)
| I want | Open |
|---|---|
| My config and secrets | .env, from env.example; read via app/settings.py. Every setting explained in §5 |
| The database itself | data/postgres/ — relocate with OSINT_DATA_DIR (§5.5) |
| What the console fetches | osint-frontend/lib/apiClient.ts ↔ served by app/api.py |
| To add or adjust a source | app/sources/ + app/fetcher_registry.py — full steps in §12.5 |
| How long data is kept | app/housekeeping.py and RETENTION_* (§11.3) |
A GDELT event's whole life. Every path is a link.
| # | Stage | File | What happens |
|---|---|---|---|
| 1 | fetch | app/sources/gdelt_fetcher.py |
request the newest export |
| 2 | register | app/fetcher_registry.py |
name → fetcher lookup |
| 3 | schedule | app/tasks.py |
Celery Beat fires it on cadence |
| 4 | dedup + store | app/persistence.py |
upsert on stable identity → events |
| 5 | normalise | app/composite/normalization.py |
rolling within-country z-score (§14.4) |
| 6 | score | app/composite/scoring.py |
weighted z → sigmoid → scores |
| 7 | serve | app/api.py |
/events, /scores, SSE stream |
| 8 | fetch in browser | osint-frontend/lib/apiClient.ts |
every backend call in one file |
| 9 | render | osint-frontend/components/ |
map, cards, panels |
| Source | Fetcher | Output |
|---|---|---|
| GDELT live | app/sources/gdelt_fetcher.py |
events, rolling window |
| GDELT history | app/composite/gdelt.py |
monthly checkpoints + daily volume |
| USGS · GDACS · FIRMS · EONET | app/sources/ |
events → footprint enrichment |
| yfinance · FRED | app/sources/ |
events, market and macro |
| ACLED (labels) | local drop folder (§5.3) | labels — ground truth, kept separate |
| RSS, 55 feeds | app/sources/rss_feeds.json |
events → stories |
| Concern | Folder | Formula | Method in |
|---|---|---|---|
| Composite index | app/composite/ |
normalization.py · scoring.py |
§14.4 |
| Corroboration | app/corroboration/ |
score.py |
§14.2 |
| Telling divergence | app/disagreement/ |
tellings.py |
§14.3 |
| Lead-time gate | app/divergence/ |
config.py |
§14.8 |
| Country Instability Index | app/cii/ |
scoring.py |
§14.9 |
| Prediction journal | app/journal/ |
emit.py |
§15.10 |
| Evaluations | app/onset/ · app/within/ · app/baselines/ |
— | §15 |
| Retention | app/housekeeping.py |
— | §11.3 |
This handbook is the entry point. The files below are the primary records it draws on — read the protocol before the result, which is the order they were written in.
| Document | What it is |
|---|---|
docs/methodology.md |
Part A: the pre-registered evaluation protocol — ground truth, splits, baselines, metrics, sensitivity programme, reporting checklist. Part B: the literature baseline with citations and reading priority. |
docs/onset-eval.md |
The onset evaluation, frozen 2026-07-10, with its single run and amendment log (§15.6). |
docs/within-country-eval.md |
The within-country evaluation, frozen 2026-07-22, and its NEGATIVE verdict (§15.7). |
docs/disagreement-exam.md |
A forward evaluation — not gradable until enough predictions mature. |
docs/backtest/ |
Lead-time gate reports, including threshold sensitivity (§15.8). |
docs/severity-grading.md |
How headline severity is decided, which model, and the measured agreement (§14.5). |
docs/analytical-agenda.md |
The workstreams: what is actually done with the data — quantify, validate, predict. |
docs/project-direction.md |
What the project is, who it serves, and the long-term path. |
docs/data-coverage.md |
The operational record of what actually landed in storage (§16.3). |
docs/storage.md |
Storage layout, retention, move, back up, restore, wipe (§11). |
docs/acled-non-api-collection.md |
How the label data is obtained without an API. |
docs/security.md |
Repository security baseline: what is enabled, what hardening remains (§25.4). |
docs/architecture/ |
The build specification: 01 overview · 02 storage · 03 ingestion · 04 schema · 05 originality · 06 validation · 07 risks |
docs/architecture/CII-METHODOLOGY.md |
Per-country baseline and the four-component event blend (§14.9). |
docs/architecture/ENRICHMENT-METHODOLOGY.md |
Sentiment, NER, city resolution, news-scope classification. |
docs/audits/ · docs/frontend/ |
Hand-checked audits; console design notes. |
A candid read of where the project stands, kept with the code so it stays current rather than becoming a pitch.
| Helpful | Harmful | |
|---|---|---|
| Internal | Strengths | Weaknesses |
| External | Opportunities | Threats |
- A pre-registration and verdict machine that structurally cannot flatter itself: protocols frozen first, verdicts computed in code, negatives published in full, none buried (§0.3).
- Reproducible, local-first, no cloud. Idempotent backfills, every method version-stamped and never edited in place.
- Genuine multi-modal ingestion with structural deduplication and published coverage bias rather than a disclaimer (§16).
- The corroboration and divergence engine works today and is independent of the composite claim that failed.
- The headline claim failed. The composite beats none of the single-domain baselines, on either window (§15.5).
- The running version has never been evaluated. Every published result
grades
v1.0; the live system emitsv3.0(§14.1). - No forward evidence yet — 1,695 predictions issued, none graded (§15.10).
- The live composite is degenerate at 0.5 from the retention-versus-z-score mismatch, and the label panel is maintained by hand.
- Five of six declared robustness tests have not been run (§15.9).
- Built by one maintainer with heavy assistance from language models; depth of understanding of the internals is the standing risk, and this handbook plus §14 is the mitigation.
- Slow-onset hazards — drought, flood, sustained unrest — are the untested anchor where a sensor could plausibly lead coverage. An open question, not a settled failure.
- The hazard domain is the strongest single indicator measured so far and beats the composite that contains it. Making it survive a fair onset evaluation is a well-defined next task.
- The corroboration and coverage engine has standalone value: it answers "is this independently reported" without needing the composite to work.
- Upstream drift: GDELT gaps, label-source access changes, RSS format changes, and sensor values that turn out to measure something other than what they appear to.
- Retention versus evaluation needs can silently flatten a signal before it is ever measured — the class of defect that produced 1,101 forecasts of a constant.
- The hardest questions to answer are the within-country construction and the 0.5 degeneracy — both of which the composite's published results depend on.
Start here: §0 for what this claims and refuses to claim, Quick start to run it, and HANDBOOK.md for everything else.