Open Source Gap

An interactive visualization of the performance gap between frontier open and closed-source AI models across multiple benchmarks.

Overview

This application visualizes how long it takes for open-source models to "match" the performance of state-of-the-art closed-source models. It calculates the time gap between a closed model's release and the first open model that subsequently matches or exceeds its score on various benchmarks.

Supported Benchmarks:

• Epoch Capabilities Index (ECI) - Comprehensive AI capabilities index from Epoch AI
• GPQA Diamond - Graduate-level science questions (Diamond subset)
• MATH Level 5 - Competition mathematics problems (hardest level)
• OTIS Mock AIME - Mock AIME competition problems
• SWE-Bench Verified - Software engineering bug fixing benchmark
• SimpleQA Verified - Simple factual question answering
• FrontierMath (Public) - Frontier-level mathematics problems
• Chess Puzzles - Chess tactical puzzles

Key Features:

• Multi-Benchmark Support: Compare gaps across different AI capability dimensions.
• Interactive Timeline: Explore model releases and performance gaps over time.
• Frontier Tracking: Focuses on the "frontier" of AI capabilities.
• Statistical Analysis: Automatically calculates the average gap and confidence intervals.
• Threshold-Crossing Analysis: For each benchmark score threshold, how much later the first open model reached it than the first closed model (methodology ported from open_closed_gap; published in data.json under threshold_analysis / threshold_aggregate).
• Release-Lag Bracket: The instantaneous gap bracketed by the two closed-frontier releases flanking the open frontier's current level — under-estimate, over-estimate, and an interpolated central estimate (statistics.current_lag_bracket).
• Trend Gap & Catch-Up Projection: Per-group frontier trend lines yield a backward-looking trend gap, its velocity (months of gap gained/lost per year), and a clearly labeled forward-looking catch-up projection, each with bootstrap 90% CIs (trend_gap).
• Months Behind (default Gap Over Time view): The day-by-day "months behind" series that the headline average is the mean of, published per benchmark and per framing under gap_timeline as compact linear segments plus drop events, the leader/laggard frontier release lists, and a context block: the current (matched) lead and what drives it, the last time the lead was at least / at most this large ("the last time the US lead over China was this large was May 2026, 3.6 months ago"), record-high/low flags, peak, trough, median, the current value's percentile, and trailing-12-month averages. Each segment also carries the lower bound of the measured range (months since the oldest leader model the laggard has not matched), and the block carries an expected lead series: the survival-analysis current-gap estimate recomputed as of each week, a forecast that counts unmatched leader models by their age.
• Score Gap view: Best leader score minus best laggard score on each date, in the benchmark's own units (score_gap_timeline: constant steps plus events and the same context block, e.g. "the US frontier leads by 11.6 ECI points; the last time the score gap was this large was December 2024"). It rises the day the leader releases an unmatched model, which the time-based views cannot. The chart can also be toggled to the per-release "time to match" view.
• Live Data: Fetches the latest scores daily from Epoch AI.

Methodology Notes

Model releases are discrete, so "the gap" silently picks an estimator; this project publishes three complementary ones in data.json (estimator family follows the July 2026 analysis "Have Chinese AI Models Caught Up to the US Frontier?"):

1. Matching-based (day-by-day): months since the most recent closed SOTA model the best open model has plausibly caught up to, using Epoch's paired-bootstrap / CI criterion. This drives the headline average, and the full daily series is what the Gap Over Time chart's "Gap by date" view draws (gap_timeline; see scripts/gap_timeline.py). Two properties of the series are worth knowing: it drops to zero on the day the leader releases a model the laggard's best had already matched, and it can (rarely) step up when a new laggard record with a higher point estimate fails the significance test against a model the previous record had passed. The series runs to the build date, so for benchmarks Epoch has not scored recently the tail after last_model_date reflects missing scores rather than measured progress.
2. Release-lag bracket: the current gap bracketed by the closed frontier releases just below (over-estimate) and just above (under-estimate) the open frontier's level, with a central estimate interpolated between them.
3. Trend-regression gap: the horizontal offset between per-group frontier trend lines, plus the gap's velocity (shrinking or growing) and a forward-looking catch-up projection. Backward-looking numbers are measurements; the projection is a trend-conditional forecast and is labeled as such.

Bounded 0–100 benchmark scores are fitted and interpolated in logit space (raw-space fits compress progress near the floor and ceiling); METR time horizons use log space; ECI is unbounded and fitted linearly. The "90% Range (Daily Gap)" statistic is the 5th–95th percentile spread of the day-by-day gap series — dispersion, not a confidence interval of the mean.

Setup

Prerequisites

• Python 3.10+
• pip

Installation

1. Clone the repository:

   git clone https://github.com/TheoBearman/Open-Source-Gap.git
   cd Open-Source-Gap

2. Install dependencies:

   pip install -r requirements.txt

Running Locally

To run the Flask application locally:

python app.py

Open http://localhost:8080 in your browser.

Deployment

GitHub Pages (Automated)

This project is configured to deploy automatically to GitHub Pages.

• How it works: A GitHub Actions workflow (.github/workflows/deploy.yml) runs daily. It fetches the latest data, builds a static version of the site, and deploys it.
• Static Build: The build_static.py script generates a static data.json file so the frontend works without a backend server.

To enable:

1. Go to your repository Settings > Pages.
2. Under Build and deployment, set Source to GitHub Actions.

Project Structure

• app.py: Flask backend and core logic for gap calculation.
• static/: CSS and JavaScript files for the frontend.
• templates/: HTML templates.
• build_static.py: Script to generate static files for GitHub Pages.
• .github/workflows/: CI/CD configuration for automated deployment.

Data Source & Attribution

Data is sourced from Epoch AI.

Epoch AI’s data is free to use, distribute, and reproduce provided the source and authors are credited under the Creative Commons Attribution license.

Attribution: Data is provided by Epoch AI and is licensed under the Creative Commons Attribution 4.0 International License.

If you use this data or visualization, please credit Epoch AI as the source.