Skip to content

new: [lists] NetAPI Top 1M warninglists with a keyless generator - #463

Merged
adulau merged 1 commit into
MISP:mainfrom
slavaolesik:netapi-top1m
Oct 29, 2026
Merged

adulau merged 1 commit into
MISP:mainfrom
slavaolesik:netapi-top1m

Conversation

@slavaolesik

Copy link
Copy Markdown
Contributor

Adds three warninglists built from the NetAPI Top 1M, a free daily ranking of the 1,000,000 most popular websites as registrable domains (CC BY 4.0, https://netapi.com/top-1-mln-websites/), next to the existing Tranco, Cloudflare Radar and Chrome CrUX lists:

  • lists/netapi-top1m (1,000,000 entries), lists/netapi-top10k and lists/netapi-top1k
  • tools/generate-netapi-top1m.py, wired into generate_all.sh right after generate_tranco.py
  • three README rows in alphabetical order

The generator follows generate_tranco.py: type hostname, matching attributes hostname / domain / url / domain|ip, version from get_version(). The source is one plain CSV (NetAPI_Rank,Domain, 1,000,000 rows, rebuilt every day, Last-Modified header set), so download_to_file() skips the download when nothing has changed. No API key or sign-up is needed, unlike the Cloudflare Radar generator.

Why another top list: generate_alexa.py is marked BROKEN and the Alexa list is frozen since 2023, the Cloudflare generator needs a token and is commented out, so Tranco is the only keyless daily top-1M generator left in generate_all.sh. A second independent ranking helps with the usual warninglist question "is this a well-known site", and the 10K and 1K slices give the stricter variants that tranco10k and cisco_top1000 provide today.

NetAPI is already part of the MISP ecosystem: misp-modules ships the netapi expansion module (PR 1014, merged on 27 October), which looks up the Top 1M rank and the compromised feed listing of a domain. These lists come from the same data.

The three list files pass schema.json and tools/validate_values.py, are formatted like jq -S ., carry no executable flag, and all 1,000,000 entries satisfy the hostname rule of validate_values.py.

Thanks!

Slava

@adulau
adulau merged commit 7ac2dec into MISP:main Oct 29, 2026
5 checks passed
@adulau

adulau commented Oct 29, 2026

Copy link
Copy Markdown
Member

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants