Repository navigation
Conversation
MISP core matches "string" lists by exact, case-sensitive lookup with the
entries loaded as-is, while "hostname" lists have their entries dot-trimmed
and lowercased and match the value's host or any parent domain of it (also
when the value is a URL). Six lists rely on the "hostname" behaviour while
being typed "string":
- microsoft, microsoft-win10-connection-endpoints,
microsoft-win11-connection-endpoints: entries with a leading dot
(".aadrm.com"), meant as "this domain and its subdomains", never matched
anything (250 of 322 entries of microsoft). Entries are now stored bare.
- microsoft-office365: the generator reduces Microsoft's "*.foo.com" to
"foo.com", which under "string" only matched the apex.
- rfc6761: "test", "localhost", "invalid", "example" and the
in-addr.arpa zones reserve every name under them.
- public-ipfs-gateways: has url/uri in matching_attributes, which a bare
host name can never equal exactly; subdomain gateways did not match either.
Because "hostname" entries also cover their subdomains, entries that are
themselves public suffixes (azurewebsites.net, cloudapp.net, azureedge.net,
azure-api.net, azure-mobile.net, akadns.net, akamai.net, akamaiedge.net,
blob.core.windows.net) are dropped from the three Microsoft lists: they
would otherwise match names registered by any customer of those platforms.
Entries with a "*" inside a name (".autodiscover.*.onmicrosoft.com",
"watson.*.microsoft.com") are dropped, as no list type supports them; the
onmicrosoft.com ones were already covered by ".onmicrosoft.com".
Generators are updated so the next run keeps the new type and shape, and
filter wildcards and public suffixes the same way (via a shared
get_public_suffixes() helper reading the second-level-tlds list).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wC3ntBnXWYwYidQwyNq9W
Now that the list is of type "hostname", the former leading-dot entries match every subdomain. These eleven are shared CDN zones serving many non-Microsoft customers (hwcdn.net customer subdomains can serve arbitrary content), so they would suppress warnings well beyond Windows 10 endpoints: g.akamaiedge.net, b.akamaiedge.net, dscb1.akamaiedge.net, dspb.akamaiedge.net, dspg.akamaiedge.net, g.akamai.net, dscd.akamai.net, dspw65.akamai.net, hwcdn.net, wac.edgecastcdn.net, wac.phicdn.net. Specific hosts on those CDNs (a122.dscd.akamai.net, Microsoft-named *.akadns.net / *.nsatc.net / *.footprint.net CNAMEs) are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012wC3ntBnXWYwYidQwyNq9W
Microsoft publishes the CRL / OCSP / AIA endpoints of the CAs Office 365 relies on (*.verisign.com, *.entrust.net, crl3.digicert.com, ...). Now that the list is of type "hostname" they would match every subdomain of those CAs, and suppress them for every other vendor using the same CA. Drop the 22 such entries, keeping Microsoft's own (crl.microsoft.com, mscrl.microsoft.com, oneocsp.microsoft.com, ocsp.msocsp.com). The CA registrable domains generate-microsoft.py already excluded move to generator.py as CERTIFICATE_AUTHORITY_DOMAINS, and generate-office365.py now filters on them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012wC3ntBnXWYwYidQwyNq9W
cloudflare-top* (11 lists), majestic_million, moz-top500 and university_domains list registrable domains, but were of type "string": www.example.com did not match example.com, and the url/uri attributes in their matching_attributes could never match a bare domain. tranco, tranco10k and alexa, the same kind of list, already use "hostname". "hostname" entries match every subdomain, so switching as-is would turn some entries into whitelists of third-party content. A new shared helper, filter_hostname_entries() in generator.py, normalises entries the way MISP core loads them (trimmed, dot-stripped, lowercased) and drops: - junk: values in the generator's JUNK blacklist, or that are not a valid host name of at least two labels (the "domain" CSV header in every Cloudflare dataset, "web"/"ws", a broken '"com,...host"' line); - too wide: values in the generator's TOO_WIDE blacklist, public suffixes (ac.uk, github.io, blogspot.com, cloudfront.net, ...), and domains a public suffix sits under (amazonaws.com above s3.amazonaws.com, windows.net above blob.core.windows.net, kuleuven.be above ezproxy.kuleuven.be). Each generator now carries its own TOO_WIDE and JUNK blacklists. The top lists' TOO_WIDE is MULTI_TENANT_DOMAINS, a curated set of domains whose subdomains belong to customers or users but which are not on the PSL (wordpress.com, sharepoint.com, zendesk.com, googleusercontent.com, ...). The university list's TOO_WIDE holds mil.lv, listed for the National Defence Academy of Latvia. The committed lists are filtered with the same code; regenerating university_domains from upstream yields the identical list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012wC3ntBnXWYwYidQwyNq9W
elhoim
changed the title
It would still be interesting to warn on exact matches for such domains. |
Member
|
Good catch @johnmacsmith94 - @elhoim can you update the PR to support this use-case too? |
The hostname lists drop public suffixes, domains a public suffix sits under and multi-tenant domains (windows.net, blob.core.windows.net, amazonaws.com, sharepoint.com, ...), because a hostname entry would also match every customer's site under them. Those domains themselves are still benign, and an attribute that is exactly windows.net deserves a warning. split_hostname_entries() returns the dropped too-wide entries next to the kept ones, and write_exact_match_warninglist() writes them to a companion "string" list, which MISP matches exactly. The four generators that filter now write one each: cloudflare-top-exact (union over all Cloudflare Radar sizes), majestic_million-exact, moz-top500-exact and university_domains-exact. They match hostname, domain and domain|ip, not url, since a URL never equals a bare domain. The committed lists were produced by running each generator's process() on the pre-PR list data; the hostname lists come out unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BLUF
get_public_suffixes()helper. This PR's own changes are96bf98a(the switch tohostname) and07bbd05(exact-match companion lists).cloudflare-top*×11,majestic_million,moz-top500,university_domains) switch from"type": "string"to"type": "hostname". This matchestranco/alexa.string,www.example.comdoesn't matchexample.com, and a URL can never match.filter_hostname_entries()drops entries that would match other people's sites. These are TLDs (ws,web), public suffixes, domains with a PSL suffix below them (amazonaws.com,windows.net,kuleuven.be) and a curatedMULTI_TENANT_DOMAINSset (sharepoint.com,wordpress.com, …). Lists shrink by 0.05–11 %. The dropped domains move to 4 new exact-match ("type": "string") companion lists, so an attribute that is exactlywindows.netstill warns.docs.google.com/forms/…) will now get a warninglist hit. Droppingurl/urifrommatching_attributeswould avoid that.Why
cloudflare-top*(11 lists),majestic_million,moz-top500anduniversity_domainslist registrable domains, but use"type": "string". MISP core matchesstringlists exactly. As a result:www.example.comorcs.mit.edudoes not matchexample.comormit.edu.url/uriinmatching_attributescan never match, because a URL never equals a bare domain.tranco,tranco10kandalexaare the same kind of list and already usehostname.A
hostnameentry matches every subdomain, though, so switching as-is would be harmful:cloudflare-top1m/top500kcontain the TLDswebandws, which would match every.wsdomain.ac.uk,github.io,blogspot.comandcloudfront.net.amazonaws.com,sharepoint.comandwordpress.comwould cover every customer's site, malicious ones included.What changes
Shared helper:
filter_hostname_entries(entries, too_wide, junk)intools/generator.pyIt first normalises each entry the way MISP core loads a
hostnamelist: trims whitespace, strips surrounding dots and lowercases. It then drops two kinds of entry.Junk:
JUNKblacklist;Too wide:
TOO_WIDEblacklist;hostnameentry matches them anyway because matching walks up to the parent. Examples:amazonaws.comhass3.amazonaws.combelow it;windows.nethasblob.core.windows.net;fastly.nethasglobal.ssl.fastly.net;kuleuven.behasezproxy.kuleuven.be, a proxy that serves arbitrary third-party sites as subdomains.Public suffixes are read from the committed
second-level-tldslist, which is the PSL.Per-generator blacklists
TOO_WIDEJUNKgenerate-cloudflare-top-domains.pyMULTI_TENANT_DOMAINSdomain(the CSV header in every dataset) and"com,thefatherofsalmon.com.host"generate_majestic-million.pyMULTI_TENANT_DOMAINSgenerate_moz-top500.pyMULTI_TENANT_DOMAINSgenerate-university-domain-list.pymil.lv(listed for the National Defence Academy of Latvia; it would match the whole Latvian military domain)MULTI_TENANT_DOMAINSis a curated set of domains whose subdomains are handed to customers or users but which are not on the PSL. It is grouped as:wordpress.com,tumblr.com,medium.com,substack.com,weebly.com,squarespace.com,typeform.com, …sharepoint.com,atlassian.net,zendesk.com,okta.com,force.com,zoom.us,box.com, …amazonaws.com,googleusercontent.com,storage.googleapis.com,windows.net,b-cdn.net,firebaseio.com,loca.lt, …The generators also write
"type": "hostname".Exact-match companion lists
split_hostname_entries()returns the dropped too-wide entries next to the kept ones, andwrite_exact_match_warninglist()writes them to a-exactlist of typestring. The four filtering generators each write one, and the README index lists them next to their hostname twins.Data
The committed lists were filtered with the same code. Regenerating
university_domainsfrom the live upstream feed gives an identical list.cloudflare-top200cloudflare-top1kcloudflare-top2kcloudflare-top5kcloudflare-top10kcloudflare-top20kcloudflare-top50kcloudflare-top100kcloudflare-top200kcloudflare-top500kcloudflare-top1mmajestic_millionmoz-top500university_domainsFor example,
cloudflare-top200dropsdomain,amazonaws.com,cloudfront.net,akamai*.net,azureedge.net,azurefd.net,blogspot.com,googleapis.com,googleusercontent.com,sharepoint.com,windows.net,zoom.us,fastly.net,edgekey.net,edgesuite.netand a few others.university_domainsdrops 5 entries:mil.lv(inTOO_WIDE, and also on the PSL);ruhr-uni-bochum.de(on the PSL);kuleuven.be,muni.cz,rit.edu(each has a PSL suffix below it).In Moz,
marketingplatform.google....is normalised tomarketingplatform.google.Points for review
urlstays inmatching_attributes, as intranco/alexa, so a URL IOC is now matched on its host. A phishing URL ondocs.google.com/forms/…,github.com/…/releases/…orpastebin.com/raw/…will get a "top domain" hit. Before,urlnever matched these lists. If you'd rather avoid that, droppingurl/urifrom these lists'matching_attributeskeeps the domain/hostname benefit without it. I can add that.hostnamecan't express "this domain only, not its subdomains", so the generators write the dropped too-wide entries to companion"type": "string"lists, which MISP matches exactly:cloudflare-top-exact(2190 entries, the union over all Cloudflare sizes),majestic_million-exact(357),moz-top500-exact(15) anduniversity_domains-exact(5).zoom.usorwindows.netitself still gets a hit;<account>.blob.core.windows.netdoes not. They matchhostname,domainanddomain|ip, noturl, since a URL never equals a bare domain. Junk entries (domain, TLDs) are not kept.MULTI_TENANT_DOMAINSis curated, so it will never be complete. The PSL rules do most of the work, and the set only covers well-known gaps actually present in these lists. Additions are welcome.CLOUDFLARE_API_TOKEN, and the Majestic/Moz downloads weren't attempted. Theirprocess()functions were run on fake source files; the university generator was run against the real upstream feed.Checks
tests/test_filter_hostname_entries.py, including the split and the exact-match writer.python -m pytest testspasses (13 tests).jsonschemapasses on all 14 lists and the 4 new-exactlists (validate_all.shpasses).jq -S .formatting check passes.tools/validate_values.pypasses.