[elhoim]

BLUF

• What: six domain lists (microsoft, microsoft-win10-connection-endpoints, microsoft-win11-connection-endpoints, microsoft-office365, rfc6761, public-ipfs-gateways) switch from "type": "string" to "type": "hostname". Their generators are updated so the next scheduled run keeps the new type.
• Why: under string, leading-dot entries (250 of 322 in microsoft) never match. Entries cut down from *.foo.com only match the apex, and url/uri attributes never match at all. In practice the Microsoft lists are dead today.
• Safety: hostname matches subdomains, so entries that would match third-party infrastructure are dropped. These are public suffixes (azurewebsites.net, blob.core.windows.net, …), shared CDN zones (g.akamai.net, hwcdn.net, …) and 22 certificate-authority endpoints. They were dead under string too, so no working match is lost.
• Not tested against live feeds: the generators were run against mocked Microsoft/IANA feeds only.
• Priority: medium. The fix brings back false-positive suppression on lists that silently do nothing today. Nothing breaks in the meantime.

Why

Six domain lists use "type": "string" but are written for hostname matching. In MISP core (app/Model/Warninglist.php):

• string: exact, case-sensitive lookup. Entries are loaded as they are (getFilteredEntries()), with no trimming.
• hostname: entries are loaded via strtolower(trim($v, '.')). __evalHostname() takes the host part of the value (so URLs work) and matches it or any parent domain of it.

So under string, entries with a leading dot (.aadrm.com) never match anything. Entries reduced from *.foo.com to foo.com only match the apex. And url/uri attributes never match a bare host name.

tools/generate-google.py already notes that microsoft and microsoft-win10-connection-endpoints "appear to already be non-functional" for this reason.

What changes

List	Problem with string	Entries
microsoft	250 of 322 entries had a leading dot, so they never matched	322 → 294
microsoft-win10-connection-endpoints	31 leading-dot entries never matched	144 → 123
microsoft-win11-connection-endpoints	18 leading-dot entries never matched (the generator turned *.foo into .foo)	152 → 148
microsoft-office365	The generator turns Microsoft's *.foo.com into foo.com, so subdomains were lost	183 → 161
rfc6761	RFC 6761 reserves every name under test, localhost, invalid, example and *.in-addr.arpa	25 (unchanged)
public-ipfs-gateways	Has url/uri in matching_attributes, which can't match a bare host. Subdomain gateways (<cid>.ipfs.dweb.link) didn't match either	100 (unchanged)

Entries are stored bare (no leading dot), lowercased and deduplicated, following the google/tranco precedent.

Entries removed

hostname also matches subdomains. That makes two kinds of entry either harmful or useless:

• Public suffixes, checked against the committed second-level-tlds list (the PSL). These are dropped from the three Microsoft lists:

  • azurewebsites.net, cloudapp.net, azureedge.net, azure-api.net, azure-mobile.net, akadns.net (microsoft)
  • akamai.net, akamaiedge.net, blob.core.windows.net (win10)
  • blob.core.windows.net (win11)

  As hostname entries they would match names registered by any Azure or Akamai customer, including attacker infrastructure. Under string they were dead entries anyway, so nothing that works today is lost.

  public-ipfs-gateways keeps its public-suffix entries (dweb.link, cf-ipfs.com, cloudflare-ipfs.com) on purpose. Its category is known, and "this is served through a public IPFS gateway" is exactly what it should say for <cid>.ipfs.dweb.link.

• Shared CDN zones in win10 (second commit): g.akamaiedge.net, b.akamaiedge.net, dscb1.akamaiedge.net, dspb.akamaiedge.net, dspg.akamaiedge.net, g.akamai.net, dscd.akamai.net, dspw65.akamai.net, hwcdn.net, wac.edgecastcdn.net, wac.phicdn.net.

  They aren't on the PSL, but they serve many non-Microsoft customers, and hwcdn.net customer subdomains can serve arbitrary content. As suffix matches they would suppress warnings far beyond Windows 10 endpoints. Specific hosts on those CDNs are kept, e.g. a122.dscd.akamai.net and the Microsoft-named *.akadns.net, *.nsatc.net and *.footprint.net CNAMEs.

• Certificate authorities in microsoft-office365 (third commit): 22 CRL/OCSP/AIA endpoints Microsoft publishes, e.g. *.verisign.com, *.entrust.net, *.geotrust.com, crl3.digicert.com, ocsp.globalsign.com and cert.int-x3.letsencrypt.org.

  As suffix matches they would cover these CAs entirely, for every vendor using them. Microsoft's own endpoints (crl.microsoft.com, mscrl.microsoft.com, oneocsp.microsoft.com, ocsp.msocsp.com) are kept. The CA domains generate-microsoft.py already excluded now live in generator.py as CERTIFICATE_AUTHORITY_DOMAINS, and both generators filter on them.

• * inside a name: .autodiscover.*.onmicrosoft.com|us, .autodiscover.*.mail.onmicrosoft.com|us and watson.*.microsoft.com. No list type supports * there, so these never matched. The .com ones are covered by onmicrosoft.com.

Generators

These are updated so the next scheduled run keeps the new type and shape:

• generate-microsoft.py: stores bare entries and skips wildcard/public-suffix candidates. Committed entries are normalised the same way. The CA exclusions now come from the shared CERTIFICATE_AUTHORITY_DOMAINS.
• generate-microsoft-win11-endpoints.py: *.foo becomes foo. Remaining * and public suffixes are dropped. Writes hostname.
• generate-office365.py: writes hostname. A wildcard now drops every label up to and including the label holding the *, so *-admin.foo.com gives foo.com instead of admin.foo.com, which wouldn't match tenant names. Public suffixes and certificate-authority domains are filtered.
• generate-public-ipfs-gateways.py: writes hostname.
• generate-rfc6761.py: preserves the committed type; docstring updated.
• generator.py: new get_public_suffixes() helper and CERTIFICATE_AUTHORITY_DOMAINS.
• generate-google.py: removed the now-stale remark about the two Microsoft lists.

win10 has no generator; it was edited directly.

Before / after

Computed with a Python port of Warninglist::checkValue() for string/hostname:

microsoft      aadrm.com                         before=False  after=aadrm.com
microsoft      contoso.onmicrosoft.com           before=False  after=onmicrosoft.com
microsoft      https://login.microsoftonline.com/common  before=False  after=microsoftonline.com
microsoft      evil.azurewebsites.net            before=False  after=False
win10          windowsupdate.com                 before=False  after=windowsupdate.com
win11          eu.smartscreen-prod.microsoft.com before=False  after=smartscreen-prod.microsoft.com
win11          evil.blob.core.windows.net        before=False  after=False
office365      contoso.sharepoint.com            before=False  after=sharepoint.com
rfc6761        app.localhost                     before=False  after=localhost
rfc6761        1.0.0.10.in-addr.arpa             before=False  after=10.in-addr.arpa
public-ipfs    https://gateway.ipfs.io/ipfs/bafy… before=False after=ipfs.io

Points for review

• Untested against live feeds. The Microsoft and IANA endpoints weren't reachable from my environment. The generators were run against mocked feeds instead: generate-microsoft.py is idempotent on the converted list, and the wildcard and public-suffix cases are skipped or reduced as intended.

Checks

• jsonschema passes on all six lists.
• jq -S . formatting check passes.
• tools/validate_values.py passes.

🤖 Generated with Claude Code

[adulau]

Thank you!