Reduce non-intersecting map archive downloads

[FrogAi]

Summary

• Add compact 2-degree archive ranges so default country and state downloads avoid archive groups that do not intersect their selected boundary geometry.
• Preserve the existing bounding-box behavior for custom menus and entries whose generated archive set already matches their bounding box.
• Reduce archive requests across the 229 default regions from 17,238 to 8,613 (50.03%) and their summed server-reported Content-Length from 65.41 GB to 43.54 GB (33.44%).

Root cause

Map downloads currently round each region's rectangular bounding box outward to the 2-degree archive grid, then request every archive in that rectangle. Large or irregular regions such as Alaska and Canada therefore request many archive groups that do not intersect the selected region.

The archives already contain road data rather than terrain or ocean layers. The unnecessary cost comes from selecting irrelevant archive groups, so this change narrows the requested coordinates without changing the contents of any individual archive.

Change

• Add an optional archive_ranges field to download-menu entries. Each row is [latitude, inclusive minimum longitude, exclusive maximum longitude] for one 2-degree latitude band.
• Use the same range iterator for progress totals and archive downloads. Missing or empty ranges are derived from the existing bounding box.
• Load the download menu once per download instead of re-reading it while resolving each path.
• Add scripts/update_download_regions.py to check or regenerate the ranges from pinned Natural Earth country boundaries and Census TIGER/Line state boundaries.
• Keep the existing menu authoritative for catalog entries and detached-territory scope instead of automatically making geopolitical policy decisions.
• Document the range format, precedence, fallback behavior, and regeneration commands.

Download impact

Metric	Base 7201c6b4	Candidate eb23a2ab	Difference
Region-summed archive requests	17,238	8,613	-8,625 (-50.03%)
Region-summed Content-Length	65.41 GB	43.54 GB	-21.88 GB (-33.44%)
Unique archive coordinates	9,850	6,703	-3,147 (-31.95%)
Unique-union Content-Length	23.95 GB	23.91 GB	-45.55 MB (-0.19%)
Canada archive requests	765	527	-238 (-31.11%)
Canada Content-Length	1.44 GB	504.51 MB	-931.95 MB (-64.88%)
Alaska archive requests	1,980	143	-1,837 (-92.78%)
Alaska Content-Length	2.42 GB	11.56 MB	-2.40 GB (-99.52%)

The region-summed rows reflect independently downloading each menu region, which matches the current downloader's behavior. The unique-union rows count each archive coordinate once and describe total catalog coverage.

Compatibility and scope

• The archive URL format and per-coordinate archive contents are unchanged. Only the selected coordinates change.
• Existing custom menus containing only bounding_box continue to use the previous behavior. Omitted or empty archive_ranges also fall back to the bounding box.
• The existing 229 catalog entries, names, bounding boxes, submenus, and ordering are preserved. Of those entries, 184 use explicit ranges and 45 retain bounding-box selection.
• Existing menu bounds continue to control which disconnected source components belong to an entry. New countries and newly relevant detached territories remain intentional menu changes.
• Boundary sources and Python packages are regeneration-only dependencies and are not part of the mapd runtime.
• This does not change archive extraction, cancellation, progress publication, file formats, concurrency, cross-location deduplication, retries, or installation behavior.

Engineering record and validation

Design decisions

• Add optional range rows instead of rewriting stored bounding boxes so existing custom menus and fallback behavior remain unchanged.
• Represent contiguous longitude archives as one latitude row because that matches the downloader's iteration directly without storing every coordinate.
• Use one range helper for both progress totals and downloads so the reported file count cannot drift from the requested coordinates.
• Generate the menu from pinned boundaries instead of maintaining thousands of coordinates manually.
• Use existing menu bounds to preserve established disconnected-territory scope while retaining selected components whole for ordinary boundary corrections.

Boundary-generation policy

The updater pins:

• Natural Earth 10m Admin 0 countries at commit f1890d9f152c896d250a77557a5751a93d494776, SHA-256 239eec57ac17f100a11e2536cffc56752c318b50ae765b0918ff7aab4ce8f255.
• Census TIGER/Line 2025 states, SHA-256 59a220888a8d9be8117c4fcd38f542bd02d81abf0d198c78113595ad540dd957.
• Exact Python versions for NumPy, pyshp, and Shapely through the script's inline uv metadata.

Each existing menu bounding box is rounded to the archive grid and used as the policy seed for selecting source polygon components. Selected components are retained whole so ordinary boundary corrections are not clipped by stale decimal bounds. Their intersecting 2-degree archives are then compacted into latitude rows.

Natural Earth's nonstandard country identifiers are handled explicitly: FR, NO, and TW are selected through ADM0_A3 values FRA, NOR, and TWN. mapd's historical us_state.GM entry maps to Census STUSPS=GU.

Invalid source geometry is repaired with Shapely make_valid before component selection. Antimeridian jumps are rejected instead of generating ambiguous longitude ranges.

The updater never adds or removes region definitions in the download catalog. New countries and newly relevant detached territories require an intentional menu change and review of the generated diff.

The candidate union removes 3,159 legacy coordinates and adds 12 coordinates reached by retaining selected components whole, producing a net reduction of 3,147 coordinates.

Measurement method

The live sweep ran from 2026-08-22 16:03:13 through 16:15:53 UTC. Base coordinates were derived from each menu bounding box using mapd's 2-degree floor/ceiling behavior. Candidate coordinates used non-empty archive_ranges when present and the same bounding-box fallback otherwise.

Each coordinate maps to https://map-data.pfeifer.dev/offline/{latitude}/{longitude}.tar.gz. The combined base/candidate union contained 9,862 unique archive coordinates, each mapping to one URL. The audit queried every URL exactly once with HEAD using 24 workers, a 30-second timeout, and up to four attempts with exponential backoff. A parseable positive Content-Length was required. Every URL passed on its first attempt, so no retries were used and no archive payloads were downloaded.

One coordinate-to-size map was then summed two ways:

• Region-summed results count an archive each time it belongs to a separately selected region.
• Unique-union results count each archive coordinate once.

Unrounded Content-Length totals:

• All regions: 65,410,800,451 -> 43,535,791,511
• Unique union: 23,952,699,874 -> 23,907,153,765
• Canada: 1,436,458,288 -> 504,505,102
• Alaska: 2,415,287,103 -> 11,561,464

The displayed MB and GB values use decimal units. The totals exclude HTTP/TLS overhead, retries, and extracted on-disk size.

The one-off measurement harness and per-URL manifest were not retained. Boundary and menu generation remain reproducible from pinned sources, but the historical size figures are contemporaneous evidence and cannot be independently recalculated or diagnosed per coordinate from the retained record. Because hosted archive sizes can change, rerunning this method verifies the current hosted data rather than the exact historical totals above.

Validation

Compared:

• Base: 7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee
• Candidate: eb23a2ab25557cb4aa62384878cf589a2b4512bf

The candidate changes only:

• docs/overriding-internal-defaults.md
• scripts/update_download_regions.py
• settings/download.go
• settings/download_menu.json

Exact-head checks:

• uv run scripts/update_download_regions.py --check
  • 229 regions, 184 with explicit ranges; 17238 legacy archive occurrences -> 8613 selected
• go test ./...
• go vet ./...
• docker buildx build --platform linux/arm64 --load .
• uvx ruff format --check --config "indent-width=2" --config "line-length=160" scripts/update_download_regions.py
• git diff --check 7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee...eb23a2ab25557cb4aa62384878cf589a2b4512bf

Audit-only external geometry reconstruction across all 229 entries:

• Matched every generated archive set exactly.
• Confirmed every selected archive intersects its pinned source geometry.
• Confirmed no intersecting archive within the retained component scope was omitted.
• Confirmed archive rows are aligned, ordered, non-overlapping, compact, and within world bounds.
• Confirmed names, bounding boxes, submenus, catalog entries, and ordering remain unchanged.
• Reproduced the committed menu byte-for-byte using the pinned sources and dependencies.

The exact-head Go tests and vet ran in a Linux/amd64 container. The Docker build compiled the candidate for Linux/ARM64.

No exact-head race test or on-device/end-to-end map download was performed. No permanent tests for the new selection behavior are included.

The one-off independent geometry checker was not retained. Its results are contemporaneous audit evidence rather than durable regression coverage; the committed updater's --check mode remains the reproducible check that pinned sources generate the committed menu.