Skip to content

Routing bake 4/4: work tables in files, and sorts on disk #3241

Closed

Description

@timohueser

Status: done

Part of #3228. Depends on #3240. The words of #3228 apply.

TL;DR

Goal: The large tables of the routing bake live in files that the bake reads through memory maps, and the large sorts run on disk. The bake's own memory then stays at or below 4 GiB for every region up to a continent, in every stage except the route catalog, and the rest of the RAM caches the files.
Problem: After #3238–#3240, the tables are flat but still in RAM: about 14 GB for Germany, and about 45–70 GB for Europe (estimated).
Decision: Column and Lists store their values in files in a work folder. One merge sort of fixed-width records sorts on disk within the budget.
Systems: routing
Contracts: none
Size: M (one system, about 600 changed lines)

Plan

  1. column.rs: Column and Lists store their values in files in the work folder and read them through memmap2. A column holds only fixed-width plain values: integers and fixed-width records. A sealed trait names these types. Variable-length data (tags, roles) is already Lists<u8> (Routing bake 1/4: one compact copy of the OSM source data #3238).
  2. The work folder is OBC_DATA_WORK when it is set. Else it is the temporary folder of the step (TMPDIR), which obc data already puts beside the output, on the disk of the store (owner decision). The bake removes its work files at the end, also after an error.
  3. sort.rs (new): a merge sort of fixed-width records on disk. It sorts runs within the budget in RAM, writes each run to the work folder, then merges the runs. The node references of the import, the road order of spatial_order, the spatial-index pairs and the edges use it.
  4. The memory budget is at most 4 GiB, and at most OBC_DATA_MEMORY_BYTES when the run sets it. The sort runs take their size from it.
  5. planner/router-build/README.md: the work folder, OBC_DATA_WORK, the memory budget, and the disk need (about 60–80 GB for Europe, estimated).
  6. New tests: a Column in a file reads back what it took; a sort over several runs gives the order of a sort in RAM.

Done when

  • obc test -p planner-router-build and clippy pass.
  • A Switzerland bake gives the same manifest ID as before.
  • A Germany bake with terrain and all 17 profiles runs in a cloud container with 15 GB of RAM. Its own memory (anonymous resident memory, not file cache) stays at or below 4 GiB in every stage except the route catalog, which needs about 5–7 GB until Routing: hierarchical search and disk storage #3227 (estimated). The pull request records the peak of each stage and the time.

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions