Router: record the Python router's Python-specific behavior as language-neutral fixtures for the Go port #10306

Part of github/gh-aw-firewall#10261
(step 1 of 7). No prerequisite changes are needed; the Python router must still exist.

Summary

Before the router is ported to Go, record the behavior that comes from Python itself, rather
than from the router's contract, as small language-neutral fixtures. The Go port replays them as
unit tests and keeps them after Python is removed. The fixtures cover four functions:

• the heuristic labels;
• request decoding;
• the context estimate;
• the classifier prompt.

They also record the Unicode data those functions depend on.

This PR adds fixtures, a generator and a Python replay test. It doesn't change router behavior.

Background

#10261 asks for "differential coverage for malformed input, Unicode, classification
prompts, routing ties, context limits, explicit/omitted efforts, automatic modes, and
no-route cases". Confidence in the port comes from three places, and only the first is this
step:

1. Fixtures for facts that disappear with Python (this step). Some behavior is defined by
   Python, pydantic or the re module, not by the router's code or contract. Once Python is
   gone, nothing else records it, so it has to be written down first.
2. Go unit and integration tests (steps 2 and 3). Routing, classifier choices, table
   validation, the CLI and HTTP are the router's own logic, specified by its code, tests and
   openapi.yaml. The Go port ports the Python tests (test_routing.py, test_service.py,
   test_classification.py, test_table_schema.py, test_cli.py, test_http.py) as Go
   table-driven unit tests and integration tests. These stay after Python is removed.
3. A temporary differential job (steps 2 and 3). CI runs the Python and Go routers on the
   same inputs and fails on any unapproved difference. Its inputs come from the
   routing-contract corpus, from these fixtures, and from seeded generated inputs. It is
   removed with Python.

The existing corpus in containers/router/tests/fixtures/routing-contract/ was written for API
consumers: a few reviewed cases per endpoint, comparing only status and code for errors. It
doesn't record Python-specific behavior that a port can quietly change.

The observations below are recorded in the fixtures. They came from gh-aw-router 0.1.3 at
0e06f1e2, Python 3.12.13 and pydantic 2.13.5 / pydantic-core 2.46.5. The locked reference, not
these examples, determines the baseline: --check fails if a dependency update changes any
recorded output.

• Unicode version. Python 3.12's str methods and re use Unicode 15.0.0, but Go
  1.27's unicode package uses 17.0.0.

• Whitespace. str.split() and str.strip() treat U+001C–U+001F, U+0085, U+00A0 and
  other Unicode spaces as whitespace, while U+200B is not whitespace.
  heuristics.py normalizes text by casefolding it, splitting on whitespace and
  rejoining it with single spaces.

• Heuristic patterns. They use Unicode \w inside lookbehind and lookahead
  (heuristics.py lines 12 and 16). Go's RE2 supports neither lookaround. When a cue
  alternative matches but fails the lookahead, Python goes back and tries the later
  alternatives.

• Case folding. str.casefold() uses full case folding: ß becomes ss and the fi
  ligature becomes fi.

• Code points. Lengths, the 400-code-point limit for prior classification turns, and
  the 500-code-point limit for error details all count code points, not bytes or UTF-16
  units.

• Context estimate. A tool call's size is len(name) plus the length of its input
  serialized as Python's compact JSON with ensure_ascii=False. That output uses Python's
  float repr:

  • 1e16 → 1e+16
  • 1.5e-07 stays as is
  • -0.0 stays as is
  • NaN and Infinity stay as is, and 1e400 becomes Infinity
  • 1E5 → 100000.0
  • integer -0 → 0

  String escaping is also Python's: ", \, \n, \r, \t, \b and \f are escaped,
  other control characters become \u00XX, and DEL and non-ASCII characters are left as
  they are.

• JSON parsing (pydantic's parser):

  • Duplicate keys at any depth: the last value wins.
  • Accepted inside the free-form tool_call.input: NaN, Infinity and 1e400.
  • Accepted: integers up to 4,300 digits, and trailing whitespace.
  • Rejected as malformed JSON (HTTP 400 invalid_json):
    • a BOM;
    • trailing data;
    • invalid UTF-8;
    • lone surrogate escapes;
    • integers over 4,300 digits (a minus sign counts towards the limit);
    • nesting past a fixed depth. At tool_call.input, which is already six levels deep,
      195 nested empty arrays are accepted and 196 are rejected.

• Strict validation.

  • Rejected as invalid (HTTP 422 invalid_json): wrong-case keys, null for strings and
    arrays, and 1.0 or true for an integer field.
  • Accepted: very large integers for context_window, and a model id with a trailing
    newline.

Scope

1. A new fixture directory, containers/router/tests/fixtures/parity/ (about 380 KB):
   • labels.json, decoding.json, context.json and classification.json: one case
     document per function;
   • unicode/: character data generated from Python;
   • README.md, the only hand-written file: the case format, how to regenerate, the
     reference versions, where everything that isn't in the fixtures is tested, the decisions
     for the Go port, and what happens to each part after the Go image cut-over.
2. Python test modules in containers/router/tests/:
   • parity_cases.py builds every case's inputs;
   • parity_corpus.py is the generator, with --write and --check modes;
   • test_parity_corpus.py replays the committed fixtures against Python.
3. CI: run --check before the tests in .github/workflows/test-router.yml, on both
   operating systems in the checks matrix.
4. A short "Parity corpus" section in the router README.

Specification

Case format

The fixture README is the full specification; it must be usable without reading the Python
code. Files are UTF-8 with LF line endings, 2-space indentation and sorted keys, and non-ASCII
characters are written as \u escapes. Each document has a kind (labels, decoding,
context or classification), a description and cases. Each case has a unique id, a
description, its inputs and an expect object. A JSON-valued input is passed to the function
as its JSON text; inputs whose exact spelling matters (NaN, duplicate keys, escapes) use a text
field instead.

The classification document also records the fixed system_prompt and
classification_contract text, so the Go port can embed them verbatim.

Required coverage

• Labels.
  • Every cue in every heuristic cue list, in a frame where no earlier rule can fire first.
  • Cue boundaries: at the start, middle and end of the text, and next to _, digits,
    punctuation and non-ASCII letters.
  • Cue alternatives where one cue is a prefix of another, so the first alternative fails the
    boundary check and a later one succeeds.
  • Rule precedence, repeated polite prefixes (_POLITE_PREFIX) and the two-word imperative
    gate (len(action.split()) >= 2).
  • Unicode: each Python-only whitespace code point and U+200B, which isn't whitespace;
    letters and digits added in Unicode 16 or 17, which Go's unicode would class
    differently; combining marks and non-ASCII letters next to a cue; full case folding; and
    astral characters.
• Decoding.
  • Each JSON parsing and strict-validation item in the background list above.
  • Number and literal spellings inside tool_call.input, and JSON syntax that Python's
    parser accepts or rejects (comments, single quotes, trailing commas, control characters,
    escapes, surrogates, UTF-8 encodings).
  • The exact nesting-depth and digit-count boundaries on both sides. The generator finds
    them by binary search and pins them.
  • context_window, model id, choice id and effort spellings that the schema accepts or
    rejects, including a context_window larger than 64 bits.
• Context. Each part of the estimate (text, tool-call names, tool-call inputs and tool
  results), with non-ASCII and astral text, floats, big integers, NaN, Infinity, float
  overflow and underflow, string escapes, nested containers and duplicate keys.
• Classification prompts.
  • Only &, < and > are escaped; quotes are left as they are.
  • A prior turn of exactly 400 code points and one of 401, an astral character at the
    limit, stripping before truncation and escaping after it.
  • More prior turns than the limit, including how turns are numbered after older ones are
    dropped.
  • Which messages are authored turns: tool-only messages, whitespace-only text (Python
    whitespace), U+200B, and non-user roles.
  • Messages with several text parts, (none) when there are no prior turns, and the
    current request stripped but not truncated.

Unicode data

• Generate the data from the running interpreter, and record python_version and
  unicode_version (unicodedata.unidata_version) in the files. The generator refuses to
  run unless unicode_version is 15.0.0.
• Cover every code point from U+0000 to U+10FFFF except surrogates, stored as sorted
  code-point ranges:
  • whitespace: str.isspace(). The same set drives split(), strip() and \s.
  • word: Python re's \w for str patterns. Check it against a real re match
    for every code point.
  • printable: str.isprintable(). It's used to sanitize error details.
• Add a casefold mapping for the code points whose casefold() result differs from the
  code point itself.

Generator and checks

• --write. Runs each case's inputs through the Python function and records expect.
  Builders can assert facts before recording, for example that a decoding case is accepted.
  The generator deletes files it no longer produces, except README.md.
• --check. Regenerates everything in memory and fails, naming each changed case id,
  when a committed file differs. It also validates each document's structure: unknown keys,
  missing inputs, duplicate ids and the shape of expect. It runs deterministically without
  network access in a few seconds, and it must pass, byte for byte, on Linux and Windows.
• The replay test. test_parity_corpus.py loads every committed case and checks that the
  Python function still returns expect. It adds about 630 fast tests to the router's suite.

What the fixtures don't cover

The fixture README lists where everything else is tested, so the reduced scope is explicit:

Decisions for the Go port (maintainers decide)

The fixture README lists the places where the Go router deliberately behaves differently from
Python. Merging this PR approves them. Steps 2 and 3 implement them with tests, and the
differential job allows exactly these differences. If maintainers reject one, remove it from
the README before merging, and Go keeps Python's behavior.

• Error responses. Compare only status and code. detail is diagnostic text, as the
  router README already states for the contract corpus.
• Trailing slashes. /healthz/ and /route/ return 404 not_found, not a 307
  redirect, matching the openapi.yaml rule that unknown paths return 404.
• Unsupported transfer codings. Go's HTTP/1.1 server rejects a Transfer-Encoding
  other than plain chunked with a bare 501, as RFC 9112 recommends, before any handler
  runs; Uvicorn uses a bare 400. Both reject the request, and changing Go's status
  would mean replacing its parser.
• Default tables path. With no option or environment variable, Python falls back to
  the routing directory next to its source. Go uses ./routing in the working
  directory if it exists, and otherwise requires the option. The image always sets
  GH_AW_ROUTER_ROUTING_TABLES, so this affects only local runs.
• Option abbreviations. argparse accepts unambiguous prefixes such as --routing-t; Go
  accepts full option names only.
• CLI newlines. CLI stdout uses LF on every platform; Python writes CRLF to a Windows
  pipe.
• Bind ports. A bind port is ASCII decimal digits only. Python's int() also accepts
  +80, 8_737, surrounding whitespace and non-ASCII digits.
• Log levels. An invalid GH_AW_ROUTER_LOG_LEVEL value is a usage error (exit 2).
  Python doesn't validate the env default today.
• serve stderr (required, not a proposal). Structured JSON replaces plain-text
  startup and configuration errors, as the parent issue requires. Exit statuses and CLI
  stdout don't change.

Everything else in the background list, including NaN acceptance and the depth and digit
limits, is behavior the port must keep.

After the Go image cut-over

Write the README so these rules are already clear before the cut-over.

Out of scope

• Any Go code; that belongs to the subsequent core/CLI and HTTP-server ports.
• CLI, HTTP, routing and table-loading fixtures. The Go port tests those directly.
• Changing the Python router's behavior.
• Changing the routing-contract fixtures, or the api-proxy's copy in
  containers/api-proxy/fixtures/routing-contract/.

Acceptance criteria

• tests/fixtures/parity/ exists, and its README documents the format, how to
  regenerate the fixtures, the reference versions, where the areas the fixtures don't
  cover are tested, the decisions for the Go port, and which parts are kept or deleted
  after the Go image cut-over.
• Every coverage group listed above has cases.
• The Unicode data is generated from Python with Unicode 15.0.0, and the generator
  rejects any other version.
• parity_corpus.py --check passes in CI on ubuntu-latest and windows-latest,
  and fails, naming the case, when a recorded output changes.
• The Python replay test passes on both operating systems.
• No production code under src/gh_aw_router/ changes.

Validation

In containers/router:

• uv run --locked python tests/parity_corpus.py --check
• uv run --locked python -m pytest
• The existing ruff and ty checks.
• Change one recorded case by hand, and confirm that --check names it and the replay
  fails.

Files likely touched

• containers/router/tests/fixtures/parity/** (new)
• containers/router/tests/parity_cases.py (new)
• containers/router/tests/parity_corpus.py (new)
• containers/router/tests/test_parity_corpus.py (new)
• containers/router/README.md (a new "Parity corpus" section, and --check in the
  development checks)
• .github/workflows/test-router.yml (a --check step in the checks job)