Skip to content

ADR 0004 phase 8: unify catalogue maintenance, drift review, and feed publication #10298

Open

Description

@lpcox

Objective and dependencies

Implement ADR 0004 Gate 8. Parent: #10290.

Scheduled source maintenance is required, not optional follow-up. The model-management system must not rely on a maintainer remembering to discover model changes, edit sources, or regenerate outputs.

Phase 3 (#10293) merged in #10317, providing the canonical sources/import/compiler foundation. Source refresh and update-PR automation can proceed against it. Central feed publication remains separately gated on Phase 5 loader/source-authentication integration; publication is not a prerequisite for scheduled checked-in source upkeep.

Delivered maintenance slice (2026-10-22)

  • feat(ci): automate scheduled canonical source maintenance #10439 replaced the old manual/read-only endpoint updater with daily scheduled and main-only manual source maintenance, 11 approved evidence captures, scoped proposals, governed import refresh, deterministic regeneration, drift/impact reports, deduplicated review PRs, and independent evidence/bundle validation before privileged writes.
  • fix(ci): unblock unattended model source maintenance #10466 fixed unattended Copilot command-permission matching using a narrowly permitted executable with seven fixed operations; no blanket node/npm permission was added.
  • Run 39849323674 passed every job and created chore: refresh canonical model sources #10469, merged on 2026-10-22. All 11 sources were captured successfully; this proposal changed the governed OpenRouter import and all affected projections, not maintained native-provider/Copilot records. The committed report records 2,957 catalogue field changes and 663 runtime lookup changes.
  • Offline fixtures cover update/no-op/deduplication, failed/partial retrieval, evidence receipts, source disappearance without inferred retirement, high-impact review gates and actual bundle ownership. No-op behavior and updates to an existing PR have fixture coverage; the successful run above exercised creation of a new PR, not every branch.

This phase remains open for central snapshot publication and Phase 5 loader/source-authentication integration. Do not rebuild the delivered updater. Successful documentation capture is not semantic verification of all maintained facts; human evidence/scope review remains mandatory.

Scope

Replace endpoint-only and ad hoc maintenance with one source-aware scheduled updater using existing GitHub Actions/gh-aw mechanisms:

  1. Run daily, with workflow_dispatch for on-demand reruns.
  2. Fetch approved provider model APIs/documentation and governed upstream snapshots, retaining source/revision/capture/hash/license evidence as applicable. Validate retrieval failures explicitly; a missing page must not be interpreted as model retirement.
  3. Compare captured evidence with canonical source records and propose supported source changes. Preserve provider/channel/deployment boundaries, authority, confidence, and unknown states; no family-name or marketing-based compatibility inference.
  4. Run canonical validation/import checks, npm run models:generate, and focused model regression checks, then npm run models:check.
  5. Create or update one deduplicated review PR containing source changes, governed import metadata, every affected generated projection, and machine/reviewer-readable impact reports. Never author generated outputs independently.
  6. No material change means a no-op, not timestamp-only churn. Retrieval/validation failures surface visibly with affected source names and last successful evidence.

Automate discovery, proposal, validation and regeneration; retain human review before merging model-fact changes. Do not auto-merge or automatically activate staged pricing, lifecycle, compatibility, or model-selection corrections. The bundled catalogue and generated retirement projection already have legacy runtime consumers: merging a reviewed refresh can change fallback accounting, lookup winners or retirement rejection, and reports/consumer regressions must expose those effects.

Scheduled drift comparison covers missing models, prices beyond documented provider-specific rounding tolerance, changed effort sets/limits, lifecycle, compatibility, harness constraints, and source health.

Publish reviewed versioned central snapshots through the same canonical pipeline only after Phase 5 integration is ready. Static package outputs remain reproducible and downloaded snapshots never rewrite package assets. A content digest is not publisher authentication.

Entry points: .github/workflows/model-api-mapping-updater.md and lock, canonical compiler/importer from Phase 3, CI/staleness checks, and existing release/publication machinery. Follow workflow-source compilation/post-processing requirements. Use existing execution/sandbox controls; no new kernel/controller/runtime-hardening architecture.

Acceptance criteria

  • A checked-in recurring schedule triggers approved source retrieval and update proposal without a maintainer editing sources or remembering regeneration; manual reruns remain available. feat(ci): automate scheduled canonical source maintenance #10439/fix(ci): unblock unattended model source maintenance #10466; successful end-to-end run linked above.
  • One deduplicated update PR contains canonical changes, governed imports/metadata, every affected projection, and a machine-readable impact report; generated outputs are never independent authoring targets. feat(ci): automate scheduled canonical source maintenance #10439; first merged output chore: refresh canonical model sources #10469.
  • Reports cover adds/removals, lifecycle/channel/replacements, prices/thresholds/effective dates/promotion expiry, effort/limit/feature/harness changes, confidence, stale/missing sources, collisions/rejections. Generic exact-field diffs and review classifications in scripts/model-information/maintain.cjs; chore: refresh canonical model sources #10469 includes actual runtime lookup effects.
  • Tests enforce explicit review for price decreases, retirement, compatibility contraction, confidence downgrade, and source disappearance. scripts/model-information/maintain.test.ts.
  • Recurring drift fixtures detect missing models, tolerated versus material price changes, and effort/context drift without live external dependencies. Decimal-equivalent formatting is a no-op; tolerance is zero unless separately reviewed provider-specific tolerance is documented, not a guessed allowance.
  • Unchanged upstream facts produce a no-op without timestamp-only catalogue changes or duplicate PRs. Offline no-op and existing-PR fixtures in feat(ci): automate scheduled canonical source maintenance #10439.
  • Failed retrieval, invalid data, and partial source coverage are visible; they do not silently erase existing facts or count as a successful refresh. Named capture failures and independent validation in feat(ci): automate scheduled canonical source maintenance #10439.
  • Runtime observations only propose maintained changes; they cannot silently promote uncertain facts or make advertised discounts billable. Marketing prose does not establish verified capability. Evidence-bound data proposals require human review; no new runtime-observation promotion mechanism was enabled.
  • Governed imports capture source/revision/time/hash/importer/license/counts/collisions; malformed/ambiguous updates fail visibly. feat(ci): automate scheduled canonical source maintenance #10439/chore: refresh canonical model sources #10469; undeclared OpenRouter licenses remain explicitly unknown.
  • Published snapshots carry schema/revision/publication/effective/provenance/integrity metadata and pass Phase 5 loader/source-authentication integration tests before publication is enabled.
  • Obsolete endpoint-only authoring is replaced; source ownership, scheduled upkeep and review boundaries are documented in the ADR. feat(ci): automate scheduled canonical source maintenance #10439.
  • Document the central publication/release procedure and verify published-snapshot consumers after Phase 5 integration; maintenance documentation does not establish publication readiness.

Non-goals

Do not auto-accept model-fact changes, bypass network controls, publish unreviewed provider observations, or trigger live workflows merely to validate this implementation. No new execution-security architecture or /reflect/v2. Agent workflows consume packaged model data; this scheduled maintenance process does not turn the offline compiler into a runtime network fetcher.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions