feat(catalog): opt-in periodic model-catalog auto-refresh

devlog/_plan/260914_r2l8_catalog_autorefresh/010_roadmap.md

@@ -0,0 +1,98 @@
+# R2-L8 — catalog auto-refresh and capability declarations
+
+Lane R2-L8 of the round-23 delivery unit. Branch `codex/260914-l8-catalog-autorefresh`,
+one pull request against `dev`. Write scope is the one the lane assignment fixed:
+`src/codex/catalog-refresh-status.ts`, `src/codex/convergence.ts`, `src/config.ts`,
+`src/types/config.ts`, `src/types/provider.ts`, `src/server/background-lifecycle.ts`
+and their tests. L8 is wave A's only config-schema owner, so the schema edits stay
+additive and self-contained.
+
+## What the two issues actually need
+
+**Periodic catalog auto-refresh (issue 3630).** A running proxy only re-discovers
+provider models when someone runs `ocx sync` or restarts. The reporter watched a
+newly released upstream model stay absent from `/v1/models` and from the on-disk
+catalog until they remembered to sync by hand. The ask is a configurable interval
+that drives the same converge path `ocx sync` drives, plus visibility when the
+served model set actually changes.
+
+**Per-model capability declarations (issue 3377).** The declaration half is already
+on `dev`: `ModelCapabilities` in `src/types/provider.ts` carries `inputModalities`,
+`contextTier` and `video.processing`; `src/config/provider-validation.ts` validates
+and merges it; `ocx provider add`/`edit` accept it. Only the text-only axis is live —
+`configuredInputModalities` in `src/codex/catalog/parsing.ts`'s neighbour
+`catalog/provider-fetch.ts` reads it, and `src/vision/` honours it. `contextTier` and
+`video.processing` are stored and inert, and both activation sites
+(`src/providers/github-copilot-transport.ts`, `src/adapters/google.ts`,
+`src/responses/schema.ts`) sit outside this lane's write scope. This lane therefore
+does not close issue 3377; it pins the part it can own.
+
+## Design
+
+### Config surface
+
+```ts
+export interface OcxCatalogAutoRefreshConfig {
+  enabled?: boolean;         // master switch, default false
+  intervalMinutes?: number;  // default 60, floor 15, 0 keeps the timer dormant
+}
+```
+
+on `OcxConfig.catalogAutoRefresh`. Opt-in rather than default-on: a refresh spends a
+live `/models` call against every enabled provider, and the repository's existing
+optional-subsystem rule is that a default install runs no detection code. The floor
+exists for the same reason `src/quota/reset-poller.ts` has one — provider catalogs
+are cached for minutes upstream, so a one-minute cadence buys nothing and costs a
+rate limit.
+
+Resolvers exported from `src/config.ts`:
+
+- `isCatalogAutoRefreshEnabled(config?)` — true only when the section is present and
+  `enabled === true`.
+- `resolveCatalogAutoRefreshIntervalMs(config?)` — bounded milliseconds, or `0` when
+  the operator disabled polling explicitly.
+
+### Scheduler
+
+New `src/codex/catalog-auto-refresh.ts`, shaped after `src/quota/reset-poller.ts`:
+a module-singleton `setInterval` that is unref'd, an in-flight guard so a slow
+provider fetch cannot stack ticks, and a generation counter so a probe still in
+flight when the timer stops cannot publish into the next generation. The config gate
+lives in the callee, which is what lets an operator toggle the setting without a
+restart. Every heavy import — the config barrel, the catalog admission snapshot,
+the convergence path — is a dynamic `import()` inside the tick, so importing this
+module costs nothing at startup.
+
+A tick captures a catalog admission snapshot and calls `convergeCodexCatalog` with
+`{ scope: "catalog", action: "converge" }`, which is exactly the path `ocx sync`
+uses. The existing "external provider owns config.toml" guard lives inside that
+path, so it is respected by construction rather than re-implemented here.
+
+### Observability
+
+`src/codex/catalog-refresh-status.ts` gains a last-outcome record: when the refresh
+ran, its normalized `CatalogDisposition`, whether the served model set changed, and
+a consecutive-failure count. A tick that changes the model set logs one line. The
+per-model "N new models discovered" count issue 3630 asks for is not delivered:
+`convergeCodexCatalog` returns a boolean, not a diff, and widening its return type
+reaches into the catalog writers this lane does not own. The dashboard surface for
+this record belongs to R2-L9.
+
+### Registration
+
+`src/server/background-lifecycle.ts` starts and stops the scheduler alongside the
+quota reset poller, and fires the cadence sync as a floating promise for the same
+reason that one does: startup must not await an optional subsystem.
+
+## Cycle plan
+
+1. Docs cycle — this file.
+2. Config schema and resolvers, with focused tests.
+3. Scheduler, status record, and lifecycle registration, with focused tests.
+4. Capability-declaration regression pinning that a periodic refresh preserves the
+   declared axes, plus the `structure/config.md` update the SSOT rule requires.
+
+## Verification posture
+
+No local suite, no typecheck, no install, no GUI build. Hosted CI at the exact final
+head is the only proof this lane reports.

devlog/_plan/260914_r2l8_catalog_autorefresh/020_issue_3377_capability_audit.md

@@ -0,0 +1,85 @@
+# Issue 3377 capability-declaration audit
+
+Lane R2-L8 of the round-23 delivery unit. This note is the honest reading of the
+current tree against issue 3377, not a plan to close it. The declaration half
+is already on `dev`; what remains is activation, and every remaining activation
+site sits outside this lane's write scope.
+
+## What the type actually carries
+
+`ModelCapabilities` in `src/types/provider.ts` is the stored declaration. An
+exact model-ID entry may set `inputModalities` (`text` / `image` / `audio` /
+`video`), `contextTier` (`default` / `long_context`), and `video.processing`
+(`static` / `agentic`). The comment on `contextTier` is load-bearing: it is a
+requested tier only, and storing it does not imply an upstream window or
+activate an unverified wire. `src/config/provider-validation.ts` is the write
+gate for all three axes. It rejects unknown axes, validates each vocabulary,
+merges PATCH objects without sharing live rows, and treats null
+map/model/axis/processing values as tombstones. File load retains valid axes
+and restricts a malformed explicit modality list to text. Gather fingerprints
+in `src/codex/catalog/provider-fetch.ts` include the whole map, so a periodic
+catalog refresh will preserve whatever was declared; preservation is not
+activation.
+
+## Which axes run today
+
+Only `inputModalities` is consumed at runtime. `configuredInputModalities` in
+`src/codex/catalog/provider-fetch.ts` reads an exact `modelCapabilities[id].inputModalities`
+entry before the legacy `modelInputModalities` record and writes it onto the
+catalog row, including over a live `/models` vote that would otherwise win.
+`src/vision/eligibility.ts` (`isModelVisionSidecarConsumer`, exported as
+`isModelTextOnly`) treats a declared text-without-image list as the sidecar
+consumer, and `src/vision/plan.ts` (`requiresVisionPreprocessing`) consults the
+same declaration before legacy hints and vendor metadata. That is the text-only
+axis issue 3377 asked for, and it is live.
+
+`contextTier` is stored and inert. Nothing in `src/providers/github-copilot-transport.ts`
+reads it. That file still only stamps Copilot editor-fingerprint headers and
+fail-closes the OAuth bearer onto an allowlisted `*.githubcopilot.com` host. A
+follow-up that wants `long_context` to mean a larger Copilot window has to own
+that transport and the Copilot-specific request header or body field it would
+emit; the pricing `contextTier` in `src/usage/cost.ts` is a different vocabulary
+and must not be mistaken for this declaration. The catalog already has
+`modelContextWindows` / `contextWindow` as a separate numeric contract, and
+storing `contextTier: "long_context"` does not advertise those windows.
+
+`video.processing` is stored and inert in the same way. The inbound and adapter
+path already knows how to *carry* a video part: `src/chat/inbound.ts` translates
+`video_url` into `input_video`, `src/responses/schema.ts` accepts that block,
+and `src/adapters/google.ts` inlines Gemini video bytes (or a short marker for
+a remote URL) whenever a `video` part is present. None of those sites reads
+`modelCapabilities[id].video.processing`. Static versus agentic is therefore a
+label in config, not a processing-mode switch. Activation would have to land in
+those three files, and it would have to decide what "agentic" means on the
+existing media-bridge loop rather than assuming the Google inline path is
+enough.
+
+## What the surfaces already accept
+
+The management API already takes the full map. `src/server/management/provider-routes.ts`
+validates PATCH `modelCapabilities` with tombstones allowed, merges axes onto
+the live row, and on POST/PUT replacement runs the same merge so a complete
+body cannot smuggle a tombstone through. Operators can therefore persist
+`contextTier` and `video.processing` from the dashboard or a raw editor today;
+the proxy will store them, fingerprint them, and do nothing else with them.
+
+The CLI is narrower. `ocx provider add` in `src/cli/provider.ts` and
+`ocx provider edit` in `src/cli/provider-runtime.ts` accept `--model <id> --text-only`
+and write `inputModalities: ["text"]` for that one id, preserving sibling
+declarations through `mergeModelCapabilities`. There is no `--context-tier` or
+`video.processing` flag. An operator who wants those axes from the CLI has to
+hand-edit `config.json` or PATCH the management API.
+
+## What this lane does not close
+
+Lane R2-L8 does not close issue 3377. Its write scope is the catalog
+auto-refresh scheduler, the config section that gates it, the last-outcome
+record, and the tests and structure paragraph that pin those. Every remaining
+activation site — `src/providers/github-copilot-transport.ts` for the context
+tier, `src/adapters/google.ts` plus `src/responses/schema.ts` plus
+`src/chat/inbound.ts` for video processing, and a CLI flag surface if the
+follow-up wants operator-facing declarations beyond management JSON — is
+outside that scope. A later lane that actually closes 3377 has to own those
+files, prove the Copilot long-context wire and the static/agentic video split
+on a real request, and keep the storage contract in `src/config/provider-validation.ts`
+as the write gate rather than re-implementing it.

scripts/test-layout/layout.json

@@ -276,6 +276,7 @@
     "bun-runtime.test.ts": "ci-workflows",
     "bun-stream-caps.test.ts": "lib",
     "cancel-body-on-abort.test.ts": "server",
+    "catalog-auto-refresh-scheduler.test.ts": "codex-integration",
     "catalog-cursor-search.test.ts": "codex-integration",
     "catalog-free-pricing-status.test.ts": "codex-integration",
     "catalog-full-picker-order.test.ts": "codex-integration",
@@ -550,6 +551,7 @@
     "compatibility-manifest.test.ts": "codex-integration",
     "compatibility-provider-equivalence.test.ts": "routing",
     "compatibility-version.test.ts": "ci-workflows",
+    "config-catalog-auto-refresh.test.ts": "config",
     "config-load-degrade.test.ts": "config",
     "config-mutation-lock.test.ts": "config",
     "config-ownership-uninstall.test.ts": "config",

src/codex/catalog-auto-refresh.ts

@@ -0,0 +1,182 @@
+/**
+ * Opt-in periodic catalog refresh so newly released models appear without a
+ * manual `ocx sync` (issue #3630).
+ *
+ * This is load-bearing, not a convenience. The served model set is otherwise
+ * only rewritten by an explicit sync, a management mutation, or startup
+ * convergence, so an overnight provider release stays invisible until someone
+ * happens to run one of those. The overnight case is the whole reason the
+ * scheduler exists.
+ *
+ * Shape follows src/quota/reset-poller.ts: a module-singleton unref'd interval
+ * whose config gate lives in the callee, so toggling `enabled` or changing the
+ * cadence takes effect on the next tick without a restart (the rationale
+ * spelled out at src/oauth/token-guardian.ts:276). Importing this module at
+ * startup must cost nothing — src/server/background-lifecycle.ts loads it
+ * statically — so the config barrel, the admission snapshot, and the
+ * convergence funnel are all dynamic import()s inside the tick.
+ */
+
+/**
+ * Keep these numeric literals aligned with CATALOG_AUTO_REFRESH_* in src/config.ts.
+ * They cannot be imported from there: this module is a static edge from
+ * background-lifecycle, and the config barrel is a heavy import reserved for the tick.
+ */
+const DEFAULT_INTERVAL_MS = 60 * 60_000;
+const MIN_INTERVAL_MS = 15 * 60_000;
+/**
+ * Commit-lock wait only. Gather already has per-provider timeouts, and automatic
+ * callers fail fast and defer (ConvergeRequest.mode) rather than holding the
+ * write lock across a slow tick.
+ */
+const TICK_DEADLINE_MS = 1_000;
+
+let timer: ReturnType<typeof setInterval> | null = null;
+let detachShutdownHook: (() => void) | null = null;
+/** The bounded cadence the live timer was created with, so a tick can notice config drift. */
+let liveIntervalMs: number | null = null;
+/**
+ * Bumped by every start and stop. A tick captures it on entry and re-checks before publishing,
+ * so a converge still in flight when the timer stops cannot publish into the next generation.
+ */
+let generation = 0;
+/** setInterval does not skip a firing while the previous callback is still awaiting. */
+let inFlight = false;
+
+/** Number of ticks that have run. Test-only observability; carries no catalog data. */
+let tickCount = 0;
+
+function boundedInterval(value: number): number {
+  return Math.max(MIN_INTERVAL_MS, Math.floor(value));
+}
+
+/** Re-arm the timer when the operator changed the cadence since it was created. */
+function restartIfCadenceChanged(configured: number): void {
+  if (timer === null || boundedInterval(configured) === liveIntervalMs) return;
+  stopCatalogAutoRefresh();
+  startCatalogAutoRefresh(configured);
+}
+
+async function tick(): Promise<void> {
+  // An interval firing while the previous converge is still awaiting would stack
+  // provider fetches precisely when a slow /models call is already in flight.
+  if (inFlight) return;
+  inFlight = true;
+  const entryGeneration = generation;
+  try {
+    const {
+      loadConfig,
+      isCatalogAutoRefreshEnabled,
+      resolveCatalogAutoRefreshIntervalMs,
+    } = await import("../config");
+    const config = loadConfig();
+    if (!isCatalogAutoRefreshEnabled(config)) return;
+    const configured = resolveCatalogAutoRefreshIntervalMs(config);
+    // 0 is dormant: the section stays configured but this tick must not converge,
+    // and the unref'd timer is left running so flipping the minutes back on is
+    // picked up without a process restart.
+    if (configured === 0) return;
+    // A stop or restart landed while the config resolved: this tick no longer owns the timer,
+    // so it must neither count as a refresh nor adopt a cadence for a generation that is gone.
+    if (entryGeneration !== generation) return;
+    // Adopt a changed cadence without a restart, which is why the config gate lives in the
+    // callee at all. Only while this tick still owns the timer.
+    restartIfCadenceChanged(configured);
+    tickCount += 1;
+    const [{ createManagementConvergeCodex }, { createCatalogConvergeRequest }] = await Promise.all([
+      import("./management-convergence"),
+      import("./catalog-admission"),
+    ]);
+    // A stop or restart landed while the funnel was loading: the result belongs to a
+    // generation that no longer owns the timer, so it must not publish.
+    if (entryGeneration !== generation) return;
+    const converge = createManagementConvergeCodex(config);
+    const outcome = await converge(createCatalogConvergeRequest({ deadlineMs: TICK_DEADLINE_MS }));
+    if (entryGeneration !== generation) return;
+    // createManagementConvergeCodex always projects catalog-only. Any other kind is a
+    // funnel contract break, not something this scheduler should re-classify.
+    if (outcome.kind !== "catalog-only") return;
+    const { recordCatalogAutoRefreshOutcome } = await import("./catalog-refresh-status");
+    recordCatalogAutoRefreshOutcome(outcome.catalogRefresh, outcome.changed);
+    if (outcome.changed) {
+      // Privacy scan: no provider names, model ids, paths, or account identifiers.
+      console.info("[catalog-auto-refresh] served model set changed");
+    }
+  } catch {
+    // A failed refresh is not an error worth surfacing: the next tick tries again.
+  } finally {
+    inFlight = false;
+  }
+}
+
+/** Idempotent. A second call while running is a no-op, matching startQuotaResetPoller. */
+export function startCatalogAutoRefresh(intervalMs = DEFAULT_INTERVAL_MS): void {
+  if (timer) return;
+  const bounded = boundedInterval(intervalMs);
+  generation += 1;
+  liveIntervalMs = bounded;
+  timer = setInterval(() => void tick(), bounded);
+  // Never keep the process alive for a catalog refresh.
+  timer.unref?.();
+  void import("../lib/optional-shutdown-hooks")
+    .then(hooks => {
+      detachShutdownHook = hooks.registerOptionalShutdownHook(
+        "catalog-auto-refresh",
+        stopCatalogAutoRefresh,
+      );
+    })
+    .catch(() => {
+      // Without the hook the unref'd timer still cannot delay exit.
+    });
+}
+
+export function stopCatalogAutoRefresh(): void {
+  if (timer) {
+    clearInterval(timer);
+    timer = null;
+  }
+  liveIntervalMs = null;
+  generation += 1;
+  detachShutdownHook?.();
+  detachShutdownHook = null;
+}
+
+export function isCatalogAutoRefreshRunning(): boolean {
+  return timer !== null;
+}
+
+/**
+ * Adopt the operator's configured cadence at startup.
+ *
+ * The caller starts the scheduler synchronously with the default interval, because
+ * resolving the config here would mean a static edge to ../config from a module
+ * background-lifecycle imports at load time. Resolving it through import() keeps
+ * that edge dynamic, at the cost of the timer running at the default for the few
+ * microtasks before this settles.
+ */
+export async function syncCatalogAutoRefreshCadence(): Promise<void> {
+  const { loadConfig, resolveCatalogAutoRefreshIntervalMs } = await import("../config");
+  const configured = resolveCatalogAutoRefreshIntervalMs(loadConfig());
+  // 0 is dormant: tick() already returns before converging, and the timer stays unref'd.
+  if (configured === 0) return;
+  restartIfCadenceChanged(configured);
+}
+
+/** Test-only: run one tick synchronously rather than waiting out the interval. */
+export async function runCatalogAutoRefreshTickForTests(): Promise<void> {
+  await tick();
+}
+
+export function catalogAutoRefreshTickCountForTests(): number {
+  return tickCount;
+}
+
+/** Test-only: the bounded cadence the live timer is running at, or null when stopped. */
+export function catalogAutoRefreshIntervalForTests(): number | null {
+  return liveIntervalMs;
+}
+
+export function resetCatalogAutoRefreshForTests(): void {
+  stopCatalogAutoRefresh();
+  tickCount = 0;
+}