Skip to content

Follow the applied assignment type for schema sources - #77

Merged
define-null merged 2 commits into
masterfrom
schema-source-follows-assignment
Sep 10, 2026
Merged

define-null merged 2 commits into
masterfrom
schema-source-follows-assignment

Conversation

@define-null

@define-null define-null commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

After the engine bump in #76, the CDN's old schema layout fails to parse, while the scheduler's bundle already uses the new layout. Workers continued polling the CDN even after applying a split assignment whose chunks resolve schemas by ID.

The bundle loader also rejected any existing schema ID republished with different bytes. This made workers with cached schemas reject updates that fresh workers accepted.

What changes

The assignment applier owns a concurrent CDN refresh task:

  • It starts initially so legacy schemas load in parallel with assignment fetching.
  • Successfully applying a split assignment cancels and awaits the task, interrupting in-flight HTTP requests and retry waits.
  • Successfully applying a legacy assignment starts the task if absent, with an immediate refresh. Further legacy assignments reuse the running task.
  • Refused or failed assignment updates leave the task unchanged. Previously loaded type schemas remain available after it stops.

CDN fetch or parse failures do not fail or block legacy assignment application. The task retains previously loaded schemas and retries with exponential backoff. Before the first successful load, dynamic queries requiring type schemas return server_error. Split assignments still require their separate schema bundle to download, validate, and install successfully.

The watch channel and separate CDN subsystem are removed. The applier observes refresh-task failures, and dropping it aborts any remaining refresh task. The end-to-end harness uses the same ownership model.

A bundle that republishes an existing schema ID with different contents replaces its stored file and loaded schema. Identical files remain untouched, and IDs absent from the new bundle remain in the accumulated store.

Validation

  • All 115 tests passed with cargo test --locked --offline --all-targets.
  • The strengthened assignment-transition test also passed separately, checking that failed split installs leave the existing CDN task running.
  • Coverage includes immediate startup/restart, repeated starts without duplicate tasks, refused and failed assignment transitions, cancellation during manifest and schema HTTP requests, retry cancellation, retention of loaded schemas, and propagation of refresh-task panics.
  • Bundle replacement is tested with both bundle-installed and startup-adopted schemas.
  • cargo fmt --check and git diff --check passed. Clippy reports only the existing warning in polars_target.rs.

README and the relevant schema-source specifications are updated.

Remaining behavior

  • Legacy workers still need the CDN YAMLs republished in the new layout to serve affected dynamic queries. This change does not repair those documents.
  • The initial CDN task may log a warning before the first split assignment applies.
  • Type schemas are not persisted. Retained chunks outside the current split assignment can lack a type schema if no CDN load succeeded before the task stopped.
  • Bundle installation still uses individual file renames; this change does not make multi-file replacement transactional.

Two changes to how the worker sources query-engine schemas.

The CDN manifest loop now follows the assignment in force instead of
polling unconditionally. The applier publishes whether the applied
assignment resolves chunk schemas by dataset type: true from startup
and under legacy, false once a split assignment applies, true again
when a legacy one does. While false the manifest is not fetched at all
and any retry backoff is cut short; on resume the manifest is refreshed
at once rather than at the next tick. What the type registry loaded
before a pause stays in memory, so chunks held from an earlier
assignment still resolve by type under split if their type had loaded.

A schema id republished with different contents now replaces the
stored copy instead of refusing the whole bundle as a permanent fault.
The bundle in force is the meaning of its ids; an identical file is
still left where it is, and the merge log reports how many ids were
replaced.

The p2p controller builds the applier once so both loops share it, and
the e2e harness is wired the same way. README, IB-44, IB-44b, FM-53,
FM-53b, P-SCHEMA-REFRESH and ADR-23 are updated; ADR-23 carries a dated
revision note.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@define-null
define-null force-pushed the schema-source-follows-assignment branch from 2d8bde2 to 4cf9ec1 Compare September 8, 2026 14:05
@define-null
define-null merged commit 092208e into master Sep 10, 2026
6 checks passed
@define-null
define-null deleted the schema-source-follows-assignment branch September 10, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant