Bump sqd-query-engine to 6df565a, version 2.16.0-rc.1 - #76
Merged
Merged
Conversation
Move the experimental query engine from 54b9020 (pinned since the initial integration in July) to the current upstream master tip, and start the 2.16.0 release candidate line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The engine pin moves to the current master tip, which groups a table's keys into `request`, `output` and storage blocks and closes both public surfaces: a column is filterable only if `request.filters` names it and selectable only if `output.fields` does. The fixtures still spoke the flat layout — `query_name`, `field_name`, `json_encoding` — and the loader rejects an unknown key rather than ignoring it, so every test that parsed one failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEHFzAGrwxmu429ck2oDk9
A parse fault was a verdict on the whole staged bundle, so one stale catalog stopped the worker serving every kind — including the kinds it holds chunks of and could answer. A schema is load-bearing only for the chunks that name it, and `DatasetsIndex` already refuses an assignment reaching for an id the bundle does not offer, so dropping the schema puts the refusal where the dependency actually is. The store path had this right already: `read_store` skips a file it cannot read and lets a later bundle reinstall the id. A bundle in which nothing parses is still permanent — there is no query it could serve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEHFzAGrwxmu429ck2oDk9
define-null
added a commit
that referenced
this pull request
Sep 10, 2026
## Why After the engine bump in #76, the CDN's old schema layout fails to parse, while the scheduler's bundle already uses the new layout. Workers continued polling the CDN even after applying a split assignment whose chunks resolve schemas by ID. The bundle loader also rejected any existing schema ID republished with different bytes. This made workers with cached schemas reject updates that fresh workers accepted. ## What changes The assignment applier owns a concurrent CDN refresh task: - It starts initially so legacy schemas load in parallel with assignment fetching. - Successfully applying a split assignment cancels and awaits the task, interrupting in-flight HTTP requests and retry waits. - Successfully applying a legacy assignment starts the task if absent, with an immediate refresh. Further legacy assignments reuse the running task. - Refused or failed assignment updates leave the task unchanged. Previously loaded type schemas remain available after it stops. **CDN fetch or parse failures do not fail or block legacy assignment application.** The task retains previously loaded schemas and retries with exponential backoff. Before the first successful load, dynamic queries requiring type schemas return `server_error`. Split assignments still require their separate schema bundle to download, validate, and install successfully. The watch channel and separate CDN subsystem are removed. The applier observes refresh-task failures, and dropping it aborts any remaining refresh task. The end-to-end harness uses the same ownership model. A bundle that republishes an existing schema ID with different contents replaces its stored file and loaded schema. Identical files remain untouched, and IDs absent from the new bundle remain in the accumulated store. ## Validation - All 115 tests passed with `cargo test --locked --offline --all-targets`. - The strengthened assignment-transition test also passed separately, checking that failed split installs leave the existing CDN task running. - Coverage includes immediate startup/restart, repeated starts without duplicate tasks, refused and failed assignment transitions, cancellation during manifest and schema HTTP requests, retry cancellation, retention of loaded schemas, and propagation of refresh-task panics. - Bundle replacement is tested with both bundle-installed and startup-adopted schemas. - `cargo fmt --check` and `git diff --check` passed. Clippy reports only the existing warning in `polars_target.rs`. README and the relevant schema-source specifications are updated. ## Remaining behavior - Legacy workers still need the CDN YAMLs republished in the new layout to serve affected dynamic queries. This change does not repair those documents. - The initial CDN task may log a warning before the first split assignment applies. - Type schemas are not persisted. Retained chunks outside the current split assignment can lack a type schema if no CDN load succeeded before the task stopped. - Bundle installation still uses individual file renames; this change does not make multi-file replacement transactional. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bump
sqd-query-engine(the experimental dynamic query engine) from54b9020to6df565a, the current upstream master tip, and set the worker version to2.16.0-rc.1. The previous pin dated from the initial integration in July and had been orphaned upstream (not reachable from master).Upstream range: 52 commits. Full list: subsquid/sqd-query-engine@54b9020...6df565a
The catalog format changed
Two of those commits reshape the dataset description the engine loads, and
DatasetDescriptionisdeny_unknown_fields, so a stale catalog is a load error rather than a silently ignored key:c31a4a0makesfiltersa required, closed list — a table that accepts none saysfilters: [].b72c404groups a table's keys intorequest(what a client may send),output(what a client may see) and the storage keys.query_name→request.name,field_name→output.name,json_encoding: hex→encoding: hex_bytes;stats,dictionary,childrenandparent_keyare gone.Both surfaces are now closed: a column is filterable only if
request.filtersnames it, and selectable only ifoutput.fieldsdoes.The four inlined test catalogs still spoke the flat layout, which is what the first CI run caught — 23 failures, all of them
missing field 'filters'or an unknown key. They are rewritten here; no production code needed changes.Note for whoever ships this: the catalogs the worker fetches at runtime have to be republished in the new layout too, or the worker hits the same load error outside the tests.
One bad schema no longer takes the bundle down
Separate commit. Parsing a staged schema failed the whole bundle, which is how a stale catalog for
a kind this worker holds no chunk of would stop it serving the kinds it does hold.
load_dirnowdrops the schema it cannot parse, warns, and counts it (
schema_bundle_schemas_skipped), leavingthe rest installed.
That is not a loosening: the id is left out of
bundle_ids, andDatasetsIndexalready refuses anassignment whose chunks reach for an id the bundle does not offer — so the refusal lands on the
assignment that actually depends on the schema rather than on every assignment. A bundle in which
nothing parses stays a permanent fault, since no query could be served from it. The store path had
this shape already:
read_storeskips a file it cannot read and lets a later bundle reinstall theid.
Test plan
cargo check --all-targets --lockedcargo test --all-features --lib --bins --test state_pbt --test state_regressione2e,query_surface,query_concurrency), default and--all-features🤖 Generated with Claude Code
https://claude.ai/code/session_01MEHFzAGrwxmu429ck2oDk9