Skip to content

Bump sqd-query-engine to 6df565a, version 2.16.0-rc.1 - #76

Merged
define-null merged 3 commits into
masterfrom
bump-experimental-query-engine
Sep 8, 2026
Merged

define-null merged 3 commits into
masterfrom
bump-experimental-query-engine

Conversation

@define-null

@define-null define-null commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Bump sqd-query-engine (the experimental dynamic query engine) from 54b9020 to 6df565a, the current upstream master tip, and set the worker version to 2.16.0-rc.1. The previous pin dated from the initial integration in July and had been orphaned upstream (not reachable from master).

Upstream range: 52 commits. Full list: subsquid/sqd-query-engine@54b9020...6df565a

The catalog format changed

Two of those commits reshape the dataset description the engine loads, and DatasetDescription is deny_unknown_fields, so a stale catalog is a load error rather than a silently ignored key:

  • c31a4a0 makes filters a required, closed list — a table that accepts none says filters: [].
  • b72c404 groups a table's keys into request (what a client may send), output (what a client may see) and the storage keys. query_name → request.name, field_name → output.name, json_encoding: hex → encoding: hex_bytes; stats, dictionary, children and parent_key are gone.

Both surfaces are now closed: a column is filterable only if request.filters names it, and selectable only if output.fields does.

The four inlined test catalogs still spoke the flat layout, which is what the first CI run caught — 23 failures, all of them missing field 'filters' or an unknown key. They are rewritten here; no production code needed changes.

Note for whoever ships this: the catalogs the worker fetches at runtime have to be republished in the new layout too, or the worker hits the same load error outside the tests.

One bad schema no longer takes the bundle down

Separate commit. Parsing a staged schema failed the whole bundle, which is how a stale catalog for
a kind this worker holds no chunk of would stop it serving the kinds it does hold. load_dir now
drops the schema it cannot parse, warns, and counts it (schema_bundle_schemas_skipped), leaving
the rest installed.

That is not a loosening: the id is left out of bundle_ids, and DatasetsIndex already refuses an
assignment whose chunks reach for an id the bundle does not offer — so the refusal lands on the
assignment that actually depends on the schema rather than on every assignment. A bundle in which
nothing parses stays a permanent fault, since no query could be served from it. The store path had
this shape already: read_store skips a file it cannot read and lets a later bundle reinstall the
id.

Test plan

  • cargo check --all-targets --locked
  • cargo test --all-features --lib --bins --test state_pbt --test state_regression
  • Conformance tier (e2e, query_surface, query_concurrency), default and --all-features
  • New: a bundle carrying one unparseable schema installs the other two and offers only their ids
  • New: a bundle in which nothing parses is still refused as permanent, with nothing left in the store

🤖 Generated with Claude Code

https://claude.ai/code/session_01MEHFzAGrwxmu429ck2oDk9

define-null and others added 2 commits September 4, 2026 15:45
Move the experimental query engine from 54b9020 (pinned since the
initial integration in July) to the current upstream master tip, and
start the 2.16.0 release candidate line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The engine pin moves to the current master tip, which groups a table's
keys into `request`, `output` and storage blocks and closes both public
surfaces: a column is filterable only if `request.filters` names it and
selectable only if `output.fields` does. The fixtures still spoke the
flat layout — `query_name`, `field_name`, `json_encoding` — and the
loader rejects an unknown key rather than ignoring it, so every test that
parsed one failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MEHFzAGrwxmu429ck2oDk9
@mo4islona mo4islona changed the title Bump sqd-query-engine to 6e3f691, version 2.16.0-rc.1 Bump sqd-query-engine to 6df565a, version 2.16.0-rc.1 Sep 5, 2026
A parse fault was a verdict on the whole staged bundle, so one stale
catalog stopped the worker serving every kind — including the kinds it
holds chunks of and could answer. A schema is load-bearing only for the
chunks that name it, and `DatasetsIndex` already refuses an assignment
reaching for an id the bundle does not offer, so dropping the schema puts
the refusal where the dependency actually is.

The store path had this right already: `read_store` skips a file it cannot
read and lets a later bundle reinstall the id. A bundle in which nothing
parses is still permanent — there is no query it could serve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MEHFzAGrwxmu429ck2oDk9
@define-null
define-null merged commit 01f33b7 into master Sep 8, 2026
5 checks passed
@define-null
define-null deleted the bump-experimental-query-engine branch September 8, 2026 09:45
define-null added a commit that referenced this pull request Sep 10, 2026
## Why

After the engine bump in #76, the CDN's old schema layout fails to
parse, while the scheduler's bundle already uses the new layout. Workers
continued polling the CDN even after applying a split assignment whose
chunks resolve schemas by ID.

The bundle loader also rejected any existing schema ID republished with
different bytes. This made workers with cached schemas reject updates
that fresh workers accepted.

## What changes

The assignment applier owns a concurrent CDN refresh task:

- It starts initially so legacy schemas load in parallel with assignment
fetching.
- Successfully applying a split assignment cancels and awaits the task,
interrupting in-flight HTTP requests and retry waits.
- Successfully applying a legacy assignment starts the task if absent,
with an immediate refresh. Further legacy assignments reuse the running
task.
- Refused or failed assignment updates leave the task unchanged.
Previously loaded type schemas remain available after it stops.

**CDN fetch or parse failures do not fail or block legacy assignment
application.** The task retains previously loaded schemas and retries
with exponential backoff. Before the first successful load, dynamic
queries requiring type schemas return `server_error`. Split assignments
still require their separate schema bundle to download, validate, and
install successfully.

The watch channel and separate CDN subsystem are removed. The applier
observes refresh-task failures, and dropping it aborts any remaining
refresh task. The end-to-end harness uses the same ownership model.

A bundle that republishes an existing schema ID with different contents
replaces its stored file and loaded schema. Identical files remain
untouched, and IDs absent from the new bundle remain in the accumulated
store.

## Validation

- All 115 tests passed with `cargo test --locked --offline
--all-targets`.
- The strengthened assignment-transition test also passed separately,
checking that failed split installs leave the existing CDN task running.
- Coverage includes immediate startup/restart, repeated starts without
duplicate tasks, refused and failed assignment transitions, cancellation
during manifest and schema HTTP requests, retry cancellation, retention
of loaded schemas, and propagation of refresh-task panics.
- Bundle replacement is tested with both bundle-installed and
startup-adopted schemas.
- `cargo fmt --check` and `git diff --check` passed. Clippy reports only
the existing warning in `polars_target.rs`.

README and the relevant schema-source specifications are updated.

## Remaining behavior

- Legacy workers still need the CDN YAMLs republished in the new layout
to serve affected dynamic queries. This change does not repair those
documents.
- The initial CDN task may log a warning before the first split
assignment applies.
- Type schemas are not persisted. Retained chunks outside the current
split assignment can lack a type schema if no CDN load succeeded before
the task stopped.
- Bundle installation still uses individual file renames; this change
does not make multi-file replacement transactional.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants