Skip to content

[pull] master from cube-js:master - #690

Merged
pull[bot] merged 4 commits into
code:masterfrom
cube-js:master
Aug 26, 2026
Merged

[pull] master from cube-js:master#690
pull[bot] merged 4 commits into
code:masterfrom
cube-js:master

Conversation

@pull

@pull pull Bot commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown

See Commits and Changes for more details.

Created by pull[bot] (v2.0.0-alpha.4)

Can you help keep this open source service alive? 💖 Please sponsor : )

ovr and others added 4 commits August 26, 2026 15:25
@ovr
refactor(query-orchestrator): Queue - drop getQueryDef lookup on skip…
c05a70f 
…ped processing (#11654)
@ovr
v1.7.27
43964f8
@paveltiunov @claude
feat(cubesql): Push UNION down to the data source (#11651)
2aba3ac 
* test(cubesql): Cover UNION of SELECT DISTINCTs in limitless post processing

A UNION of two SELECT DISTINCTs over different cubes cannot be pushed down:
the branches read different cubes, so the UNION only runs in DataFusion, and
the per-branch DISTINCTs collapse into the one above it, leaving each branch an
ungrouped scan of raw rows. Pin that shape - the plan it compiles to by
default, that the check rejects it, that grouping each branch does not change
the verdict, and that a limit on each branch does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(cubesql): Pin where an outer limit reaches a union's branches

UnionSortLimitPushDown copies a limit above a UNION into each of its inputs,
so UNION ALL under a limit ends up with bounded scans. A plain UNION dedups
above the branches, the limit cannot move through it, and the branches stay
unlimited.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* feat(cubesql): Push UNION down to the data source

A UNION of queries that all reach the same data source had no representation
in the wrapper, so the set operation always ran in DataFusion on top of Cube
queries the row limit had already truncated. `SELECT DISTINCT ... UNION SELECT
DISTINCT ...` over two views is the shape this shows up as, and it answers with
values silently missing.

Add a WrappedUnion node the rewriter can put a union into, a rule that moves
the wrappers of every query into one wrapper around them, and a rule that folds
the DISTINCT of a plain UNION into it rather than leaving the deduplication
above. Anything above the union - a projection, a filter - is then pushed into
a select that reads from it, and the row cap bounds the result of the whole
operation instead of each query.

Queries that reach different data sources are left alone: one data source
cannot evaluate a set operation over another's rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* fix(cubesql): Pass assert messages as format arguments

The crate is on the 2018 edition, where an inline argument in a panic
message is not interpolated, and clippy rejects it. Also render the row
cap of a pushed down union in the dialects that have no LIMIT: T-SQL
reads the set operation through a derived table with TOP, Oracle ends it
with FETCH NEXT.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* fix(cubesql): Only push a union down where the data source can render it

Gate the rule on the data source having a union template, the way the join
and subquery rules gate on theirs. Without it a data source with no template
fails at SQL generation, where the plan is already chosen and there is no way
back to post processing.

Give ClickHouse a template of its own: it rejects a bare UNION unless
union_default_mode is set, so the set operation has to name which one it is.

Read the e-graph so the rule fails closed: an input list is only read when
its e-class holds one list, and an input's data source only when every way
of wrapping it agrees on the same one, rather than committing to whichever
representation comes first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(cubesql): Cover joining against a pushed down union

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* fix(cubesql): Give every dialect a union it can parse, or none at all

BigQuery reads a bare UNION as a parse error the way ClickHouse does:
GoogleSQL wants the set operator to name its mode. Give it the same
template.

SQLite takes no parenthesised operands in a compound select, Druid has no
UNION of distinct rows, and ksqlDB has no set operations at all. Drop the
template in those three: the rule only pushes a union down where a template
says the data source can render one, so they keep doing what they do today
and leave it to post processing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(cubesql): Pin that the join reads the union as its subquery

The assertion held on any SQL that mentioned a union anywhere, so a join
that lost its union, or a union that lost its join, would both have passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(schema-compiler): Cover the union template of every dialect

Seven dialects now decide something about statements/union - four write
their own, three drop it - and nothing pinned any of it. Read the templates
off the adapter directory rather than a list, so a dialect added later
cannot escape the invariants: every query rendered once, the row cap
rendered, ALL on the branch that keeps duplicates and no other, and blocks
that close. A missing endif and a renamed loop variable both fail it.

Druid and ksqlDB drop the template in their own packages and are not
covered here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(schema-compiler): Assert the union template renders what it loops over

The per-dialect assertions pinned the loop and the limit guard but neither
interpolation, so a template that rendered the operator, the parentheses and
a LIMIT while dropping the query body satisfied all of them - under a test
named for rendering every query. Pin both, and pin every ALL to the branch
that keeps duplicates rather than to its presence anywhere in the string.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(schema-compiler): Share one dialect scan between the tests that need it

Two copies of "read the adapter directory so a dialect added later cannot
escape this" is one copy too many. Move it to test/unit/allDialects.ts,
returning each dialect with its name so a failure says which one, and let
positional-params drop the names it does not use.

Also name the assumption the union template test rests on: it reads
templates off a bare prototype, which holds for a template a dialect writes
unconditionally and would not for one gated on instance state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(schema-compiler): assert the union template names the operator

Tightening the `ALL` placement check dropped the only assertion that the
word `UNION` appears at all, so a template rendering `(q1)\nALL (q2)`
passed every assertion in a file about the union operator. Assert it, and
make the loop and tag-balance patterns tolerate a tag written without a
space after the opener.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(schema-compiler): name the dialect floor and drop a wrapper alias

The floor message interpolated the count it found rather than the one it
wanted, so a scan that matched three dialects reported "more dialects than
3" without saying how many were expected. Name the threshold in both the
constant and the message.

In the positional-params caller, the local one-call wrapper discarded each
dialect's name and then recovered it from `QueryClass.name`; filter and map
over the pairs instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* refactor(cubesql): Read the union input list the way the converter does

`list_node_ids` walked `UnionInputs` as cons cells and declined anything
else, while `match_list_node_ids!` — which the converter uses to read this
same list — recurses and treats any non-list node as an element. The cons
shape is only what `add_plan_list_node!` happens to build today, so the
rule was coupled to a detail the reader on the other side does not assume.

Mirror that traversal, and use `var_list_iter!` rather than an open-coded
`filter_map`. The guards stay: an e-class holding more than one
`UnionInputs` still fails closed, since picking one would drop or reorder
the queries of the union, and a self-containing list still terminates.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* docs(cubesql): Record how rewrite rules must traverse lists

Every matcher should match one level, and lists are consumed by dedicated
push-down/pull-up rules that peel one cons cell at a time. Write down that
rule, the helpers that generate the traversal, and why an imperative reader
over an e-class is the wrong shape: an e-class holds many representations,
so picking one is arbitrary, and the cons shape is an implementation detail
of how the list was built.

Also record that `wrapper-subqueries-wrapped-scan-to-pull` re-contexts a
pushdown replacer without comparing input data sources, so unification on a
shared pattern variable does not by itself guarantee agreement for
plan-level list elements.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* refactor(cubesql): Traverse the union input list with rewrite rules

The rule read `UnionInputs` by walking it in a transform, which is not how
lists are consumed here: every matcher should match one level, and a list is
peeled one cons cell at a time by dedicated rules.

Split it into a push-down rule that matches the union and the head of its
list, a pull-up rule that peels one query per application, and a tail rule
for the terminal, and drop `list_node_ids` and `wrapped_input` entirely.

The queries of a union arrive already pulled up, so the list carries a
pull-up replacer rather than a push-down one. A push-down replacer over a
finished wrapper is the subquery path's shape, and reaching it here would
have matched `wrapper-subqueries-wrapped-scan-to-pull`, which re-contexts an
element without comparing input data sources.

Data sources now agree by unification instead of by comparison: the head
binds `?input_data_source`, and the pull-up rule shares that variable
between each query's own context and the list's, so a query reaching
another source leaves the list unconsumed and the post processing plan
outprices it. Record the rule in CLAUDE.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* docs(cubesql): Record that e-graph lists are flat, not head/tail

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* refactor(cubesql): Make the union input list flat

`UnionInputs` was built as cons cells by `add_plan_list_node!`, and the rules
walked it one cell at a time. Head/tail lists are legacy here, so build it
flat — one node holding every query — and match the whole list in one rule.

The union now needs a single rule rather than three. It reads every query out
of the list at once and unwraps each into the set operation, with
`?input_data_source` as a top level element variable so the queries have to
agree on a data source for the rule to match at all. Nothing compares data
sources any more, and there is no cons walk left to get wrong.

That rule needs a transform for what a pattern cannot spell out: the alias
converted from `UnionAlias` to `WrappedUnionAlias`, the cleared context, and
the union template check. So add `transforming_list_rewrite_with_lists_and_vars`,
and run the transform inside `ListNodeApplier` once the substitution carries
the list's own variables — wrapping the applier from outside is too early, as
the elements' variables are merged in by the applier itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* test(cubesql): Pin that a post-processed union keeps every query

`UnionInputs` is flat now, and the converter reads it back with
`match_list_node_ids!`. That reader is shape agnostic, but nothing pinned it
above two queries — where flat and cons actually differ — and the existing
no-template test only asserted the plan is still a `Union`, not how many
queries survived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

* fix(cubesql): Restore the two query minimum and the applier var check

Three review findings, all on the flat list rework.

A single query is not a union. Moving the list walk into the pattern dropped
the `inputs.len() < 2` guard the transform used to carry, and with one query
`wrapper-push-down-distinct-to-union` folds the `Distinct` in while the
template emits no operator at all — the deduplication would go silently
missing, which is the failure this PR exists to fix. Express it where the
matching happens: `ListNodeSearcher` now takes a minimum element count, and
the union rule asks for two.

`ListNodeApplier::vars` returned nothing whenever a transform was set, which
switched off egg's check for the list and element patterns too, turning a
typo there from a panic at rule construction into one on the first matching
plan. Take the transform's own variables alongside it and subtract only
those, the way `new_list_var` already is.

The round trip test counted three inputs and three scans, but identity and
order are what a positional read puts at risk: a reversed list, or one query
twice, leaves three of everything. Assert each query's members in order.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xo5DArPYy2Q7ER7HyMo1Ao

---------

Co-authored-by: Claude <noreply@anthropic.com>
@ovr @claude @claude
docs: Cube Store message size limits (#11655)
3080bd4 
* docs: document Cube Store message size limits as a known limitation

Cube talks to the cache and queue store over one WebSocket connection per
node, and query results are delivered through it (QUEUE ACK) and written to
the cache — so a result set above the transport ceiling cannot be delivered
at all. A message over that ceiling is not answered with an error: Cube
Store logs `Capacity(MessageTooLong { .. })` and closes the socket, and the
driver resends everything in flight on reconnect, including the oversized
message, which kills the new connection too. The deployment then reports
`ConnectionError: CubeStore connection error: write EPIPE` in a loop, and
unrelated queries sharing that connection fail alongside it.

Adds a "Message size limits" section to the Cube Store architecture page
describing the limits, the known limitation, and how to stay under it, plus
the three previously undocumented env vars:

| Limit                  | Env var                                | Default | Range          |
| ---------------------- | -------------------------------------- | ------- | -------------- |
| Transport message size | `CUBESTORE_TRANSPORT_MAX_MESSAGE_SIZE` | 64 MiB  | 16 MiB–256 MiB |
| Transport frame size   | `CUBESTORE_TRANSPORT_MAX_FRAME_SIZE`   | 64 MiB  | 4 MiB–256 MiB  |
| Single cache entry     | `CUBESTORE_CACHE_MAX_ENTRY_SIZE`       | 63 MiB  | ≤ message − 1 MiB |

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: trim the Cube Store message size known limitation callout

Drop the resend-loop mechanics and the blast-radius note, keeping just the
symptom users search for: `write EPIPE`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: drop the per-node connection claim from Cube Store message limits

The number of connections between Cube and the cache and queue store is an
implementation detail that may change; the size ceiling is what the section
is about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Update docs-mintlify/docs/pre-aggregations/cube-store-architecture.mdx

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* Update docs-mintlify/docs/pre-aggregations/cube-store-architecture.mdx

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* docs: name both transport limits in the Cube Store mitigation advice

A message is sent as a single frame, so raising the message size alone still
trips `CUBESTORE_TRANSPORT_MAX_FRAME_SIZE`. Also shortens the paragraph.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
@pull pull Bot locked and limited conversation to collaborators Aug 26, 2026
@pull pull Bot added the ⤵️ pull label Aug 26, 2026
@pull
pull Bot merged commit 3080bd4 into code:master Aug 26, 2026
15 of 17 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.