[pull] master from cube-js:master - #697
Merged
Merged
Conversation
…11607) **Description of Changes Made** Refresh key queries were stored under one cache key and looked up under another. Three things followed from that: the per-request memo in `PreAggregationLoadCache` could never hit, so every `waitForRenew: false` request took the deferred path; the refresh scheduler and the pre-aggregation loader marked each other's entry stale whenever a cube `refreshKey` and a pre-aggregation `invalidateKeyQuery` resolved to the same SQL; and the same SQL run against different engines or different data sources shared a single entry and served each other's rows. This PR gives a refresh key one identity — `[sql, params, external, dataSource]` — computed in one place (`QueryCache.refreshKeyIdentity`) and applied by one helper (`cacheRefreshKeyResult`) that owns the cache key, the renewal key and the renewal threshold, so a caller can no longer store an entry under a key it will later look up by a different one. --- ### How it broke, and how it is fixed **One entity, three different hashes** ``` ONE refresh key query [ "SELECT FLOOR(EXTRACT(EPOCH FROM NOW())/86400) refresh_key", [], { renewalThreshold: 86400, external: false, incremental: false } ] dataSource: "default" └──── sql ────┘ └──── options ────┘ BEFORE AFTER ───────────────────────────────────────────── ──────────────────────────────────── H2 = md5([sql, params]) <- cacheKey H = md5([sql, params, H2 <- renewalKey (loader) !!external, H3 = md5([sql, params, options]) dataSource || 'default']) <- renewalKey (scheduler) ^ H3 <- memo lookup | refreshKeyIdentity() | one hash everywhere H2 != H3 => the bugs below +-- cacheKey = renewalKey = memo key ``` **Bug 1 — the load cache memo could never hit** `PreAggregationLoadCache` lives for one request and memoizes resolved refresh key values so that several partitions or pre-aggregations do not each re-run the same SQL. ``` PreAggregationLoadCache.queryResults (in memory, one per request) +--------------------------------------------------------------+ | { "H2": Promise<[{refresh_key: 20685}]> } | +--------------------------------------------------------------+ ^ WRITE READ ^ | keyQueryResult() | hasKeyQueryResult() | queryCacheKey([query, values]) | queryCacheKey(keyQuery) | = H2 | = H3 <- the whole 3-tuple | | -----+-----------------------------------------------+----- H2 != H3 -> miss, 100% of the time, for four years (regression from #3061: `options` moved into the tuple, only the store side was updated) ``` Both sides now go through `refreshKeyCacheKey()` -> `refreshKeyIdentity()`, so they agree on `H`. **Bug 1, consequence — a dead branch in `loadPreAggregation`** ``` BEFORE AFTER ──────────────────────────────────── ────────────────────────────────────── notLoadedKey = keys.find(k => !warm(k)) allWarm = keys.every(k => warm(k)) warm() === false ----+ warm() actually works => always truthy | v if (isJob || !(notLoadedKey && !waitForRenew)) if (isJob || (!externalRefresh && (waitForRenew || allWarm))) collapses to: isJob || waitForRenew <- "keys are warm" never contributed ``` On a live request against a partitioned rollup: ``` POST /v1/load (default cache mode => waitForRenew = false) | +- PartitionRangeLoader.loadRangeQuery() | +- getInvalidationKeyValues() --> keyQueryResult(q) --> queryResults{ H: 20685 } | ^ memo warmed here +- partition 2025-06 -+ +- partition 2025-07 -+--> loadPreAggregation() +- partition 2025-08 -+ | +-- BEFORE: hasKeyQueryResult -> H3 -> miss | => Case 3: serve what exists, refresh in background | => refreshKeyValues: [] | +-- AFTER: hasKeyQueryResult -> H -> hit => Case 1: confirm freshness inline => refreshKeyValues: [{refresh_key: 20685}] ``` Re-enabling that branch needed a guard, which also closes a hole that exists on master today: ``` externalRefresh: BEFORE warm-key path unreachable (always a miss) => Case 3 [ok] but invalidateKeyQueries: [] left notLoadedKey undefined, so !(undefined && ...) was true => Case 1, BUILDS [bug] naive fix warm keys now route those requests into => Case 1, BUILDS [bug] THIS PR !externalRefresh guard: both cases report the partition as missing [ok] ``` **Bug 2 — renewal key ping-pong between the scheduler and the loader** When a cube level `refreshKey` and a pre-aggregation `invalidateKeyQuery` resolve to the same SQL, both paths write **the same cache entry** with a different `renewalKey`, and `decideCacheAction` treats `entry.renewalKey !== renewalKey` as a reason to re-fetch. ``` t0 refresh scheduler - loadRefreshKey(q) ------> entry{ value: 20685, renewalKey: H3 } | t1 /v1/load - keyQueryResult(q) -> reads entry | expects H2, finds H3 --> isKeyMismatch --> SQL --> entry{ 20685, H2 } -----+ | t2 refresh scheduler - loadRefreshKey(q) -> reads entry | expects H3, finds H2 --> isKeyMismatch --> SQL --> entry{ 20685, H3 } -----+ ^ t3 ... on every touch, while the value never changed AFTER: both paths -> cacheRefreshKeyResult() -> renewalKey = cacheKey = H +----------------------------------------------------------+ | entry{ value: 20685, renewalKey: H } <- both read, hit | +----------------------------------------------------------+ ``` The same mismatch also disqualifies the in-memory entry (`isMemoryEntryUsable` requires equality too), so the miss was doubled. **Bug 3 — entries collided across engines and across data sources** `cacheQueryResult` routes a refresh key query on two dimensions — `external` picks the engine, `dataSource` picks the queue and the driver — and the cache prefix separates tenants, not data sources. Neither dimension was in the key. ``` BEFORE: identity = [sql, params] "SELECT MAX(updated_at) FROM orders" +-- prod / source DB --+ +-- staging / source DB --+--> md5([sql, params]) = H2 --> ONE entry +-- prod / Cube Store --+ whoever ran first served everyone AFTER: identity = [sql, params, !!external, dataSource || 'default'] prod / source DB --> Hp -+ staging / source DB --> Hs -+-> three distinct entries prod / Cube Store --> Hx -+ both defaults normalized: external: undefined | false -> false dataSource: undefined | 'default' -> 'default' (JSON.stringify renders undefined as null, so the spellings had to collapse; `getQueue` already resolves an absent dataSource to `default`) ``` **Not a bug, just deduplicated** `invalidateKeyQueries[0].slice(0, 2)` was spelled out twice — on the write side (`loadRangeQuery`) and on the read side (`checkPartitionsBuildRangeCache`), each with a comment telling the reader to keep it identical to the other. There is now one definition. ``` loadRangeQuery() -+ -+ +- buildRangeInvalidateKey -+ one call, nothing left to drift checkPartitionsBuild… -+ -+ ``` Also in this series: `QueryCache.queryRedisKey` is renamed to `queryCacheKey` — the cache driver has not been Redis-specific since the driver abstraction landed. Mechanical, byte-identical output, no entry moves. `LocalQueueDriverConnection.queryRedisKey` keeps its name: it builds queue keys, not cache keys.
* docs: document per-group subtotals for table chart row grouping * docs(table): disambiguate per-group vs pivot subtotals and fix the exclusion link --------- Co-authored-by: igorlukanin <3852894+igorlukanin@users.noreply.github.com>
* docs: signed embedding session renewal, and folders in Creator Mode Two surgical updates to the iframe embedding docs: - Signed embedding sessions: a token's usable life is ~23 hours, not the nominal 24, and a long-lived tab can now renew its session in place instead of dropping to a "Session expired" message. Documents the two new events (`cube:event:session-expiring`, `cube:event:session-expired`) and the new action (`cube:action:set-session`) on the Events & actions page, with a worked renewal example, and updates the session lifecycle summary on the Signed embedding page. - Creator Mode: embed users can now create, rename, and delete folders in their workspace, so a growing set of creators isn't stuck with a flat content list. * docs: drop the autoRenewSession Tip, an undocumented API reference Review feedback: @cube-dev/embed-sdk's connectCubeEmbed/autoRenewSession aren't documented anywhere in docs-mintlify (the React Embed SDK page only covers CubeEmbedProvider), so the Tip was a dead end for readers. The plain postMessage recipe above it is self-contained and stays. * docs: fix review findings — renewal race, error signal, wording - Guard the renewal example against a double mint: session-expiring and session-expired can both fire for one session, and the unguarded version sent two set-session actions. Adds an in-flight flag and a .catch. - Name the concrete cube:event:error signal (context: "session-renewal", name: "EmbedSessionExchangeError") a rejected set-session id reports, so "you can retry" is actually actionable. - Settle the token's usable life on one number (~23h) instead of stating it two different ways across signed.mdx and events.mdx. - Creator Mode: match the file's one-line bullet style, and drop the tooltip detail from the folder-creation caveat — the durable fact is that it's disabled in a shared folder, not how that's surfaced in the UI. * docs: list session-renewal in the error-event context examples Nit from review: the cube:event:error payload table only showed "embed-render" as an example context value, but the new set-session section tells readers to filter on "session-renewal" specifically. * docs: drop the Creator Mode folders note Folder create/rename/delete for embed-tenant users isn't worth calling out separately — scoping this PR back to the signed embedding session renewal docs. --------- Co-authored-by: Claude <noreply@anthropic.com>
#11685) The Parent section told readers to "set **Default value**" in the control's settings. That field is being removed (CUB-4201, cubedevinc/cubejs-enterprise#14501): no other control has one, and a second writer for the same field could disagree with the dashboard itself. The default is now what a filter's static default and a time granularity switcher's default already were — the value you last picked in the control on the dashboard, saved on the widget. What follows from the pick is unchanged, so the paragraph about the children's saved defaults lining up stays as it was. Also states what happens when the option serving as the default is deleted: it is cleared, and the parent goes back to opening on nothing.
…11686) Cube Cloud reworked how a scheduled refresh picks its notification recipients (CUB-3302), and both of these pages described the old UI. Recipients used to be one picker with Users and User groups as sections inside it; they are now two separate controls, the second a dropdown holding an expandable checklist of groups and their members. That checklist is the feature: tick a group, expand it, and untick anyone who should be skipped, stored as an exception to that group on that schedule. The viewer's subscribe toggle changed meaning as well. Someone reached only through a group previously had no recipient row to remove, so the toggle could not turn itself off; it now also records them as an exception in every group notifying them, and subscribing reverses both halves. The email footer's Unsubscribe link goes through the same path, so it covers group delivery too. Also corrects three smaller drifts on the same surfaces: the delivery channel is radio buttons rather than a toggle, the frequency select is now labelled Schedule (its anchor is kept as #frequency for inbound links), and switching a schedule to Slack clears group exceptions as well as individual subscriptions.
Two gaps, one of them newly closed in the product.
`controls.mdx` never said that a published dashboard's URL carries the
viewer's control values at all — so the page described three controls whose
selections looked, to a reader, like they lived only in that browser tab. It
now has a "Sharing the current selection" section covering both parameters,
what does and does not travel in a link (a static default and a
user-attribute default deliberately stay out, so a shared link neither pins a
stale default nor leaks the sharer's own attribute value onto the recipient),
and when an inbound parameter is ignored.
`embedding/iframe/dashboards.mdx` documented only `?f_…`, though `?tg_…` for
time granularity has worked on the way in for as long. Its section now covers
both, in a table, and says the values also travel back out. The heading gained
"and granularities" with an explicit `{#pre-set-dashboard-filters-via-url}`
anchor so the existing link from `events.mdx` keeps resolving.
Companion to cubedevinc/cubejs-enterprise#14497 (CUB-4198), which added the
outbound half for time granularity — a viewer's granularity pick now reaches
the URL the way a filter pick already did.
Co-authored-by: Gleb <gleb@unknownd2d4c7120595.localdomain>
…1689) A filter could resolve its default from the viewer's user attribute; a parent control could not, so a dashboard that groups its controls behind one had no way to open per viewer. Documents the new toggle under the parent's Default option: how the value is matched (against the option LABELS, case- and space-insensitively), what an array attribute does (first entry that names an option, since the control is single-select), and what happens when nothing matches (the picked default stands, children included). Also states the two precedence rules a reader will hit in practice — a viewer's own pick outranks the attribute for the session, and `?f_…` parameters outrank both, so deep links keep working.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )