Repository navigation
feat(db)!: make grouping, includes, and pooling opt-in - #2232
Open
KyleAMathews wants to merge 33 commits into
Open
KyleAMathews wants to merge 33 commits into
KyleAMathews wants to merge 33 commits into
Conversation
Joins take an operator value: .join(leftJoin(source, on)). The operator carries the join compiler and the optimizer's join pushdown onto the IR join clause, so apps without joins do not bundle them. Co-authored-by: Isaac <no-reply@databricks.com>
…ll shape Co-authored-by: Isaac <no-reply@databricks.com>
Aggregate values (count, sum, avg, min, max) carry the grouping compiler. A groupBy with no aggregate takes groupKeys(...); without it, compilation throws GroupingOperatorRequiredError. orderBy rewrites aggregates only when it finds one. Co-authored-by: Isaac <no-reply@databricks.com>
include(), toArray(), and materialize() carry the materializer and the Collection-facade adapter. A bare subquery in select() throws IncludeOperatorRequiredError; include(subquery) gives the Collection-valued include. Join operators carry the materializer for custom-key reduction. Co-authored-by: Isaac <no-reply@databricks.com>
Hooks pool a query only when its config passes pool: pooledLiveQueries. The value carries the pooled view and its identity shortcut, so apps that do not pool do not bundle them. React Suspense and DbClient queries pass pool: false and never pool. Co-authored-by: Isaac <no-reply@databricks.com>
Join calls use operator values, bare includes use include(), and the conformance pooling scenario passes pooledLiveQueries through each driver's config form. A pooled config with a query function builds it once. Co-authored-by: Isaac <no-reply@databricks.com>
The live-queries guide, JSDoc examples, and generated reference use join operators, include(), groupKeys(), and pooledLiveQueries. The join size scenario uses leftJoin; size:compare reports n/a for a scenario that cannot build against an older base. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: a19d861 The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
Important Review skippedToo many files! This PR contains 372 files, which is 72 over the limit of 300. To get a review, reduce the PR to 300 files or fewer by splitting it into smaller PRs or changing its base branch. Usage-priced reviews support at most 300 files.
 |
Contributor
|
Size Change: +2.74 kB (+1.23%) Total Size: 225 kB 📦 View Changed
ℹ️ View Unchanged
|
More templates
@tanstack/angular-db
@tanstack/browser-db-sqlite-persistence
@tanstack/capacitor-db-sqlite-persistence
@tanstack/cloudflare-durable-objects-db-sqlite-persistence
@tanstack/db
@tanstack/db-ivm
@tanstack/db-sqlite-persistence-core
@tanstack/electric-db-collection
@tanstack/electron-db-sqlite-persistence
@tanstack/expo-db-sqlite-persistence
@tanstack/node-db-sqlite-persistence
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/preact-db
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/react-native-db-sqlite-persistence
@tanstack/react-router-with-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/tauri-db-sqlite-persistence
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit: |
… pool
A DbClient query never pools, so prepareLiveQueryValue drops pool and a
server preload hashes like a pooling hook. pool moves from
LiveQueryCollectionConfig to LiveQueryHookConfig, which the framework hooks
accept. Angular's { params, query } form takes pool. Join operators have a
named function type so their declarations stay portable. The benchmark runs
against either join API, and the minified API check lists the new errors.
Co-authored-by: Isaac <no-reply@databricks.com>
…rializer A join clause carries one runtime object (compiler, pushdown, validation, and public-key reduction), and the optimizer copies it by spreading the clause. Public-key reduction moves to its own module, so a joined query with a custom key no longer bundles the include materializer. The bundle test now tracks the materializer and the shared reduction module. Co-authored-by: Isaac <no-reply@databricks.com>
… input caseWhen branches take include() for a subquery; the select-object branch no longer accepts a query builder. include() of a non-builder throws a coded error. The compiler finds aggregates with one walker, and the fn.select error names include(). size:compare skips only a base that lacks a scenario's new export. Co-authored-by: Isaac <no-reply@databricks.com>
The shipped agent skills use join operators, include(), groupKeys(), and pooledLiveQueries. join() and groupBy() JSDoc describe the new parameters. The changeset covers Angular's params pool and hand-built IR. Oracle files keep their opening contract prose above their imports. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Incremental update benchmarkComparing Overall median write time vs base: 0.99× · cold hydrate time: 1.00× (geometric mean of per-case ratios; lower is faster). Writes: 1 regression(s), 1 improvement(s) (threshold: ±20% and >0.05ms). Cold hydrate: 0 regression(s), 1 improvement(s) (threshold: ±50% and >5ms). Per-case flags are noisy on shared runners. Read the geometric means first. Writes
Cold hydrate
Each row aggregates the 14.4 scale/index/write-mode configurations of that query; per-configuration tables below. 100 rows/collection | source indexes: none | synced writes — geomean 1.00×, cold 0.91×
100 rows/collection | source indexes: none | optimistic writes — geomean 0.94×, cold 1.24×
100 rows/collection | source indexes: manual | synced writes — geomean 1.04×, cold 0.97×
100 rows/collection | source indexes: manual | optimistic writes — geomean 0.98×, cold 1.10×
1,000 rows/collection | source indexes: none | synced writes — geomean 0.90×, cold 1.02×, 1 change(s)
1,000 rows/collection | source indexes: none | optimistic writes — geomean 1.05×, cold 0.99×
1,000 rows/collection | source indexes: manual | synced writes — geomean 0.88×, cold 0.81×
1,000 rows/collection | source indexes: manual | optimistic writes — geomean 0.88×, cold 1.17×
10,000 rows/collection | source indexes: none | synced writes — geomean 1.21×, cold 1.10×, 1 change(s)
10,000 rows/collection | source indexes: none | optimistic writes — geomean 1.01×, cold 0.92×
10,000 rows/collection | source indexes: manual | synced writes — geomean 1.04×, cold 1.06×
10,000 rows/collection | source indexes: manual | optimistic writes — geomean 0.95×, cold 0.81×, 1 change(s)
Runner: node v24.8.0, linux 6.17.0-1022-azure, INTEL(R) XEON(R) PLATINUM 8573C. Timings on shared CI runners are noisy; treat small deltas as indicative only. |
A join operator infers the query's context from .join(). TypeScript skips that inference when noImplicitAny is off, which left every join condition with unknown refs and compile errors. The context now defaults to any there, so the condition is untyped like other unannotated callbacks in that mode. Strict mode keeps full typing. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Add scripts/check-doc-snippets.mjs, run in the CI lint job as `pnpm test:doc-snippets`. It typechecks each TypeScript example in docs/ and the shipped skills against the workspace sources. Examples that already failed are listed in a baseline by content hash, so a new or edited example must pass. The check found these errors: - Joins lost their types without noImplicitAny. Strict mode keeps them; the test compiles a wrong field in both modes. - A bare subquery in select type-checked, also nested, because the ref and nested-object shapes are open. Both select overloads now reject a value that holds a query builder outside include(), toArray(), or materialize(). - Guide and skill examples: a nested bare subquery, missing imports, an orderBy on $selected, and iteration over a Collection. Co-authored-by: Isaac <no-reply@databricks.com>
The autofix bot reformatted skill examples, which changed their hashes and made known failures look new. The baseline now hashes each example's tokens without commas, semicolons, parentheses, and spacing, so a reformatted example keeps its entry. An edit to its code still does not. Co-authored-by: Isaac <no-reply@databricks.com>
The conformance suite checked pooling only in the config form with pool. Two scenarios now run in every framework driver: - pool-opt-in: queries without pool keep a source subscription each in the function, config, and parameterized forms. - pooled-param-change: a pooled parameterized query follows its parameter between pool groups beside an unchanged peer, and releases its subscription on unmount. mountControllable takes an optional pool, and each driver uses its adapter's config form for it (React without deps, Angular params). Pooling every query by default, a pooled identity that ignores literal values, and an Angular params form that drops pool each fail them. Co-authored-by: Isaac <no-reply@databricks.com>
CI does not build the packages before lint. A subpath import such as `@tanstack/db-sqlite-persistence-core/broadcast-coordinator` resolved to dist/ locally and to nothing in CI, so one skill example failed only in CI. Each export in a package's manifest now maps to its source file. Co-authored-by: Isaac <no-reply@databricks.com>
- groupKeys() compiles without noImplicitAny, as join operators do. Both use one OperatorCallback type. The TypeScript-modes test now covers groupKeys and a reusable join operator, which passes its context explicitly and stays typed in both modes. - Document reusable join operators in the guide and the db-core skill. - Migrate the React Native shopping-list example to include() and the projects README to the current join API. - The changeset and skill say that the join methods are removed. They do not throw the coded error. The changeset notes the new type of resolveLiveQueryValue's pool option. - The bundle test runs every operator value alone and every combination of operator families, and checks group-by.ts exactly. - The doc example check reports typos of imported TanStack names and examples that do not parse, and keeps parentheses in its fingerprint. - Correct ARCHITECTURE.md and includes-runtime.ts on which module owns public-key reduction. - Remove dead code: containsAggregate, the pool key check in poolConfig, the joined-key parameter of materializeCompilation, and two needless exports. - Record the open load-subset replay counterexample (seed 1593300187), which fails the same way on main, in the coverage map. Co-authored-by: Isaac <no-reply@databricks.com>
- The TypeScript-modes test covers every operator and form: join and groupKeys, inline, reusable with context, and reusable without context, in both modes, with a stated result per cell. It compiles one program per mode, so it stays fast. - Document the typed reusable groupKeys<Context>() form. - Export JoinCondition and OperatorCallback, so a factory can name the condition type. - The doc example check reads examples through the TypeScript syntax tree: JSX text no longer breaks the fingerprint, every import form is found, and only syntax errors count as "does not parse". The typo check ignores case-only and prefix differences. A self-test with positive and negative cases runs before each check. - Regenerate the API reference. - Record the untraced includes-oracle failure, and the second seed of the load-subset counterexample, in the coverage map. Co-authored-by: Isaac <no-reply@databricks.com>
Joins are common, and models already know `.join(source, on)` and the `.leftJoin()` family. Making them opt-in saved about 3.6 kB in apps that never join. Grouping, includes, and pooling stay opt-in. - Restore `.join(source, on, type)` and the `leftJoin`, `rightJoin`, `innerJoin`, and `fullJoin` methods. Remove the join operator values, `JoinCondition`, the join runtime on IR join clauses, and `JoinOperatorRequiredError` (245). - Restore the static join compile path, main's optimizer, and main's joins.ts. Public-key reduction stays its own module, so a joined query with a custom key still does not bundle the include materializer. - Tests: a reverse codemod restored main's exact join call at each site. 25 files are identical to main again. - Docs and skills use the existing join API again. The doc example check matches a fence by its backtick count, so a ````ts example that holds a ``` block is read whole. - The TypeScript-modes test covers groupKeys only, and the bundle test treats join modules as core. Typical app: 102,023 to 94,419 B gzip (-7.5%). Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
changed the title
# Conflicts: # .github/workflows/pr.yml
- Pooling no longer ships in every adapter app. The live-query observer found a pooled view's source with `instanceof PooledLiveQuery`, which kept the pooled view in every app that uses a framework hook. Pooled views now register their source in a small core WeakMap. The bundle test's app includes every value the adapters import from @tanstack/db, and a test keeps that list in step with the adapters. - The aggregate-select oracle checks the operator laws over its select grammar: bare group keys without an aggregate throw 246, a bare query builder as a field, in a nested object, or in a conditional branch throws 247, and a wrapped include that is not a top-level select value throws. It declares these states with createOracleReach. - Export `groupingCompiler` and `includesRuntime` for hand-built IR. - Svelte and Vue skills wrap their child query in include(). The doc example check reads the TypeScript script of Svelte and Vue examples. - The doc example key keeps separator commas, so `f(a, -b)` and `f(a - b)` differ. The self-test uses current API names. - Restore LiveQueryCollectionConfig's JSDoc, the join JSDoc examples, and the comments in join.test-d.ts that the join revert dropped. - The guide and skill no longer call a Collection include the default. The glossary says that pooling needs `pool`, and adds the operator terms. - A test checks at the adapter boundary that queries without `pool` keep a source subscription each. It fails on the merge-base, which pools by default. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
… opt-in-operators - Renumber this PR's unreleased error codes after main's 245 and 246: GroupingOperatorRequiredError 247, IncludeOperatorRequiredError 248, and the two include() sites 249 and 250. - React and Svelte take main's restructured binding; their resolve calls keep this PR's pool option (`false` for Suspense and DbClient queries). - The conformance contract keeps both pool and queryKey. Every driver, Preact's included, keeps main's publication log and this PR's pooled parameterized form; React runs it on both identity paths. - preact-db accepts `pool` in its query config. - The bundle test scans preact-db, and its adapter value list follows the adapters' new imports. No new adapter import brings operator code. - The doc example check matches framework packages by exact name, so Preact examples import from preact-db and React ones from react-db. - Record an ordered-work counterexample that also fails on main. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
Every app bundled the code for grouping, includes, and live-query pooling, even when it used none of them. The query builder and the compiler imported that code statically. This change makes these three features opt-in. A feature's code arrives with the value that names it, so an app bundles only the features it imports. Joins stay in core and keep their API:
.join(source, on, type)and the.leftJoin(),.innerJoin(),.rightJoin(), and.fullJoin()methods do not change.count,sum,avg,min,max,groupKeysgroupBywith no aggregate takesgroupKeys(...)include,toArray,materializeissues: q.from(...)becomesissues: include(q.from(...))pooledLiveQueriespool: pooledLiveQueriesin the hook's query configHow it works:
groupKeys()carry the grouping compiler.orderByrewrites aggregates only when it finds one.includes-runtime.ts). They live ininclude-functions.ts, apart from the expression functions that core modules import.pooledLiveQueriescarries the pooled view and its identity shortcut. Hooks take it aspoolin their query config (LiveQueryHookConfig). Angular's{ params, query }form takes it too. React Suspense andDbClientqueries passpool: falseand never pool. ADbClientquery dropspoolfrom its identity, so a server preload and a client hook hash alike.GroupingOperatorRequiredError(247) for agroupBywith no aggregate and nogroupKeys()IncludeOperatorRequiredError(248) for a subquery inselect()without an include function. A bare subquery inselect, also in a nested object or acaseWhenbranch, is also a type error.include()of something that is not a query builder. Main took 245 and 246 first, so this PR's unreleased codes moved up by one.How a query with a custom key reaches public-key reduction:
reducePublicKeysfrompublic-key-reduction.ts. It does not bundle the include materializer or create a facade adapter. Onmain, this query went through the include materializer.An earlier version of this PR also made joins opt-in, with
.join(leftJoin(source, on)). That saved about 3.6 kB in apps that never join. The maintainer reverted that part: joins are common, and models already know the existing API.The architecture document records the include contract and how values carry operator code.
Size
pnpm size:compare(gzip bytes):app(query collection, filtered live query, ReactuseLiveQuery)app-joinapp-group-byapp-includesdbpackage bundle, every export keptThe baseline is
mainat5649a07d4, which includes the Preact adapter (#2235).The package-bundle figure is the cost of the indirection when an app uses every feature.
Evidence
This change moves behavior in areas that oracles own: query compilation, includes, pooling, and the adapter bindings. Each law below lists its evidence.
1. Pooling is opt-in per query
pool: pooledLiveQueries. Every other query has its own subscription. A pooled query with a parameter follows the parameter. Authority: the maintainer's design decision for this release (pooling off by default, opt-in by import).resolveLiveQueryValue.live-query-options.test.ts(keeps one source subscription per query without pool) resolves two equality-filtered queries there. On the merge-base427a7c31eit fails with "expected 1 to be 2": the two queries share one subscription, becausemainpools by default. The conformance suite itself cannot run onmain, becausemainhas nopooloption.resolveLiveQueryValue, rebuilt intodist/, which the adapters test against.pool-opt-infails in all five drivers that existed before the merge.pooled-param-changefails in React, and the oldereq-filter-peersscenario does not catch it.paramsform that dropspool:pooled-param-changefails.db-client-preload-identity-oraclefails twice.eqfilters on one field. The pooled live-query oracle owns the membership and order laws.2. An app bundles an operator's code only when it imports the operator
opt-in-operator-bundle.test.tsruns each of the 10 operator values alone and all 8 combinations of the three families, with rotating values (18 bundles). Each app also holds every value that the six framework adapters, Preact's included, import from@tanstack/db. A test fails when that list and the adapters' imports differ. These are fixed cases, not generated histories, so ORC-015 does not apply.427a7c31e, I bundled an app with the core and adapter values and no operator. It containsgroup-by.ts(8,849 B minified),pooled-live-query.ts(7,778 B),bucket-facade-adapter.ts(5,900 B), andmaterialized-pipeline.ts(3,635 B). On this branch, before the observer fix, 13 of 18 cases failed because the observer keptpooled-live-query.tsin every adapter app.group-by.tshelper from the live-query Collection builder fails 8 of 18 cases. A static import ofBucketFacadeAdapterinto the builder fails the absence checks. The observer's oldinstanceofcheck fails 13 of 18.@tanstack/dbwith other packages external, so it does not cover db-ivm operators. It takes the adapters' imports, not their bundles.pnpm size:comparemeasures bytes.3. A
groupKeys()callback compiles in both TypeScript modesgroupKeys()gets the query's context from.groupBy(). Inline, its callback is typed in strict mode and untyped withoutnoImplicitAny, as other unannotated callbacks are. Group keys that pass their context as a type argument,groupKeys<ContextFromSource<…>>(…), are typed in both modes. Authority: the maintainer's decisions in this PR's review.group-keys-typescript-modes.test.tscompiles a wrong field in three forms: inline, reusable with context, and reusable without context. It compiles each form in both modes and checks the stated result. A fixture with valid code must compile in both modes. These are fixed cases, not generated histories.groupKeys()is new, so the test cannot run onmain. Without thegroupKeys()fallback, the valid fixture fails withoutnoImplicitAny("'user' is possibly 'undefined'").OperatorCallbackthat is always untyped fails the cases that must report the wrong field.4. Examples in the docs and skills typecheck
docs/and in the shipped skills parses and typechecks against the workspace source, unless its fence saysnocheck. For a Svelte or Vue example, its<script lang="ts">block must.main. I wrote a scratch example for each defect, and the check fails each one. The defects are a bare subquery and a misspelled import. Others are code that does not parse, a missing default export, and a moved parenthesis. The Svelte skill's old bare-subquery example fails the check. A scratch example with the app namescollectionandcounterpasses, as it must.pnpm test:doc-snippets: 0 new failures. Its self-test runs first, on every check.scripts/doc-snippet-syntax.mjs) fails when the typo check ignores case, and when the fingerprint keeps the parentheses around JSX. It pinsf(a, -b)againstf(a - b), which the earlier key confused. The CI lint job once failed because a subpath import resolved to an unbuiltdist/. With everydist/hidden locally, the old script fails the same way.any. The check reports errors in the TanStack API, not in the example's own code. The typo check compares only with names that the example imports from a TanStack package. A Svelte or Vue template is not checked. A value typed as a plainCollectioncannot be passed touseLiveQueryin React, Preact, or Vue. Those examples are in the baseline as an older type gap.5. A missing operator throws a coded error
GroupingOperatorRequiredError(247) unless its keys are wrapped ingroupKeys(). A bare query builder in a select value throwsIncludeOperatorRequiredError(248) as a field, inside a nested object, or in a conditional branch. A wrapped include that is not a top-level select value throws, as before. When several errors apply, the query throws one of them. Authority: the maintainer's decisions in this PR that the operators are opt-in and that a missing operator throws.aggregate-select-oracle.property.test.ts) generates these dimensions over its select grammar. It declares six states withcreateOracleReach. They are a bare include at each of three positions, atoArray()include in a conditional branch, and bare group keys with and without an aggregate. Each campaign of 120 histories reaches each state 19 to 38 times.main. The first run of the extended model found that it had left out the existing top-level include rule. The model now cites that rule (db-core skill, "Includes rules").TypeError. Treating a bare builder as a Collection include, asmaindid, fails with the wrong error class.includes.test.tspins a bare builder as a field, nested, and in a conditional.Other tests:
include()andgroupKeys(). The assertions did not change. When joins went back to core, a reverse codemod restoredmain's join call at each site, and 26 test files are identical tomainagain. The codemod dropped comments beside rewritten arguments injoin.test-d.ts, which is now restored frommain.groupKeys()grouping, andinclude()Collections. Others cover pooling off by default,pool: false, and the exported operator code for hand-built IR.select, also in a nested object.Remaining limits:
poolcomputes the full IR hash of its query on each render. Measured on a two-conditionwherequery: building the query takes 1.1 µs, and building plus hashing takes 3.6 µs. The pooled identity adds almost nothing. For 1,000 mounted hooks, the hash adds about 2.5 ms for each render. Keeping the pooled identity in core would add 656 B gzip (1,693 B minified) to every app. Apps with many such hooks can passpool.load-subset-replay-refinement-oraclefails with seed1593300187, path121:1:2:3:2:2. The history is a held load,truncateMarkers, and a failed settle.mainat65d9ada50and the merge-base427a7c31efail with the same counterexample. A full suite run on the branch found it again with seed-1005431615, which also fails on the merge-base. The random campaign can fail CI intermittently until someone fixes the defect.includes-oraclerandom campaign failed on this branch without a recorded seed. Thirteen dedicated runs and two full-suite runs passed later. The coverage map records it as open, with the witness it needs. This PR does not call it pre-existing.buildNestedSelect,include()checks the field name before the query.toArray()checks the query first. Both throw a coded error. Changing the order costs 8 bytes in every app, so this PR keeps it. The eight wrapper blocks remain, because the error-site oracle freezes one message and code for each site.main, recorded in the coverage map:ordered-work-oraclefails with seed1023848973on an unindexed anti-joined window after a signalling refetch.mainat5649a07d4fails the same way.resolveLiveQueryValuewithpool: truethrows a plainTypeErrorat once. TypeScript rejects that value, and a coded error would add bytes to every app for a contract violation that already fails.query-db-collectionanddb-collection-e2e, and the link checker's 33 broken links, also occur onmain. The React Native example cannot typecheck in a workspace without built packages.✅ Checklist
mainat5649a07d4, and all pass:@tanstack/db: 13,830@tanstack/react-db: 625@tanstack/preact-db: 100@tanstack/vue-db: 142@tanstack/solid-db: 105@tanstack/svelte-db: 179@tanstack/angular-db: 80@tanstack/query-db-collection: 1,460@tanstack/db-sqlite-persistence-core: 1,697pnpm lint:oraclesandpnpm test:lint-rulespass.tscreports no new errors.eslintreports no errors.🚀 Release Impact
This is a breaking change for the first 1.0 release candidate. The changeset lists the migration for each feature.
This pull request and its description were written by Isaac.