bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations by denik · Pull Request #6954 · databricks/cli

denik · 2026-10-06T22:39:05Z

Changes

Bundle configuration no longer goes through libs/dyn; the package is deleted.

config.Root is the typed struct plus a structvar side: pure ${...} references in non-string fields and a tree of source locations (one node per YAML value). YAML is decoded straight into the structs, with the same warnings as before.
Mutators read through structvar.View and write with Set/Merge/Override/… instead of converting the whole configuration to a dyn.Value and back. Paths are structpath everywhere, including diag.Diagnostic.Paths.
Variable resolution, ${...} parsing and the resource-reference resolver run on typed values (structvar.Resolve, same semantics and error text as dynvar).
Generate commands write YAML with the new libs/structs/structyaml (identical output); --json and variable-overrides.json use structvar.ParseJSON (exact int64, same error positions).
Diagnostics about values set by mutators fall back to the location of the nearest parent.

Stacked on #6974 (diag.Location), which is stacked on #6980 (structpath diagnostic paths).

Why

Converting the configuration typed↔dyn around mutators dominated command time on large bundles.

./task bench-1k-summary (1000 jobs, wall seconds, DMS= variant), main → this PR:

benchmark	wall	user CPU	peak RSS
validate	12.6s → 1.29s (9.8x)	21.1s → 2.0s	655MB → 183MB
plan	19.2s → 1.77s (10.8x)	25.5s → 2.9s	670MB → 179MB
deploy (1000 creates)	16.4s → 2.33s (7.0x)	33.4s → 8.3s	676MB → 266MB

(deploy/DMS=true at 1000 jobs fails on main too: the test server caps a deployment version at 800 operations.)

Tests

Existing unit and acceptance tests; one golden changed (path quoting above). Parity tests against libs/dyn (decode, view, writes, merge/override, walk) passed on every bundle fixture before dyn was deleted.

This pull request and its description were written by Isaac.

eng-dev-ecosystem-bot · 2026-10-07T00:11:25Z

Integration test report

Commit: d08081b

Run: 37922885476

	Env	✅pass	🙈skip	Time
✅	aws linux-2core-8gb	276	17	4:10
✅	aws-windows-latest-4core-16gb	278	15	3:37
✅	azure linux-2core-8gb	275	17	5:13
✅	azure-windows-latest-4core-16gb	277	15	4:52
✅	gcp linux-2core-8gb	276	17	4:09
✅	gcp-windows-latest-4core-16gb	278	15	4:05

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
4:03	gcp-windows-latest-4core-16gb	TestAccept
3:35	aws-windows-latest-4core-16gb	TestAccept
3:33	azure-windows-latest-4core-16gb	TestAccept

## Changes `diag.Diagnostic.Paths` is `[]*structpath.PathNode` instead of `[]dyn.Path`. Producers convert existing dyn paths with `dyn.ToStructPaths` or build paths from keys with `structpath.NewStringKeys`/`NewStringKeysSlice`/`MustParsePaths`. `structpath.PureReferenceToPath` moves to `dynvar.PureReferenceToStructPath` so structpath does not import dyn. User-visible: diagnostic paths quote map keys that are not identifiers, e.g. `resources.jobs['${var.env}_job']` instead of `resources.jobs.${var.env}_job` (one golden). ## Why First step of removing `libs/dyn` from bundle configuration: databricks#6974 moves `Location` to `libs/diag` on top of this (it needs diag to no longer import dyn), then databricks#6954. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

…r refs and locations Bundle configuration is the typed struct plus structvar (pure ${...} references in non-string fields and a tree of source locations). YAML and JSON decode straight into the structs; mutators, validators, variable resolution, configsync and generate work on structvar views and structpath paths. libs/dyn is deleted. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

… test Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

- configsync: compare int and int64 values as equal when restoring variable references (variables decode as int, remote fields are int64). - schema annotations: merge object/sequence defaults as documented; accept an empty annotations file. - python mutator: use ResourceKeySet.AddPattern instead of a duplicate helper. - structvar.Set: don't rewrite an unchanged leaf location node (avoids cloning every ancestor per write). - ResolveConfigAgainstState: resolve all references in one call so lookups are shared. Co-authored-by: Isaac <no-reply@databricks.com>

github-actions Bot added DABs DABs related issues PyDABs labels Oct 6, 2026

denik force-pushed the denik/bundle-struct-no-dyn branch from 13fac18 to b2a1c10 Compare October 7, 2026 09:41

github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 7, 2026

denik changed the title ~~bundle: make the typed configuration the source of truth (no per-mutator dyn round trip)~~ bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations Oct 7, 2026

denik mentioned this pull request Oct 7, 2026

diag: own Location (move from libs/dyn) #6974

Merged

denik force-pushed the denik/bundle-struct-no-dyn branch from 6ee2caa to 68af62f Compare October 7, 2026 12:40

denik changed the base branch from main to denik/diag-location-structpath October 7, 2026 12:40

denik force-pushed the denik/bundle-struct-no-dyn branch from baccaac to 7086a71 Compare October 7, 2026 14:08

denik mentioned this pull request Oct 7, 2026

diag: use structpath for diagnostic paths #6980

Merged

denik force-pushed the denik/bundle-struct-no-dyn branch from 7086a71 to 4c4f0b3 Compare October 7, 2026 14:33

denik force-pushed the denik/bundle-struct-no-dyn branch from 4c4f0b3 to 3bb81b3 Compare October 7, 2026 14:38

denik force-pushed the denik/bundle-struct-no-dyn branch 2 times, most recently from 722c97c to 649bde3 Compare October 7, 2026 16:17

denik force-pushed the denik/bundle-struct-no-dyn branch from 649bde3 to 67d1321 Compare October 7, 2026 21:11

denik mentioned this pull request Oct 7, 2026

structpath: build computed paths and patterns from parts; lint rule #6984

Draft

denik force-pushed the denik/bundle-struct-no-dyn branch from 67d1321 to d49f829 Compare October 8, 2026 07:52

denik force-pushed the denik/bundle-struct-no-dyn branch from d49f829 to b22dd28 Compare October 8, 2026 11:18

denik force-pushed the denik/bundle-struct-no-dyn branch from b22dd28 to 799731f Compare October 8, 2026 11:34

denik force-pushed the denik/bundle-struct-no-dyn branch from 799731f to 13de66e Compare October 8, 2026 16:09

denik force-pushed the denik/bundle-struct-no-dyn branch from 13de66e to 6c910dc Compare October 8, 2026 16:31

denik force-pushed the denik/bundle-struct-no-dyn branch from 6c910dc to 9746279 Compare October 8, 2026 16:35

denik added this pull request to stack #6997 October 9, 2026 08:18

denik force-pushed the denik/bundle-struct-no-dyn branch 2 times, most recently from 12ad72d to 44a59f3 Compare October 9, 2026 08:44

Base automatically changed from denik/diag-location-structpath to main October 9, 2026 10:44

denik and others added 14 commits October 9, 2026 12:44

structvar: map YAML locations in the YAML source; drop mappedSource

7de52a7

Co-authored-by: Isaac <no-reply@databricks.com>

Use NewStringKeysSlice for single key paths

15e5521

Co-authored-by: Isaac <no-reply@databricks.com>

Build literal diagnostic paths with NewStringKeysSlice

431e44c

Co-authored-by: Isaac <no-reply@databricks.com>

Restack fixes: structvar grammar for structpath's PureReferenceToPath…

85a558d

… test Co-authored-by: Isaac <no-reply@databricks.com>

Restore import block after restack

a051fc6

Co-authored-by: Isaac <no-reply@databricks.com>

Use structpath.NewPath for key/index paths

6c96ca4

Co-authored-by: Isaac <no-reply@databricks.com>

acc: name the ids in bundle/run/job-ref-job-id (after #6975)

1d39642

Co-authored-by: Isaac <no-reply@databricks.com>

Build the target preset path in one NewPath call

0f5c273

Co-authored-by: Isaac <no-reply@databricks.com>

GetLocationsOf on the location tree

6d2083b

Co-authored-by: Isaac <no-reply@databricks.com>

Minimal import edits; drop unused diag import

9d0aa6f

Co-authored-by: Isaac <no-reply@databricks.com>

structvar: test the YAML decoder with the yamlloader fixtures

6b10470

Co-authored-by: Isaac <no-reply@databricks.com>

structvar: add the YAML test helpers

6c50307

Co-authored-by: Isaac <no-reply@databricks.com>

Port the job environment_variables files pattern (#6985) to structpath

b210f66

Co-authored-by: Isaac <no-reply@databricks.com>

denik force-pushed the denik/bundle-struct-no-dyn branch from 44a59f3 to b210f66 Compare October 9, 2026 10:44

Navigation Menu

Repository navigation

bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations - #6954

bundle: delete libs/dyn; configuration is typed structs with structvar refs and locations#6954
denik wants to merge 15 commits into
mainfrom
denik/bundle-struct-no-dyn

denik commented Oct 6, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

denik commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

denik commented Oct 6, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 7, 2026 •

edited

Loading