Repository navigation
Deploy dbt projects and other non-DAG bundles with astro deploy bundle - #2313
Merged
Merged
Conversation
…t, and retire astro dbt cleanup The pre-deploy step produces its .astro/dbt_metadata.json and .astro/manifest.slim.json in memory, and the bundle tarball carries them at the same paths with the same bytes, through a TarOverlay on fileutil.Tar. Nothing is written into the user's tree. Every bundle deploy leaves out the sidecars an earlier CLI wrote into the tree (producer marker "astro"), enabled or not, so a stale one never ships. One from an unrecognized producer still stops an enabled deploy, and ships untouched when the step is off. With nothing left to clean, astro dbt cleanup becomes a removed-command stub, and its cleanup code, json shape and golden go. Co-Authored-By: Claude <noreply@anthropic.com>
…t of generated images The bundle judges each .astro/dbt_metadata.json and manifest.slim.json by its producer marker inside Tar's own walk (TarOverlay.Skip), rather than in a separate pre-scan: what it leaves out is exactly what the tarball would have carried (one under logs/ too), and a deploy with the step off never fails over Cosmos Boost - a missing dags/ makes the empty tarball it always did. A foreign file still refuses an enabled deploy, now even where a fresh sidecar would replace it. The predicate lives in pkg/airflowrt, a leaf both the root module and pkg/imagebuild already require. A generated image's project copy (and its survey and tag digest) leaves the same files out; a declared Dockerfile's context carrying them is warned about, naming them. Tar names entries with filepath.Rel, so a source of "." (astro dbt deploy --project-path .) names .astro/ and other dotfiles right. A directory or link where a sidecar goes fails that unit instead of colliding with it in the bundle. The dbt cleanup stub says what v2 does now, and why old files are still worth deleting by hand. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…decar paths, and keep the tarball out of the bundle UploadBundle writes the tarball to a temporary directory of its own. It used to land in the working directory, which under astro dbt deploy --project-path . is the directory being bundled, so it could carry itself. Tar asks the overlay's Skip about every entry, links and directories too, and anything at a sidecar's path that is not a regular file is left out with a warning. With the step on, a link there is replaced by the fresh sidecar; a directory cannot be, so that unit fails. A generated image leaves them out the same way, and a declared Dockerfile's survey names them for its warning. The foreign-producer refusal now covers only the .astro/ of a unit the step stamps, where the plugin reads, and runs on those few paths before anything is hashed, so it comes before any "step complete". One under dbt_packages/ or logs/ is the project's file and ships. precompute takes the sidecar directory, names and producer from airflowrt, and the marker check stream-decodes only the marker. Run splits into Discover and Units.Run so the check sees the units first. Co-Authored-By: Claude <noreply@anthropic.com>
…e gzipped file The foreign-producer refusal is decided from the paths this deploy actually writes a fresh sidecar to, after the step runs and before it reports complete; a foreign file anywhere else ships. That drops the separate pre-hash check and the Discover/SidecarDirs split. With the step off, only regular files an Astro CLI wrote are left out; links and everything else at a sidecar path ship as they are, in the bundle and in a generated image. With it on, whatever is at an artifact's path, a link or a directory too, is left out for the fresh sidecar; a unit fails only when its .astro is not a directory. The producer check goes back to unmarshaling the bytes already read. UploadBundle streams tar into gzip into one file (fileutil.TarGz) and checks every close, so a full disk fails the deploy instead of uploading a truncated bundle. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
astro dbt deploy, astro dbt delete and astro deploy --non-dags did the same upload with different defaults, target selection and JSON. They become one subcommand, astro deploy bundle [PATH], with --delete for the removal. - A table of bundle types (internal/platform/astro/deploy/bundletype.go) gives each type its detection, check and default mount path. dbt is the one entry: a directory with dbt_project.yml, mounted at /usr/local/airflow/dbt/<name>. Any other --type is a label that needs --mount-path. - One target path for deploy and delete: --deployment (a link of the project here, or an id), the picker on the current host only, and input_required when the run cannot be asked. The project's host login now reaches DeleteBundle too. - One JSON shape each for deploy and delete, pinned as deploy-bundle and deploy-bundle-delete; dbt-deploy, dbt-delete and deploy-non-dags go. - astro dbt is one hidden stub naming the replacement for each subcommand; --non-dags and its --non-dags-* flags join the removed-flag registry. - bundle is refused as a link name, and a deploy never auto-links a picked Deployment under it, since astro deploy bundle is the subcommand. Co-Authored-By: Claude <noreply@anthropic.com>
Coverage Report for CI Build 38627931420Coverage decreased (-0.007%) to 57.944%Details
Uncovered Changes
Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
…d reserve bundle in scaffold Review of the first cut: - --deployment resolves as the other commands do: a link of the project here, else a Deployment id, else a Deployment's name in the workspace (--workspace, the project's, the login's). No match, or several (listed by id), is an error. - One resolveBundle settles type, check and mount path for deploy and delete. PATH is checked only when the mount path is derived from it, so --type dbt with --mount-path labels any directory, as 1.x's --non-dags-bundle-type dbt did. A delete with no bundle at PATH takes its type from the mount path (BundleType.MountPathPrefix, also the default's directory). - The JSON gains bundle_name (a dbt project's name), from the table's BundleName; the docs map 1.x's project/project_path to it. - bundle is refused as a link name by scaffold's link validation, so every writer, Astro Desktop included, refuses it; astro link add keeps its early usage error. - No -d on deploy bundle (it is --dags on astro deploy); -t is --wait-time again, as on 1.x's astro dbt. - The astro dbt stub reads the subcommand past flags and their values. Co-Authored-By: Claude <noreply@anthropic.com>
schnie
marked this pull request as ready for review
October 14, 2026 19:40
schnie
marked this pull request as draft
October 14, 2026 19:47
…y and delete Second review: - One type rule for deploy and delete: --type, else what PATH holds, else the type whose default mount path --mount-path has the shape of (prefix and one component, after path.Clean), so the pair records the same type. - bundle_name only when the mount path is the one derived from PATH. - A Deployment name with no match, or several, is a usage error, as a mistyped link name is; the lookup (DeploymentsNamed) is shared with astro deployment create --clone. - scaffold refuses only a new link called bundle: one already in the manifest saves again. - The astro dbt stub reads a shorthand group's value (-wt 5m, -nprod). - The no-target message names ids, names and link names; the docs say the name lookup is astro deploy bundle's, as astro deploy takes links or ids. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #2311 (Cosmos Boost sidecars generated into the bundle,
astro dbt cleanupretired), which has now merged into v2. The branch merges origin/v2, so the diff shows only the bundle restructure.What changes
astro dbt deploy,astro dbt deleteandastro deploy --non-dagsall calledastrodeploy.DeployBundle/DeleteBundlewith the same inputs. They differed in their defaults, in how they picked a Deployment, and in their JSON. This PR replaces all three with one subcommand:PATH is the bundle's directory and defaults to the working directory. On deploy it must be a directory and must not be inside a project (the same
utils.Locatewalk as before).Bundle types come from a table in
internal/platform/astro/deploy/bundletype.go. Each entry hasName,Detect,Validate,BundleNameandMountPathPrefix, so adding a type means adding one entry.dbtdbt_project.ymlin PATHdbt_project.ymlpresentnamefromdbt_project.yml/usr/local/airflow/dbt/(default…/<name>)none)--mount-pathis requiredThe type is chosen like this:
--typegiven: the matching entry is used (case-insensitive). An unknown value is used as a free-form label.--type: the first entry whoseDetectmatches PATH is used.--mount-path: the entry whose default it has the shape of (MountPathPrefixplus one path component, afterpath.Clean).none.One
resolveBundlehelper settles type, check and mount path for both deploy and delete, so a deploy and a delete of the same bundle record the same type. PATH is validated only when the mount path is derived from it. With--mount-pathgiven,--type dbtaccepts any directory, as 1.x's--non-dags-bundle-type dbtdid.--deleteuses--mount-pathif given. Otherwise it uses the mount path PATH's type would deploy to.--deploymentis resolved in the order the other commands use:--workspace, else the project's[tool.astro] workspace, else the login's.A name with no match, or with several matches (listed by id), is a usage error, as a mistyped link name is. The name lookup is specific to
astro deploy bundle(1.x's dbt commands had--deployment-name);astro deploytakes links or ids. It sharesdeployment.DeploymentsNamedwithastro deployment create --clone. There is no-dshorthand, because-dis--dagsonastro deploy.Without
--deployment, a picker lists Deployments, but only on the current context's host. Under-o jsonor with no terminal the run fails withinput_required.Another host: from a project on another host, both deploy and delete run under that host's login (
DeleteBundleInputgainsLogin).--waitusesWaitForBundle/WaitForBundleInunder that login.Routing:
bundleis a subcommand ofastro deploy, soastro deploy bundlenever means a Deployment named bundle.pkg/scaffoldrefuses a new link namedbundle(BundleLinkName, checked inSaveLink's edit against the manifest), so every writer refuses it, including Astro Desktop. A link already namedbundlesaves again.astro link add bundlekeeps an early usage error with the same message.bundle.bundlestill loads and is reached withastro deploy --deployment bundle, and the help says so.Astro Desktop:
deploylinks.NewAstroLinkNamederives names withscaffold.LinkNameForand skips onlylocal. A Deployment named "Bundle" would be offered the namebundle, whichSaveLinknow refuses. That is correct, but Desktop should addscaffold.BundleLinkNameto the names it skips, as the CLI'snewLinkNamedoes.Old to new
astro dbt deploy [ID] --project-path P --deployment-name N --workspace-id Wastro deploy bundle [P] --deployment <id, name or link> --workspace W(-m,-w,-tunchanged)astro dbt delete [ID] ...astro deploy bundle --delete [P] --deployment ...astro dbt cleanupastro dbt(group)cliout.RemovedCommandstub; it reads the subcommand past flags and their values (astro dbt -n prod delete)astro deploy --non-dagsastro deploy bundle [PATH]--non-dags-local-path--non-dags-mount-path--mount-path--non-dags-bundle-type--typeThe
--non-dags*flags are entries incmd/removed_flags.go.TestEveryV1CommandStillRunsOrSaysWhatReplacedItandTestEveryV1FlagStillWorksOrSaysWhatReplacedItpass. No hidden aliases were needed.JSON
deploy-bundle.json:deployment, deployment_name, workspace, deploy_id, bundle_type, bundle_name?, bundle_path, mount_path, bundle_version, waited, wait_error?, git?deploy-bundle-delete.json:action, deployment, workspace?, deploy_id, bundle_type, bundle_name?, mount_path, waited, wait_error?bundle_nameis published only when the mount path is the one derived from PATH.The goldens
dbt-deploy,dbt-deleteanddeploy-non-dagsare removed.astro dbt deploy -o json'sproject/project_pathare nowbundle_name/bundle_path, besidebundle_type; this is documented in docs/deploy.md and docs/upgrading-from-v1.md. The Emit floor drops from 52 to 51: three shapes became two.Tests
--type dbt/DBT, free-form types,BundleTypeForMountPath--type dbtwith and without--mount-pathon a non-dbt directorybundle_typedbt--deployment: link, id, a name with one match (deploy and delete), no match, two matches;input_required; the picker; another host's login; the picker refused on another hostbundle_name), and--waitin text and JSONdeploy bundlevs a positional idbundlerefused inscaffold.SaveLinkand inlink add, and skipped bynewLinkNameastro dbtstubs, including--workspace-id ws1 deploy,-n prod deleteand-wt 5m deploy../and nested mount paths;bundle_nameonly for the derived mount path; no-match and ambiguous names as usage errors under-o json; re-saving an existingbundlelink--non-dags*flagsGates
make test,make lint,make lint-goos,make lint-submodules,make lint-e2e,make deadcode(clean),make test-e2e(tier 0),make update-schemas, andGOOS=windows go build/vet ./...for the root module and pkg/scaffold all pass.make test-submodulespasses for pkg/scaffold. Locally it fails only onpkg/airflowrtTestHasProxyConfigured, which reads the environment's proxy variables (the sandbox sets them). This PR does not touch that module.