Repository navigation
Deploy only pyproject.toml projects in v2 - #2295
Merged
Merged
Conversation
v2 deploys only projects with a pyproject.toml, on Astro and on Astro Private Cloud. Astro CLI 1.x keeps deploying the projects it made, so v2 drops the 1.x deploy rather than carrying it. Astro: astro deploy in a 1.x-layout project (a Dockerfile beside .astro, or a .astro/config.yaml) fails with no_project, saying to convert it with astro init or deploy it with Astro CLI 1.x. Outside any project it gives the no-project advice, now also no_project. --non-dags runs first, from anywhere: it deploys a separate directory, and from a pyproject.toml project it no longer runs a whole project deploy instead. APC: v2 builds no project. A deploy without --image-name is refused as a usage error saying to use Astro CLI 1.x (and, from a pyproject.toml project, that support is coming). --image-name (and --remote) deploy from anywhere but a 1.x project; --dags uploads a pyproject.toml project's dags/. Any deploy from a 1.x project is refused. The 1.x deploy is deleted with its tests: deploy.Deploy and its build, pytest and parse steps, the named DAG bundle create, APC's Dockerfile build, airflow's DAGChecker and ImageHandler.Pytest, and what deadcode found behind them. The flags only it read are removed and tombstoned in the removed-flags registry: --save, --pytest, --env, --test, --parse, --deployment-name, --dags-path and --dag-bundle-name on Astro, --save and --no-cache on APC. From #2290: config.IsHomeDir in IsProjectDir, the home-directory advice, and the bound-value --image-name check. Co-Authored-By: Claude <noreply@anthropic.com>
Coverage Report for CI Build 38587833118Coverage decreased (-0.2%) to 57.896%Details
Uncovered Changes
Coverage Regressions19 previously-covered lines in 6 files lost coverage.
Coverage Stats
💛 - Coveralls |
From review of the first commit: - astro deploy --image-name runs from any directory again, a 1.x project's included: it reads nothing from the project. Outside a pyproject.toml project it ships the image alone (no dags/), to an id named by the argument or --deployment, or the workspace's pick. - --non-dags validates and honours --output (one json object, pinned as deploy-non-dags), names its target as a deploy does (link or id, the argument and --deployment must agree) and reads --workspace. - APC deploy no longer falls back to the project.deployment a 1.x .astro/config.yaml saved; a build outside any project is no_project, as on Astro; the 1.x refusal no longer talks about building. - What a directory is comes from project.Discover, so a deploy below a 1.x project names it, and one below a pyproject.toml project says to run from its root. utils.Is1xLayout and EnsureProjectDir are gone; one helper gives every refusal, in plain text. - astro remote deploy follows the same rule, with --image-name anywhere. - astro deploy --prompt is removed and tombstoned. --force stays, hidden and read by nothing, because astronomer/deploy-action passes it. - The unreachable errNoImageName check and ansi.Red go. Co-Authored-By: Claude <noreply@anthropic.com>
…latforms From the second review: - --image-name follows one rule on Astro and APC: refused in or below a 1.x project and below a pyproject.toml project's root, as every deploy there is; the project's deploy at its root; and outside any project the image alone. There, Astro warns when the Deployment takes DAG deploys (stderr, and a warnings field in the deploy result), and APC uploads no dags/ and says the DAGs were not updated. - utils.Locate is the one walk astro deploy, remote deploy and APC deploy decide by: a pyproject.toml with [tool.astro] stops it, a tooling-only one does not, a Dockerfile beside .astro is a 1.x project except in the home directory, and an unreadable directory holds neither. No project.Discover or hostname derivation. - The flag combinations 1.x refused come back where they apply: --wait-time without --wait, --no-dags-base-dir with a deploy that ships no DAGs, and --dags with --image/--image-name naming only those given. - --non-dags takes a Deployment id without reading the manifest, and a link under the login for the project's host. - An image-only deploy with no target asks for an id, not a project link. - astro config get and set refuse project.deployment, which nothing reads. - The APC --no-cache tombstone points at Astro CLI 1.x only. Co-Authored-By: Claude <noreply@anthropic.com>
- --prompt and -p stay on astro deploy, hidden and read by nothing, as a pyproject.toml project's deploy always took them; the tombstone goes. - utils.IsManifestRoot is the one test for a project's root, and the deploy routes on Locate: a pyproject.toml that fails to parse is a root only when its text declares tool.astro, one that cannot be read is a root (as project.HasManifest has it), and a directory that cannot be looked in is passed over. Each ancestor's pyproject.toml is read once, and NoDeployableProject takes the walk's answer instead of walking again. - astro config list leaves out the removed keys get and set refuse. - --non-dags from a project deploys a bare Deployment id under the project's host, as a link already was; its bundle-path check is the walk. - An --image-name deploy outside a project to a Deployment without DAG deploys warns that it runs the DAGs inside the image. - APC: an image deployed alone from outside a project skips the uncommitted-changes check, and --dags with --image-name is refused in astro deploy's words. Co-Authored-By: Claude <noreply@anthropic.com>
- --non-dags under --output json sends what the platform prints (the git note) to stderr, so stdout is the one result object. - --non-dags naming no Deployment is refused with input_required when the run cannot be asked, as astro deploy is, instead of opening the picker. - Its login follows astro deploy from the same directory: the project's, for ids, links and the picker, in a pyproject.toml project that loads; the current context's anywhere else. The picker is offered only on the current context's host. - astro dbt deploy's containment check is the walk too, so it and --non-dags agree; config.IsWithinProjectDir and isWithinManifestProject go. - The walk carries the manifest it loaded (or why it did not), which the deploy and --non-dags use instead of loading it again, and looks the home directory up once. - nonDagsDeployJSON is built from the result directly; cmp.Or replaces a local helper. Co-Authored-By: Claude <noreply@anthropic.com>
Brings in #2285, #2291, #2297, #2300-#2310 and #2306. - cmd/astro/deploy.go: v2's prompt (#2305: the default link alone preselected, no deployPreselect, the ASTRO_DEPLOYMENT note), the engine memo and ListDeployments (#2301, #2306), and the deploy result's dags_in_image and remote_execution beside this branch's warnings. The unlinked pick takes v2's signature and this branch's errPickerOtherHost. The ASTRO_DEPLOYMENT note returns early outside a project, where there is no manifest. - manifest_image.go: both sides' result fields. #2306 refuses an image-only deploy to a Deployment with DAG deploys off; an --image-name deploy from outside any project (ImageAlone) is let through, as this branch intends, with its warning that the Deployment runs the image's DAGs. - apc/deploy_test.go: this branch's tag-and-push test; the Dockerfile build test #2297 made portable is gone with the build. - docs: v2's prompt and bundle-delete text, this branch's "deploy needs a converted project", and a row for --image-name outside a project with DAG deploys off. Co-Authored-By: Claude <noreply@anthropic.com>
utils.Locate had its own 1.x rule: a Dockerfile beside a .astro directory, except in the home directory. astro init and the local commands' advice use scaffold.Is1xProject (#2285): a Dockerfile and a .astro/config.yaml that is not the CLI's own settings file. The two disagreed on a home directory that is a 1.x project (HOME=/usr/local/airflow in a 1.x image), which init refused and the deploy took for no project, and on a Dockerfile beside a .astro with no config.yaml, which the deploy refused as 1.x and init would convert as a fresh directory. Locate now asks scaffold.Is1xProject, and homeMatcher and has1xLayout go. TestLocateAndInitAgreeOn1xProjects runs both walks over the same layouts. The 1.x fixtures in cmd/astro write a .astro/config.yaml with a project key. Co-Authored-By: Claude <noreply@anthropic.com>
…oject.toml once Locate passed over a pyproject.toml that failed to parse unless its text declared tool.astro, where scaffold.Find1xProject stops at any manifest scaffold.HasManifest sees, broken ones included. In a 1.x checkout with a broken tooling pyproject.toml the deploy refused it as 1.x while init reported the manifest. Locate's root is now scaffold.HasManifest's answer, and the deploy reports that file's error; the declaresToolAstro heuristic goes. scaffold exports Has1xLayout and ManifestPresent, the two halves Is1xProject and HasManifest are made of, so the walk loads each directory's pyproject.toml once and keeps scaffold the one definition. TestLocateAndInitAgreeOn1xProjects gains the broken-pyproject layouts. Co-Authored-By: Claude <noreply@anthropic.com>
…deploy warning in -o json - An --image-name deploy outside a project that picks a Deployment from the workspace no longer tries to save a link: there is no pyproject.toml to save it into (linkedTarget with no manifest). - DeployManifestImage reports whether the Deployment runs remote execution, so an image shipped alone to a remote-execution Deployment with DAG deploys off is not told it runs the DAGs inside the image. - Request.Warn appends to the result's warnings as it prints them, so -o json carries every warning text prints on stderr (#2306's DAG checks, the build checks, a link not saved), not only the image-alone one. - TestDeployImageNameOutsideAProject asserts ImageAlone reaches the transport. Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The decision
v2 builds and deploys only
pyproject.toml(manifest) projects, on Astro and on Astro Private Cloud. Nobody is forced onto v2 and Astro CLI 1.x keeps working, so v2 does not carry a deploy for the 1.x project layout (Dockerfile+.astro/config.yaml). A 1.x-layout project is converted withastro init, or deployed with Astro CLI 1.x.What a directory is, and where the deploy routes, comes from one walk,
utils.Locate. It is used byastro deploy,astro remote deploy, APC's deploy, and the bundle-path checks of--non-dagsandastro dbt deploy; it carries the manifest it loaded, so routing reads each file once. From the working directory up, at each directory:utils.IsManifestRoot) stops the walk;scaffold.Is1xProject(Refuse to convert a 1.x project under an APC context, and give the same advice everywhere #2285), the one definitionastro initand the local commands' advice use: a Dockerfile and a.astro/config.yamlthat is not the CLI's own settings file. A stray~/Dockerfilebeside the global config is not one; a home directory that is a 1.x project is;A root is what
scaffold.HasManifest(and soastro init's walk,scaffold.Find1xProject) calls one: apyproject.tomlwhose[tool.astro]loads, or one that fails to validate, to parse or to be read, so the deploy reports why. Only apyproject.tomlthat parses with no[tool.astro](a monorepo root's[tool.ruff]) is passed over. Each ancestor'spyproject.tomlis loaded once: the walk uses scaffold's two halves,ManifestPresentandHas1xLayout, rather thanIs1xProject, which would load it again.Behaviour
Astro (
astro deploy)--non-dags--image-name <ref>dags/includedthis project uses the Astro CLI 1.x layout (a Dockerfile and .astro/config.yaml), and Astro CLI v2 deploys only pyproject.toml projects. Convert it with astro init, or deploy it with Astro CLI 1.x; from below:this directory is inside a project at <dir> that uses …. Convert it with astro init in <dir>, …--image-name X --force) would silently leave its DAGs stalethis directory is inside the project at <dir>. Run the deploy from the project directory, <dir>this is not an Astro project directory. Change to an Astro project directory, or run astro init …(in~, no init advice)dags/, no git commit; target by id (argument /--deployment) or the workspace pick; non-interactive with no target:pass the Deployment id as the argument or with --deployment. Always warns, on stderr and in the result's newwarningsfield: with DAG deploys, that the Deployment keeps its DAGs; without, that it runs the DAGs inside the image (APC's wording)--non-dagsruns from anywhere, before the walk.--outputis validated (a bad value is a usage error, exit 2).-o jsonpublishes one object (new goldendeploy-non-dags.json), and what the platform prints on the way (the git note) goes to stderr.astro deploy <id>from the same directory chooses it. In a pyproject project here whose manifest loads, the project's login (its host) is used for everything: ids, links and the picker. Outside a project, or when the manifest does not load, the current context is used for everything, so a brokenpyproject.tomldoes not stop an id-targeted deploy.--deployment, which must agree. With no name, a run that cannot be asked (-o json, no terminal) gets the sameinput_requiredrefusal asastro deploy. One that can is offered the picker (--workspace/--workspace-id/ the context) only on the current context's host; on another host it is told to pass--deployment <id>or switch context.astro dbt deploy.Flag combinations, refused as usage errors before anything is read:
--wait-timewithout--wait(all modes, as 1.x did);--no-dags-base-dirwith a deploy that ships no DAGs (--image, or--image-nameoutside a project; at a project root--image-nameships the project's DAGs, so the flag applies there);--dagswith--image/--image-namenames only the flags given.astro remote deployfollows the same walk: build at a pyproject root;--image-nameat a root or outside any project; refused in/below a 1.x project and below a root.Astro Private Cloud (
astro deployunder an APC context)--image-name, or--image-name=)--image-name/--remote--dagsAstro CLI v2 cannot build and deploy projects to Astro Private Cloud yet: use Astro CLI 1.x for now. Support for pyproject.toml projects on Astro Private Cloud is coming. …dags/to a Deployment that takes uploadsdags/… uses the Astro CLI 1.x layout (…), which Astro CLI v2 does not deploy to Astro Private Cloud. Deploy it with Astro CLI 1.x… use Astro CLI 1.x for now. To deploy an image you built yourself, pass --image-namedags/upload and no uncommitted-changes check (nothing is read from the directory); when the Deployment takes DAG uploads (or that could not be read) it says the DAGs were not updated, whatevershow_warningsis--dagswith--image-nameis refused as a usage error, in Astro's words:--dags deploys only your DAGs; drop --image-name.The target is the argument or the picker. The
project.deploymenta 1.x.astro/config.yamlsaved is no longer read (a test pins a stale one being ignored), andastro config set/get project.deploymentsay it was removed (the existing removed-config-key convention; the key stays registered, asdev.modedoes).astro config listleaves out every removed key, so list and get agree. Every refusal comes before Houston is asked anything.Removed code
deploy.Deployand everything only it reached (image build,.dockerignorerewriting, pytest/parse steps, the v1alpha1 named-bundle create,ValidRuntimeVersion,WarnIfNonLatestVersion,finalizeDeploy,deployDags); APC's Dockerfile build and FROM validation;airflow.DAGChecker,DAGCheck+ mock,ProjectNameUnique,ImageHandler.Pytest;docker.GetImageTagFromParsedFile;fileutil.GetFilesWithSpecificExtension;util.Exists;ansi.Red;utils.EnsureProjectDir/EnsureDockerfileProjectDir(replaced byutils.Locate+utils.NoDeployableProject, plain text);config.IsWithinProjectDirandisWithinManifestProject(the bundle deploys' containment checks are the walk now); unused testfiles. Net: 70 files, +2297 / −4534.Removed flags (tombstoned in
cmd/removed_flags.go,under: deploy)--save,-s--deployment;astro link add+default = trueto preselect--save,-s--pytest,--test(-t),--env(-e)uv run pytest && astro deploy--parseastro local check && astro deploy--dags-pathastro deploy --dagsfrom the project--dag-bundle-name--deployment-name,-n--deployment--no-cache--force(-f) and--prompt(-p) stay on Astro deploy, hidden and read by nothing. astronomer/deploy-action v0.16.0 passes--forceon every deploy (TestDeployManifestDeployActionInvocations), and a pyproject project's deploy always accepted both.Carried over from #2290 (superseded)
#2290 is superseded by this PR. Carried over:
config.IsHomeDirinIsProjectDir(with its test); the home-directory advice; reading--image-nameby value (--image-name=is a build). Dropped as moot:EnsureDockerfileProjectDir, the APC--saveguard, the APC--dagsacceptance logic.Merged with v2 (#2285, #2291, #2297, #2300–#2310, #2306)
A merge commit, not a rebase. Conflicts and how they were resolved:
cmd/astro/deploy.godeployPreselectgone, a note whenASTRO_DEPLOYMENTis ignored; #2301:ListDeploymentsandResolveUnlinkedreturning aDeployment; #2306: the engine memo,dags_in_image/remote_executionin the resultwarningsin the result,errPickerOtherHost, nogit/ 1.x patherrPickerOtherHostinResolveUnlinked. TheASTRO_DEPLOYMENTnote returns early with no manifest (an--image-namedeploy outside a project), where it would have dereferenced nilinternal/platform/astro/deploy/manifest_image.goDagsInImage,RemoteExecutionDagDeployEnabledinternal/platform/apc/deploy/deploy_test.godocs/deploy.mddocs/upgrading-from-v1.mdastro deploystill deploys an unconverted 1.x project"astro deployneeds it converted too"One semantic fix the merge needed: #2306 refuses an image-only deploy to a Deployment with DAG deploys off (
--image, as 1.x did). An--image-namedeploy from outside any project is also image-only, and this PR deploys it there with the warning that the Deployment runs the image's DAGs.ImageAlonenow reaches the transport (ImageDeploy.ImageAlone,ManifestImageDeployInput.ImageAlone), andcheckDeploymentrefuses only a project's--image. Test:TestDeployManifestImage_DagDeployOffTakesAnImageFromOutsideAProject(fails without the fix);docs/deploy.md's DAG-deploys-off table gains the row.The walk now uses
scaffold.Is1xProjectutils.Locate's own rule (has1xLayout(d) && !isHome(d)) is replaced byscaffold.Is1xProject(d);homeMatcherandhas1xLayoutare gone. The deploy's refusal and init's/the local commands' 1.x advice now agree, which they did not on two layouts: a home directory that is a 1.x project (HOME=/usr/local/airflow), which init refused and the deploy took for no project, and a Dockerfile beside a.astro/with noconfig.yaml, which the deploy refused as 1.x and init treats as a fresh directory.TestLocateAndInitAgreeOn1xProjectsrunsLocateandscaffold.Find1xProjectover the same layouts; the home tests now use a real settings file, plus a home-that-is-1.x case. The 1.x fixtures incmd/astro(dbt,--non-dags) write.astro/config.yamlwith aproject:key. A follow-up commit (7b75665) closes the last difference:Locatepassed over apyproject.tomlthat failed to parse unless its text declaredtool.astro, whereFind1xProjectstops at it. In a 1.x checkout with a broken toolingpyproject.tomlthe deploy refused as 1.x while init reported the manifest; now both stop there and the deploy reports the parse error (thedeclaresToolAstroheuristic is gone). It also loads eachpyproject.tomlonce per level, throughscaffold.ManifestPresentandscaffold.Has1xLayout, newly exported halves ofHasManifestandIs1xProject.TestLocateAndInitAgreeOn1xProjectscovers the broken-pyproject.tomllayouts.Review fixes (667fbd5)
--image-namedeploy outside a project that picks a Deployment from the workspace saves no link: there is nopyproject.toml(linkedTargetwith no manifest).TestRun_ImageAloneOutsideAProjectLinksNothing: no save attempted, no warning.DeployManifestImagereportsRemoteExecutionEnabled, so an image shipped alone to a remote-execution Deployment with DAG deploys off is told its DAGs are its remote agents', not "runs the DAGs inside the image".TestDeployImageNameOutsideAProjectcovers DAG deploys on/off × remote execution on/off and assertsImageAlonereaches the transport.Request.Warnappends to the result'swarningsas it prints to stderr, so-o jsoncarries every deploy warning (Keep secrets out of shipped images, and deploy to DAG-deploy-off Deployments #2306's DAG checks, the build checks, a link not saved), not only the image-alone one.TestDeployManifestJSONCarriesTheTransportsWarnings.make update-schemas: no change, the shape is the same.Gates after the merge:
go build ./...,GOOS=windows go build/vetfor the root module, everypkg/*module ande2e(-tags e2e);make test;make test-submodules(onlypkg/airflowrt'sTestHasProxyConfiguredfails, which reads this Mac's system proxy throughscutil, as #2306 noted);make lint,make lint-submodules,make lint-goos,make lint-e2e,make deadcode,make fmt;make test-e2e(tier 0);make update-schemas(no diff:deploy.jsonhasdags_in_image,remote_executionandwarnings;deploy-non-dags.jsonunchanged);go test -race -shuffle=on -count=3 ./cmd/... ./internal/project/... ./internal/deploy/... ./internal/platform/astro/deploy/....Merge-tree against the open PRs (at 09cbaad, before the merge)
init-keep-apc-dockerfilecmd,cmd/astro,cmd/apc,cmd/local,cmd/utilstests passremoved-commands-and-bundle-outputcmd(both removed-flag guards),cmd/astro,cmd/apc,cmd/utilstests passproject-dir-advice-corecmd/apc/deploy.go,cmd/astro/deploy.go,cmd/astro/deploy_test.go,cmd/astro/remote_test.go,cmd/utils/utils.go,cmd/utils/utils_test.go,config/config.go,config/config_test.go(superseded)render-backticksairflow/docker.go,cmd/apc/deploy.go,cmd/astro/deploy.go,cmd/removed_flags.go,internal/platform/astro/deploy/deploy.go; take this side (already plain)deploy-ships-project-filescmd/apc/deploy.go,cmd/astro/deploy.go,cmd/astro/testdata/schema/deploy.json,internal/deploy/deploy.go,internal/platform/astro/deploy/manifest_image.go(both add to the deploy result and its routing; the golden needs both fields)Docs
docs/deploy.md(the walk, a per-location table,--image-nameoutside a project,--non-dags, flag combinations, remote deploy, an APC section),docs/upgrading-from-v1.md("v2 requires converting your project" near the top, removed flag rows,project.deploymentgone),docs/install.md,README.md,docs/architecture.md.Checks
go build ./...,GOOS=windows go build ./...,GOOS=windows go vet(touched root packages;e2ewith-tags e2e),make test,make test-e2e(tier 0),make lint(no autofix changes),make lint-submodules,make lint-e2e,make lint-goos,make deadcode(clean),make update-schemas(deploy-non-dags.jsonnew;deploy.jsongainswarnings).Tests cover:
-o jsonwith their kinds: at a root, below it, in/below a 1.x project (incl.--image-name), and in~;~with pyproject + Dockerfile +.astro→ not 1.x; non-ASCII names; an unreadable ancestor;--image-nameoutside a project: no DAGs, the DAG-deploy warning, the no-target message;--non-dagsoutput / agreement / links / id with a broken manifest, the stale saved APC deployment,config get/set project.deployment, remote deploy's rule, and a case per removed-flag entry.e2e tier 0:
TestDeployRefusesThe1xOnlyFlags,TestDeployRefusesA1xProject,TestDeployOutsideAProject,TestDeployImageNameOutsideAProject(outside: past the project check to the container engine, offline; 1.x: refused, no_project),TestAPCDeployRefusesToBuild(against a refusing Houston).Follow-ups
isCLISettingscontent heuristic (Refuse to convert a 1.x project under an APC context, and give the same advice everywhere #2285) for telling the home directory's.astro/config.yamlfrom a 1.x project's now decides deploys too: a homeconfig.yamlwith noprojectkey and none ofcontext,contexts,telemetrybeside a Dockerfile is refused as a 1.x project.Open questions
--forceon Astro deploy: drop it once deploy-action stops passing it?cosmos_boost.pre_deploynow affects only bundle and dbt deploys.