Generate Cosmos Boost sidecars into the deploy bundle, not the project, and retire astro dbt cleanup

[schnie]

Why

With cosmos_boost.pre_deploy on, every bundle deploy wrote .astro/dbt_metadata.json (and .astro/manifest.slim.json beside each dbt manifest) into the user's dbt project. That left generated files in the repo to commit or ignore. Turning the setting off left them in place, and the next deploy shipped them stale, which is why astro dbt cleanup existed.

The sidecars now go into the deploy bundle only. Nothing is written into the project, so there is no repo churn and nothing to clean up.

What changes

• Enabled: the pre-deploy step runs in memory (precompute.Run returns its artifacts instead of writing them), and fileutil.TarGz adds them to the bundle through a TarOverlay. They land at the same relative paths, with the same bytes, that were written into the tree before.
  • Nothing is copied or staged.
  • Whatever sits at an artifact's path (an old sidecar, a link, a directory) is left out, and the fresh sidecar takes its place.
  • A unit fails only when its .astro itself is not a directory.
  • Stamping is still best-effort: a failed unit gives a warning, and the deploy goes on without its sidecar.
• Unrecognized producer, enabled: the deploy is refused with a manual-removal message when another producer's regular file sits where this deploy adds a fresh sidecar, because the plugin reads that path. The check runs on the artifacts' own paths after the step, and before it says "Cosmos Boost pre-deploy step complete". A foreign file anywhere else ships, for example:
  • beside a manifest.json that is not dbt's;
  • under dbt_packages/ or logs/;
  • a slim manifest with the slim optimization off.
• Always (enabled or not), in the bundle: a regular .astro/dbt_metadata.json / .astro/manifest.slim.json whose own producer marker names this tool is left out. It is judged by a Skip predicate inside the tarball's own walk, so:
  • the files left out are exactly ones the tarball would have carried;
  • there is one walk;
  • a deploy with the step off never fails over Cosmos Boost (a missing dags/ makes the empty tarball it always did).
  • Anything else at such a path, links included, ships untouched when the step is off.
• Always, in a generated image: pkg/imagebuild's project copy, which carries dags/ since Carry the project's files in the image a deploy or astro package generates #2302/Keep secrets out of shipped images, and deploy to DAG-deploy-off Deployments #2306, leaves the same files out of the copy, the survey and the tag digest.
• Declared Dockerfile: the build is warned, not refused. The warning names the old sidecars in its context and says to delete them.
• One definition of the sidecar directory, file names and producer marker, in pkg/airflowrt, which both the root module and pkg/imagebuild already require. precompute uses those constants.
• astro dbt cleanup is now a removed-command stub (cliout.RemovedCommand, the Guard removed 1.x commands in CI, and give bundle create/update/delete -o json #2291 pattern). It is a usage error, exit 2, and under -o json the one error object. It says v2 no longer writes Cosmos Boost files into the project and leaves out old ones from every bundle and generated image. It also says to delete any old ones by hand, since a local Airflow running the plugin, or a declared Dockerfile build, may still read them. It still satisfies TestEveryV1CommandStillRunsOrSaysWhatReplacedIt.
• Removed as dead code: cosmosboost.Cleanup/EnsureClean/BestEffortPreDeploy/PreDeploy, precompute.Cleanup and its report, the on-disk writers, dbtCleanupJSON and its golden cmd/astro/testdata/schema/dbt-cleanup.json. Astro Desktop does not use pkg/cosmosboost.
• Bundle writing:
  • UploadBundle streams tar into gzip into one file, in a temporary directory of its own. That directory is never the one being bundled, so the tarball cannot carry itself, which --project-path . used to allow.
  • The tar footer, the gzip stream and the file are each closed with their errors checked, so a full disk fails the deploy.
  • Entry names are computed with filepath.Rel, so a source of . names .astro/... and other dotfiles correctly.

Plugin contract: unchanged

Same paths (<unit>/.astro/dbt_metadata.json, <unit>/.astro/manifest.slim.json), same JSON, same hash algorithms. The project hash already excludes .astro at any depth, so whether a stale sidecar sits in the tree does not change it. When a DAG deploy prepends dags/, the sidecars sit under it like every other file.

Tests

• pkg/cosmosboost, through a real tarball:
  • enabled output matches precompute.Run byte for byte, and the tree snapshot is unchanged;
  • stale files are left out when enabled and when disabled, including one under logs/ and orphans;
  • a foreign file at a written path is refused before "complete"; foreign files elsewhere ship (non-dbt manifest.json, dbt_packages/, logs/, slim off);
  • with the feature off a link ships as a link, and with it on a link or a directory is replaced;
  • a missing root with the feature on and off, and the best-effort failure.
• pkg/cosmosboost/precompute: Run writes nothing, a file named .astro fails the unit, and whatever sits at an artifact path is replaced. The existing hash and slim tests are kept.
• pkg/airflowrt: the predicate, including duplicate keys and malformed input.
• pkg/imagebuild: the generated image leaves stale sidecars out of the survey, copy and digest; a declared survey names them.
• internal/shipcheck: the declared-Dockerfile warning.
• pkg/fileutil: TarGz with the overlay, source ., a Skip error, and a compressed stream that fails at close.
• internal/platform/astro/deploy: UploadBundle / DeployBundle decode the upload and check:
  • the sidecars are in it and the tree is unchanged;
  • stale files are left out, and foreign files are handled per setting;
  • a missing dags/ with the feature off and on;
  • --project-path . from the working directory gives only the fresh sidecar and no tarball inside.
• cmd/astro: the dbt cleanup stub in text and -o json, and it touches nothing.

Notes

• Deploy only pyproject.toml projects in v2 #2295 (now merged) removed the 1.x image-build path, which was the other place this step ran.
• pkg/cosmosboost also lives on main, so future main→v2 syncs may conflict there.

[coveralls-official]

Coverage Report for CI Build 38606515661

Coverage decreased (-0.04%) to 57.861%

Details

• Coverage decreased (-0.04%) from the base build.
• Patch coverage: 21 uncovered changes across 4 files (223 of 244 lines covered, 91.39%).
• 4 coverage regressions across 3 files.

Uncovered Changes

File	Changed	Covered	%
pkg/fileutil/files.go	95	85	89.47%
pkg/cosmosboost/predeploy.go	60	54	90.0%
internal/platform/astro/deploy/bundle.go	16	12	75.0%
pkg/cosmosboost/precompute/precompute.go	46	45	97.83%
Total (10 files)	244	223	91.39%

Coverage Regressions

4 previously-covered lines in 3 files lost coverage.

File	Lines Losing Coverage	Coverage
pkg/cosmosboost/predeploy.go	2	89.87%
internal/platform/astro/deploy/bundle.go	1	76.3%
pkg/cosmosboost/precompute/precompute.go	1	98.81%

---

Coverage Stats

Relevant Lines:	69361
Covered Lines:	40133
Line Coverage:	57.86%
Coverage Strength:	276.33 hits per line

---

💛 - Coveralls