Add teardown script for POC cost-incurring resources#63
Merged
Conversation
Adds scripts/destroy_resources.sh to tear down the three always-on resources that drive POC cost: the Cloud Composer 2 environment (GKE + Airflow), the Vertex AI Feature Store online store (Bigtable), and the Artifact Registry repository. Mirrors the existing setup scripts. Deletes the feature view before the online store, captures and removes the leftover Composer DAGs bucket, and leaves BigQuery, GCS, Pub/Sub and the kfp-mlops@ service account untouched so the setup scripts can recreate everything. Safety: confirmation prompt, --dry-run, per-resource --skip-* flags, --keep-bucket, and --yes for automation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two consistency fixes to support rebuilding images after an Artifact Registry teardown: - Add workflow_dispatch so the pipeline can be triggered manually from the Actions UI without pushing a commit (previously only push/PR triggered it). - Tag and push ml-pipelines-kfp-image with :latest in addition to the branch tag, matching the fastapi and beam images. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
scripts/destroy_resources.shto tear down the three always-on resources that drive the ongoing cost of these POC projects:ml-pipelines-composer(GKE + Airflow, 24/7 — the biggest cost) plus its leftover DAGs bucketiris_featuresthen online storeml_online_store(Bigtable node billed per node-hour)sahil-experiment-docker-imagesand all imagesIt mirrors the existing
setup_*scripts so everything can be recreated later. BigQuery, GCS (sb-vertex), Pub/Sub and thekfp-mlops@service account are intentionally left untouched, since the setup scripts depend on them.Behavior / safety
--dry-run,--yes,--skip-composer,--skip-feature-store,--skip-artifact-registry,--keep-bucketdestroy) unless--yesRecreate path
./scripts/setup_artifact_registry.sh→ push to main / re-run CI to rebuild images./scripts/setup_feature_store.sh./scripts/setup_composer.sh→./scripts/sync_dags.shTesting
bash -nsyntax check passes--dry-runverified end-to-end (prints delete commands, deletes nothing)🤖 Generated with Claude Code