Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
8 changes: 8 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
*.sh text eol=lf
mvnw text eol=lf
Makefile text eol=lf
*.cmd text eol=crlf
*.jar binary
*.parquet binary
*.png binary
*.gz binary
44 changes: 44 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ on:
pull_request:
branches:
- main
schedule:
# nightly regression run against the newest snowflake-next image
- cron: '17 3 * * *'
workflow_dispatch:

permissions:
Expand All @@ -25,14 +28,22 @@ jobs:
sample:
- airflow-dbt-transformation
- citi-bike-data-app
- claims-service
- credit-card-fraud-detection-with-snowpark
- credit-scoring-with-snowpark
- fleet-telemetry
- glue-snowflake-integration
- lambda-snowpark-connector
- metering-api
- migrate-and-swap
- multi-container
- order-to-cash-cdc
- phi-guardrails
- predicting-customer-spend
- slim-ci-lab
- soda-data-quality-checks
- streamlit-snowpark-dynamic-filters
- tenantboard
defaults:
run:
working-directory: ${{ matrix.sample }}
Expand Down Expand Up @@ -60,6 +71,33 @@ jobs:
with:
terraform_wrapper: false

- name: Setup uv
if: hashFiles(format('{0}/uv.lock', matrix.sample)) != ''
uses: astral-sh/setup-uv@v6
with:
enable-cache: true

- name: Setup Java
if: hashFiles(format('{0}/pom.xml', matrix.sample)) != ''
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '21'
cache: maven

- name: Setup Go
if: hashFiles(format('{0}/go.mod', matrix.sample)) != ''
uses: actions/setup-go@v5
with:
go-version-file: ${{ matrix.sample }}/go.mod
cache-dependency-path: ${{ matrix.sample }}/go.sum

- name: Setup .NET
if: hashFiles(format('{0}/global.json', matrix.sample)) != ''
uses: actions/setup-dotnet@v4
with:
global-json-file: ${{ matrix.sample }}/global.json

- name: Install lstk and the Snowflake and AWS CLIs
run: |
npm install -g @localstack/lstk
Expand All @@ -80,8 +118,14 @@ jobs:
if grep -q '^start-localstack:' Makefile; then make start-localstack; else make start; fi

- name: Run tests
env:
# samples tested with Testcontainers start their own emulator
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
run: make test

- name: Show LocalStack logs
if: failure()
env:
# docker compose refuses to read a compose file that requires the token
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
run: make logs || true
18 changes: 18 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -172,3 +172,21 @@ cython_debug/
**/tempdata/*.txt

.idea/

# Throwaway key pairs and local config generated by `make up`
.secrets/
*.p8
*.pem
!**/fixtures/**/*.pem

# Node, Java, dbt and tool caches
node_modules/
.mvn/wrapper/maven-wrapper.jar
.ruff_cache/
**/dbt/target/
**/dbt/logs/
**/dbt/dbt_packages/
prod-artifacts/
data/raw/
.vscode/
*.swp
2 changes: 1 addition & 1 deletion CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# From: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners#example-of-a-codeowners-file

* @whummer @eruditmorina @hovaesco @sabir-akhadov-localstack
* @hovaesco @sabir-akhadov-localstack
15 changes: 13 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The samples run on the next-generation Snowflake emulator, [`localstack/snowflak
- [`lstk`](https://docs.localstack.cloud/snowflake/developer-tools/lstk/) 1.3.0 or later, which starts the emulator with `lstk start --type snowflake --image localstack/snowflake-next:latest`
- [Docker Compose](https://docs.docker.com/compose/) and the [`awslocal` CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/) for the samples that also use AWS services: `airflow-dbt-transformation`, `glue-snowflake-integration`, `lambda-snowpark-connector` and `multi-container`. Each of them has a `docker-compose.yml` that starts LocalStack AWS on port 4566 and Snowflake Next on a second port.
- [Snowflake CLI](https://docs.snowflake.com/en/developer-guide/snowflake-cli/installation/installation) (`snow`) with a `local` and a `localstack` connection, see below
- For the application samples (`claims-service`, `fleet-telemetry`, `metering-api`, `migrate-and-swap`, `order-to-cash-cdc`, `phi-guardrails`, `slim-ci-lab` and `tenantboard`), Docker Compose v2 and the sample's own toolchain: [`uv`](https://docs.astral.sh/uv/) for the Python samples, Node.js 22 for the Tenantboard UI, JDK 21 for Claims Service, Go for Metering API and the .NET 10 SDK for Fleet Telemetry. They start the emulator themselves and do not need `lstk` or `snow`. On Windows, run them inside WSL 2.

## Configuration

Expand All @@ -20,7 +21,7 @@ Set your auth token before running any sample:
export LOCALSTACK_AUTH_TOKEN=<your-auth-token>
```

Alternatively, log in with `lstk login`.
Alternatively, log in with `lstk login`. The application samples read the token from the environment or from `~/.localstack/default.env`.

Several samples use the Snowflake CLI with a connection named `local` or `localstack`. Create both, pointing at the emulator:

Expand All @@ -37,20 +38,30 @@ done
|---|---|
| [airflow-dbt-transformation](./airflow-dbt-transformation) | Run local data transformation pipelines with Airflow, dbt, and Snowpark. |
| [citi-bike-data-app](./citi-bike-data-app) | Seed Citibike data into Snowflake and query it from a web application. |
| [claims-service](./claims-service) | A Java 21 Spring Boot service with stored procedures, a Java UDF and integration tests on Testcontainers. |
| [credit-card-fraud-detection-with-snowpark](./credit-card-fraud-detection-with-snowpark) | Feature engineering and ML-oriented fraud workflows with Snowpark. |
| [credit-scoring-with-snowpark](./credit-scoring-with-snowpark) | Exploratory data analysis for credit scoring using Snowpark. |
| [fleet-telemetry](./fleet-telemetry) | An ASP.NET Core service that ingests vehicle telemetry as JSON and analyzes it with FLATTEN, window functions and a dynamic table. |
| [glue-snowflake-integration](./glue-snowflake-integration) | Integrate AWS Glue ETL jobs with Snowflake in a local environment. |
| [lambda-snowpark-connector](./lambda-snowpark-connector) | Use Snowpark from an AWS Lambda function running on LocalStack. |
| [metering-api](./metering-api) | A Go usage-metering and invoicing service with idempotent ingestion, transactional invoice close and async rollups. |
| [migrate-and-swap](./migrate-and-swap) | Zero-downtime schema releases with schemachange: migrate a clone, validate it, swap it in, roll back in one step. |
| [multi-container](./multi-container) | Run LocalStack for Snowflake and LocalStack AWS emulation in separate containers. |
| [order-to-cash-cdc](./order-to-cash-cdc) | Kafka change events processed into current-state and history tables with streams, tasks and dynamic tables. |
| [phi-guardrails](./phi-guardrails) | Governance as code: tag-based masking, row access policies and a tested access matrix. |
| [predicting-customer-spend](./predicting-customer-spend) | Predict customer spend using Snowpark and Python ML tooling. |
| [slim-ci-lab](./slim-ci-lab) | A dbt Core project with a Slim CI pull-request pipeline: state selection, defer and per-PR schemas. |
| [soda-data-quality-checks](./soda-data-quality-checks) | Execute Soda data quality checks against local Snowflake tables. |
| [streamlit-snowpark-dynamic-filters](./streamlit-snowpark-dynamic-filters) | Streamlit application with dynamic Snowpark-powered filters. |
| [tenantboard](./tenantboard) | A multi-tenant FastAPI and React analytics app with tenant isolation and PII masking enforced by Snowflake policies. |

## Testing

A fresh Snowflake Next instance only has the `SNOWFLAKE` database, as in a new Snowflake account, so each sample creates the database it uses before loading data.

Every sample has `make start` (or `make start-localstack` where `make start` runs the app) to start LocalStack, and `make test` to check that the sample works. The [CI workflow](.github/workflows/main.yml) runs `make install`, the start target and `make test` for each sample on every pull request.
Every sample has `make start` (or `make start-localstack` where `make start` runs the app) to start LocalStack, and `make test` to check that the sample works. The [CI workflow](.github/workflows/main.yml) runs `make install`, the start target and `make test` for each sample on every pull request, and every night against the newest `localstack/snowflake-next` image to catch regressions.

The application samples document `make up`, `make seed`, `make demo`, `make test` and `make down` in their READMEs, with `make start` and `make stop` as aliases for `make up` and `make down`. They bind the emulator to `127.0.0.1:4566`; set `SNOWFLAKE_PORT` to use another port. Their test suites also run against a real Snowflake account with `make test-cloud`, using the `SNOWFLAKE_*` settings in each sample's `.env.example`.

## Checking Out A Single Sample

Expand Down
31 changes: 31 additions & 0 deletions claims-service/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Every variable the sample reads. Nothing here is required for `./mvnw verify`
# except LOCALSTACK_AUTH_TOKEN; the Makefile also reads it from ~/.localstack/default.env.

# LocalStack auth token with Snowflake access (never commit a real value).
LOCALSTACK_AUTH_TOKEN=

# Emulator image; defaults to the newest build. Pin a tag or digest to freeze it.
SNOWFLAKE_IMAGE=localstack/snowflake-next:latest

# Host port of the compose emulator (make up / seed / run). Testcontainers picks its own.
SNOWFLAKE_PORT=4566

# EMULATOR (default) or SNOWFLAKE_CLOUD (make test-cloud).
TEST_TARGET=EMULATOR

# Key pair used by make seed / make run (make up generates it).
SNOWFLAKE_PRIVATE_KEY_PATH=.secrets/rsa_key.p8

# Real Snowflake only: TEST_TARGET=SNOWFLAKE_CLOUD, or SPRING_PROFILES_ACTIVE=cloud for the app.
SNOWFLAKE_ACCOUNT=
SNOWFLAKE_HOST=
SNOWFLAKE_USER=
SNOWFLAKE_ROLE=
SNOWFLAKE_WAREHOUSE=
# App only (the test suite creates and drops its own CLAIMS_IT_<random> database).
SNOWFLAKE_DATABASE=CLAIMS
SNOWFLAKE_SCHEMA=CORE
SPRING_PROFILES_ACTIVE=

# Liquibase contexts; "default,scheduled" also resumes the KPI task (every 5 minutes).
CLAIMS_LIQUIBASE_CONTEXTS=default
3 changes: 3 additions & 0 deletions claims-service/.mvn/wrapper/maven-wrapper.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
wrapperVersion=3.3.4
distributionType=only-script
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.11/apache-maven-3.9.11-bin.zip
77 changes: 77 additions & 0 deletions claims-service/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Claims Service: Spring Boot on Snowflake JDBC, tested against the Snowflake-Next emulator.
#
# make up generate the dev key, pull the emulator image, start a long-lived emulator
# make seed migrate that emulator and load deterministic sample claims (idempotent)
# make run run the service on http://localhost:8080 against that emulator
# make test ./mvnw -q verify: the full suite, on its own Testcontainers emulator
# make demo the demo moment: clean build plus full suite, timed, with a per-test summary
# make down stop and remove the long-lived emulator
# make test-cloud the same suite against real Snowflake (consumes credits)
#
# `make test` and `make demo` need only Docker and JDK 21: Testcontainers starts, provisions
# and removes a fresh emulator for every run, so they do not depend on `make up`.

-include $(HOME)/.localstack/default.env
export LOCALSTACK_AUTH_TOKEN
SNOWFLAKE_PORT ?= 4566
SNOWFLAKE_IMAGE ?= localstack/snowflake-next:latest
export SNOWFLAKE_PORT SNOWFLAKE_IMAGE

COMPOSE := docker compose -p snfs-claims-service
MVNW := ./mvnw -B
SECRETS := .secrets
DEV_KEY := $(SECRETS)/rsa_key.p8
APP_JAR := claims-api/target/claims-api-1.0.0.jar
JAVA_OPTS := -Dnet.snowflake.jdbc.disablePlatformDetection=true
# The default (no profile) config, which targets the compose emulator.
APP := java $(JAVA_OPTS) -jar $(APP_JAR) --claims.snowflake.private-key-path=$(DEV_KEY)

.PHONY: up seed run test demo down test-cloud build logs install start stop

up: $(DEV_KEY)
$(COMPOSE) pull
$(COMPOSE) up -d
scripts/wait-for-emulator.sh 180

build:
$(MVNW) -q -DskipTests package

seed: build
$(APP) --spring.main.web-application-type=none --claims.seed.enabled=true

run: build
$(APP)

test:
./mvnw -q verify
@scripts/test-summary.sh

demo:
scripts/demo.sh

down:
$(COMPOSE) down -v --remove-orphans

# Reads SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_PRIVATE_KEY_PATH, SNOWFLAKE_ROLE and
# SNOWFLAKE_WAREHOUSE; creates and drops a CLAIMS_IT_<random> database.
test-cloud:
TEST_TARGET=SNOWFLAKE_CLOUD ./mvnw -q verify
@scripts/test-summary.sh

logs:
$(COMPOSE) logs --tail=200 snowflake

# Throwaway unencrypted PKCS#8 key pair for key-pair (SNOWFLAKE_JWT) auth.
$(DEV_KEY):
mkdir -p $(SECRETS)
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out $@
openssl rsa -in $@ -pubout -out $(SECRETS)/rsa_key.pub
chmod 600 $@

# The targets the repository CI runs for every sample (see the root README)
install:
@echo "Nothing to install, make up and make test fetch the dependencies"

start: up

stop: down
Loading
Loading