getting-started.md

Getting started

This guide walks through setting up the full Data Hub development environment.

Prerequisites

Tool	Version	Purpose
Python	>= 3.12	Lambda, watcher, shared library
uv	latest	Python package manager and workspace orchestrator
Node.js	24, with npm 11	Web application and docs site. npm ci against the committed lockfile fails under npm 10.
PostgreSQL	>= 15	Database (local development)
Docker	latest	Lambda container builds
ffmpeg	latest	DishCam TIFF → MP4 encode (brew install ffmpeg / apt install ffmpeg). Encode tests skip if it is missing.
AWS SAM CLI	latest	Infrastructure deployment (optional — only needed for deploying to AWS)

Clone and install

git clone <repo-url>
cd data-hub

# Install all Python packages across the workspace.
uv sync --all-packages

# Install web app dependencies.
cd web && npm install && cd ..

# Install docs site dependencies (only needed to work on docs/site, and for `make check`).
cd docs/site && npm install && cd ../..

The Python workspace is managed by uv. The root pyproject.toml defines three workspace members — lambda, watcher, and packages/shared — and all are installed together by uv sync --all-packages.

Environment variables

Web application

The web app requires the following variables. Pull them from Vercel with the CLI:

cd web
vercel env pull

Variable	Required	Purpose
DATABASE_URL	Yes	PostgreSQL connection string
AUTH_GOOGLE_ID	Yes	Google OAuth client ID. See Create a Google OAuth client for where it comes from and who it lets sign in
AUTH_GOOGLE_SECRET	Yes	Google OAuth client secret, from the same client
AUTH_SECRET	Yes	Better Auth session encryption key (any 32+ character random string; also accepted as BETTER_AUTH_SECRET)
BETTER_AUTH_URL	Yes	Public origin of the web app and OAuth issuer base (issuer is {origin}/api/auth; e.g. http://localhost:3000 or https://your-deployment.vercel.app)
OAUTH_PROXY_URL	No	Staging origin used as the Google OAuth callback host for Vercel previews. Set on Staging and Preview; see First-time deployment
OAUTH_PROXY_SECRET	No	Shared secret for the preview↔staging OAuth proxy handoff (same value on both; do not reuse AUTH_SECRET)
ADMIN_EMAILS	No	Comma-separated emails promoted to workspace admin on sign-in. Bootstraps the first admin, so set it before anyone signs in
AWS_REGION	No	AWS region for S3 presigned URLs and Lambda Function URL SigV4 signing (defaults to us-west-1)
AWS_ROLE_ARN	No	IAM role ARN for Vercel OIDC federation. Used to presign S3 URLs and SigV4-sign Lambda Function URL invocations (only needed on Vercel)
S3_RAW_DATA_BUCKET	No	S3 bucket for raw data uploads (defaults to arcadia-data-hub-raw-staging)
S3_PROCESSED_BUCKET	No	S3 bucket the Lambda writes processed artifacts to. Only read to build the MCP Apps content security policy — download URLs come from files.s3_bucket. Leave it unset and the run report View cannot load processed images, video, or CSVs. See MCP Apps
LAMBDA_FUNCTION_URL	No	Lambda Function URL. Required for file reprocessing and run-archive downloads.
STALLED_PROCESSING_AFTER_MINUTES	No	Minutes a file may stay in processing before it is treated as stalled and becomes reprocessable (defaults to 20). Empty, zero, negative, or non-numeric values keep the default.
CRON_SECRET	No	Shared secret for Vercel Cron jobs. The upload-queue sweep (web/vercel.json) rejects invocations without it
INTEGRATION_SECRETS_KEY	No	64 hex characters (32 bytes). Encrypts Slack and Linear secrets admins save in the database. Generate with openssl rand -hex 32. Saving a secret fails until this is set. Plaintext Slack webhook URLs saved before this existed still work until an admin saves again
SLACK_BOT_TOKEN	No	Slack bot token (xoxb-…) for personal Slack DM notifications. Used only when an admin has not saved a bot token under Settings > Integrations
SLACK_CLIENT_ID	No	Slack app client ID for Connect Slack. Used only when an admin has not saved a client ID under Settings > Integrations
SLACK_CLIENT_SECRET	No	Slack app client secret for Connect Slack. Used only when an admin has not saved a client secret under Settings > Integrations
SLACK_STATE_SECRET	No	Signing secret for OAuth state tokens; falls back to AUTH_SECRET
SLACK_REDIRECT_URI	No	Override the OAuth redirect URI (defaults to <origin>/api/v1/settings/slack/callback)
SLACK_TEAM_ID	No	Restrict Slack connections to a specific workspace ID. Used only when an admin has not saved a workspace ID under Settings > Integrations

Local dev note: the Lambda Function URL is configured with AuthType: AWS_IAM, so to invoke it from make dev you need AWS credentials with lambda:InvokeFunctionUrl on the staging function ARN. The default credential chain (aws sso login, ~/.aws/credentials, or AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) is used when AWS_ROLE_ARN is unset locally.

Lambda / shared library

These are set in the Lambda runtime environment:

Variable	Required	Purpose
AWS_REGION	No	AWS region (defaults to us-east-1)
AWS_ACCESS_KEY_ID	No	AWS credentials
AWS_SECRET_ACCESS_KEY	No	AWS credentials
AWS_S3_RAW_DATA_BUCKET	No	S3 bucket for raw data
AWS_S3_PROCESSED_DATA_BUCKET	No	S3 bucket for processed data

Watcher

The watcher reads its configuration from a YAML file at ~/.data-hub/config.yaml. See the watcher docs for details. The only environment variable it uses is DATA_HUB_API_KEY (optional, can also be provided interactively during init).

Database setup

cd web

# Create a local PostgreSQL database.
createdb data-hub-local

# Push the Drizzle schema (no migration files generated).
npm run db:push

Other database commands:

Command	Description
npm run db:generate	Generate Drizzle migration files
npm run db:migrate	Apply pending migrations
npm run db:push	Push schema directly (no migration files)
npm run db:studio	Open Drizzle Studio GUI
npm run db:reset	Drop and re-create the public schema
npm run db:seed	Load a deterministic seed (see Local development)
npm run db:reseed	db:reset + db:push + db:seed in one shot

Web + API + database only? If you don't need the watcher or Lambda, see Local development for a zero-credential setup using make db-reseed and a dev-only sign-in.

Running locally

A production npm run build in web/ runs mcp-apps:build first so the MCP Apps run-report View is in the serverless bundle. See MCP Apps. make dev does not rebuild that View; re-run npm run mcp-apps:build after changing web/mcp-apps/.

# Start the web app dev server (Turbopack).
make dev

# Or equivalently:
cd web && npm run dev

The app runs at http://localhost:3000.

Running checks

Before pushing, run the full formatting, lint, and type-check suite:

make check

This runs both Python and web app checks:

Target	What it does
make py-format	Auto-fix with Ruff
make py-lint	Ruff linter
make py-typecheck	Pyright
make fe-format	Biome format + safe lint fixes (ultracite fix)
make fe-lint	Biome format + lint check (ultracite check)
make fe-typecheck	TypeScript compiler

Running tests

# Python unit tests only.
make py-test-unit

# Python integration tests (requires Postgres + builds/starts Next.js).
make py-test-integration

# All Python tests.
make py-test

# Web app unit + in-memory MCP tests (no external services).
make fe-test-unit

# Web app API integration tests (requires Postgres + builds/starts Next.js).
make fe-test-integration

Integration tests use a data_hub_test Postgres database and spin up a real Next.js production server. See Testing for the full per-package breakdown and CI and deployment for how these run in GitHub Actions.