Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
339 lines (235 loc) · 9.25 KB

DEPLOYMENT.md

File metadata and controls

339 lines (235 loc) · 9.25 KB

Deployment

This document covers how code gets from a developer's machine to production, including the CI/CD pipeline, manual deployment, rollback procedures, and backups.

How deployment works

Every push to master that passes CI is automatically deployed to production via GitHub Actions. The pipeline:

  1. Runs the full test suite against a PostGIS service container
  2. Builds Tailwind CSS to verify frontend compilation
  3. If tests pass and the push is to master, deploys to Dokku via git push

Dokku then:

  1. Builds a Docker image using the multi-stage Dockerfile (CSS build + Python app)
  2. Runs collectstatic (inside the Dockerfile)
  3. Runs python manage.py migrate --noinput (from app.json predeploy hook)
  4. Starts the new container with gunicorn
  5. Runs the health check (/health/) before routing traffic
  6. Removes the old container

CI/CD pipeline

CI workflow (.github/workflows/ci.yml)

Runs on every push and pull request.

Test job:

  • Spins up a PostGIS 17 service container
  • Installs Python 3.14 + uv, Node.js 22
  • Installs GIS system libraries (gdal, geos, proj)
  • Builds Tailwind CSS (npm ci && npm run build:css)
  • Runs the test suite (uvx nox -s tests)

Deploy job (master only, after tests pass):

  • Checks out the full git history
  • Configures SSH with the Dokku deploy key
  • Pushes to the Dokku remote
  • Syncs ops scripts (backup.sh, restore.sh) to the VPS via SCP
  • Purges the Cloudflare cache

Docs workflow (.github/workflows/docs.yml)

Runs on pushes to master when docs or API files change. Exports the OpenAPI schema from a live Django instance, builds the documentation site with Zensical, and deploys to GitHub Pages at developers.bookcorners.org.

Required GitHub secrets

Secret Description
DOKKU_SSH_PRIVATE_KEY SSH private key for pushing to Dokku
DOKKU_HOST VPS hostname (e.g., vps.bookcorners.org)

Manual deployment

When CI/CD is unavailable or you need to deploy a hotfix:

# From your local machine, in the book-corners directory
git push dokku master

The Dokku remote should already be configured:

# If not, add it:
git remote add dokku dokku@vps.bookcorners.org:book-corners

Monitoring the deploy

# Watch build output in real-time during git push

# After deploy, check status on the VPS:
sudo dokku ps:report book-corners
sudo dokku logs book-corners --tail

Post-deploy smoke checklist

After every deploy, verify these manually or review in monitoring:

  • Homepage loads with styles: https://bookcorners.org/
  • Static CSS returns 200: https://bookcorners.org/static/css/app.css
  • Login page works: https://bookcorners.org/login/
  • Map page loads with markers: https://bookcorners.org/map/
  • Admin panel accessible: https://bookcorners.org/admin/
  • Health check passes: https://bookcorners.org/health/

Rollback

Revert to the previous release

If a deploy introduces a bug, revert the commit on master and push again:

git revert HEAD
git push origin master
# CI will run tests and auto-deploy the revert

For an immediate rollback without waiting for CI:

git revert HEAD
git push dokku master

Rebuild from the current deployed code

If the container is misbehaving but the code is correct:

sudo dokku ps:rebuild book-corners

Deploy a specific commit

# Push a specific commit to Dokku
git push dokku <commit-sha>:refs/heads/master

Database migrations

Migrations run automatically on every deploy via the app.json predeploy hook:

{
  "scripts": {
    "dokku": {
      "predeploy": "python manage.py migrate --noinput"
    }
  }
}

If a migration fails, the deploy is aborted and the previous container keeps running.

Running migrations manually

sudo dokku run book-corners python manage.py migrate

Checking migration status

sudo dokku run book-corners python manage.py showmigrations

Backups

Backups use Borg with BorgBase as the offsite target.

What gets backed up

  • Database — Full PostgreSQL dump via dokku postgres:export
  • Media files — User-uploaded photos from /var/lib/dokku/data/storage/book-corners/media/

Backup scripts

Operational scripts are versioned in the repo under scripts/:

  • scripts/backup.sh — nightly backup (database dump + media archive to Borg)
  • scripts/restore.sh — interactive restore with selective mode support

Both scripts read configuration from /home/deploy/.env.backup on the VPS. They must be run as the deploy user (not with sudo), because borg authenticates to BorgBase using the deploy user's SSH key.

Prerequisite: The deploy user needs passwordless sudo for dokku commands (used by the scripts for database export/import):

echo 'deploy ALL=(ALL) NOPASSWD: /usr/bin/dokku' | sudo tee /etc/sudoers.d/deploy-dokku

Backup schedule

Nightly at 3 AM server time via cron (under the deploy user):

0 3 * * * /home/deploy/backup.sh >> /var/log/book-corners-backup.log 2>&1

Restore procedures

Use the restore.sh script on the VPS:

# List available archives
/home/deploy/restore.sh --list

# Preview what would be restored (no changes)
/home/deploy/restore.sh --dry-run

# Restore a specific archive (both DB and media)
/home/deploy/restore.sh book-corners-2026-02-28T03:00:00

# Restore only the database
/home/deploy/restore.sh --db-only

# Restore only media files
/home/deploy/restore.sh --media-only

The script prompts for confirmation before any destructive operation and suggests creating a test database first.

Manual restore (without script)

If the restore script is unavailable:

export BORG_PASSPHRASE="<your-passphrase>"
export BORG_REPO="<your-borg-repo-url>"

# Extract the dump from a borg archive
mkdir -p /tmp/restore && cd /tmp/restore
borg extract "$BORG_REPO::<archive-name>" --pattern "*/db.dump"

# Restore to a test database first
sudo dokku postgres:create book-corners-db-test
sudo dokku postgres:import book-corners-db-test < $(find . -name db.dump)

# Verify the data
sudo dokku postgres:connect book-corners-db-test
# In psql: SELECT count(*) FROM libraries_library;

# If everything looks good, restore to production
sudo dokku postgres:import book-corners-db < $(find . -name db.dump)

# Clean up
sudo dokku postgres:destroy book-corners-db-test --force
rm -rf /tmp/restore

Updating ops scripts on the VPS

The deploy job in CI automatically syncs scripts/backup.sh and scripts/restore.sh to /home/deploy/ on the VPS after every successful deploy. No manual git pull or repo checkout is needed.

The sync step uses SCP (atomic copy to /tmp/ then mv into place) and runs with continue-on-error: true so a sync failure never blocks a deploy.

Prerequisite: The CI SSH key must be authorized for the deploy user:

cat ~/.ssh/dokku_deploy.pub | ssh deploy@vps.bookcorners.org 'cat >> ~/.ssh/authorized_keys'

Dockerfile overview

The production image is built in two stages:

Stage 1 — CSS builder (node:22-alpine):

  • Installs npm dependencies
  • Compiles Tailwind CSS to static/css/app.css

Stage 2 — Python app (python:3.14-slim):

  • Installs system dependencies (GDAL, GEOS, Proj, libjpeg, zlib, gettext)
  • Installs Python packages via uv
  • Copies compiled CSS from stage 1
  • Runs compilemessages (compiles .po translation files to .mo)
  • Runs collectstatic
  • Serves via gunicorn on $PORT

Health checks

The app exposes a /health/ endpoint that Dokku checks during deployment. If the health check fails after 3 attempts, the deploy is rolled back automatically.

Configuration in app.json:

{
  "healthchecks": {
    "web": [{
      "type": "startup",
      "name": "web check",
      "path": "/health/",
      "attempts": 3,
      "timeout": 5,
      "wait": 5
    }]
  }
}

Troubleshooting

Deploy fails during build

Check the build output for errors. Common causes:

  • PostGIS extension not available — verify the database image
  • collectstatic fails — check WhiteNoise and STATIC_ROOT configuration
  • npm install fails — check package.json and package-lock.json are committed

Deploy fails during migration

# Check migration status
sudo dokku run book-corners python manage.py showmigrations

# View the full error
sudo dokku logs book-corners --tail

The old container keeps running if migration fails. Fix the migration and push again.

App is running but returning errors

# View recent logs
sudo dokku logs book-corners --tail

# Check the process status
sudo dokku ps:report book-corners

# Check environment variables
sudo dokku config:show book-corners

# Run a management command to debug
sudo dokku run book-corners python manage.py check --deploy

Static files not loading

# Verify collectstatic ran during build (check deploy output)
# WhiteNoise serves static files — no separate nginx config needed

# Check the static file URL
curl -I https://bookcorners.org/static/css/app.css

SSL certificate issues

# Check certificate status
sudo dokku letsencrypt:list

# Renew manually if needed
sudo dokku letsencrypt:enable book-corners

# Verify Cloudflare SSL mode is "Full (Strict)"