Testing

Integration, end-to-end, and local/manual service testing use containerized environments for consistency and reproducibility. Unit tests run directly under Bazel and do not require Docker.

Prerequisites

Docker must be running for all integration, e2e, and manual testing.

# Check Docker is running
docker ps

# If not running:
# - macOS: Start Docker Desktop
# - Linux: sudo systemctl start docker

Database Architecture

SubmitQueue uses two separate databases to demonstrate proper architectural separation:

1. Application Database

Purpose: Business data (requests, counters, batches)
Schema: submitqueue/gateway/extension/storage/mysql/schema, submitqueue/orchestrator/extension/storage/mysql/schema, platform/extension/counter/mysql/schema
Used by: Gateway (receipts, request logs, and read models), Orchestrator (requests, batches, builds, and counters)
Connection: MYSQL_DSN

2. Queue Database

Purpose: Messaging infrastructure (queue messages, offsets, partition leases)
Schema: platform/extension/messagequeue/mysql/schema
Used by: Gateway (publishes and consumes), Orchestrator (publishes and consumes), Runway (consumes merge work and publishes results)
Connection: QUEUE_MYSQL_DSN

Why separate?

Queue is pluggable infrastructure - you can swap MySQL queue for Kafka, SQS, etc.
Application data and messaging concerns scale independently
Clear architectural boundary between business logic and infrastructure
In production, queue infrastructure often runs separately (e.g., managed Kafka cluster)

Note: Both use MySQL in examples for simplicity, but in production the queue could use a different technology entirely.

Automated Testing

Quick Reference

# Unit tests (no Docker required)
make test

# Integration tests (Docker required)
make integration-test-submitqueue-gateway       # Gateway in isolation
make integration-test-submitqueue-orchestrator  # Orchestrator in isolation
make integration-test-extensions    # SubmitQueue and shared extension tests
make integration-test               # All integration tests

# E2E tests (Docker required)
make e2e-test

# Build
make build                          # Build all targets
make build-all-linux                # Build Linux binaries for the local docker-compose flows (tests build their own)

Testing Levels

1. Unit Tests - Fast, no containers

Location: Co-located with code ({package}/*_test.go)
Run: make test
Speed: Fast (< 1s typically)

2. Integration Tests - Service in isolation with real dependencies

Location: test/integration/{submitqueue,stovepipe,extension}/...
Run: make integration-test-submitqueue-gateway, make integration-test-submitqueue-orchestrator, make integration-test-submitqueue-consumer, or make integration-test-extensions
Containers: MySQL + one service or the extension's dependencies
Tests one service isolated from others

3. E2E Tests - Complete domain workflows

Location: test/e2e/{submitqueue,stovepipe,runway}/
Run: make e2e-test
Containers: Each suite's required services and dependencies; SubmitQueue E2E includes Gateway, Orchestrator, Runway, and MySQL
Tests end-to-end behavior, including cross-service communication where applicable

Live GitHub Tests

Two suites land real pull requests on github.com: TestGitHubLandE2E (in make e2e-test) and the Runway GitHub merger extension test (make integration-test-runway-merger). They run only when SQ_GITHUB_TOKEN and SQ_GITHUB_TEST_REPO=owner/repo are set, and skip otherwise, so every other run is unaffected. The token needs write access to the test repository; everything the tests open is named under sq-it/ and removed afterwards, while what they merge stays on its default branch.

SQ_GITHUB_TOKEN=$(gh auth token) SQ_GITHUB_TEST_REPO=<owner>/<repo> make e2e-test

CI takes the repository from the SQ_GITHUB_TEST_REPO repository variable and the token from the SQ_TEST_REPO_TOKEN repository secret. In a repository that sets the variable, every CI run except a fork pull request sets SQ_GITHUB_TEST_REQUIRED=true, which turns a missing secret into a failure; an expired or revoked token fails regardless. Fork pull requests receive no secrets, and repositories without the variable skip the suites.

Use your own test repository

Any contributor can run these suites against a repository of their own:

Create a repository on github.com with a default branch (an initial commit is enough). It must allow squash and rebase merges, and its default branch must not require reviews or status checks, since the tests merge directly.
Create a token that can write to it: a fine-grained token limited to that repository with read and write access to contents, pull requests and issues, or simply gh auth token.
Run locally with SQ_GITHUB_TOKEN and SQ_GITHUB_TEST_REPO set, as above.
For CI in your fork, set the SQ_GITHUB_TEST_REPO repository variable and the SQ_TEST_REPO_TOKEN repository secret in the fork's Actions settings. Runs in the fork then exercise your test repository.

Every run merges a few small files under sq-it/ into the test repository's default branch, so use a repository that exists for this.

How Automated Tests Work

Tests use docker-compose via ComposeStack to spin up containers automatically:

NewComposeStack() registers cleanup (stop log tailing, tear down containers)
Up() starts containers, waits for healthchecks (--wait), and auto-tails container logs to stderr
Tests run against those containers with real-time log output
On cleanup, containers are torn down automatically (set SKIP_CLEANUP=true to keep them for inspection)

Container Naming

Test containers use meaningful, context-rich, domain-qualified names for easy debugging and correlation — and so suites from different domains never collide when run in parallel.

Naming Format

{project-name}-{service-name}-{instance}
     └─┬─┘      └────┬────┘    └──┬──┘
   From test   From compose   Docker adds

Project name format:

sq-test-{context}-{shortid}
│       │         │
│       │         └─ 6 hex digits derived from the current time
│       └─────────── Test context (domain-qualified — see convention)
└─────────────────── Namespace prefix

Context naming convention

The {context} passed to NewComposeStack is domain-qualified so that the same kind of suite in different domains yields distinct, self-describing names:

{category}-{domain}-{name}

{category} — svc (service), ext (extension), core (domain-internal infra), or e2e
{domain} — submitqueue, stovepipe, … — omit for shared/cross-domain code
{name} — the specific service/extension (e.g. gateway, storage-mysql)

Shared (cross-domain) suites carry no domain segment — e.g. the shared queue extension uses ext-messagequeue-sql.

Context reference

Suite	Context	Example container
SubmitQueue gateway	`svc-submitqueue-gateway`	`sq-test-svc-submitqueue-gateway-abc123-gateway-service-1`
SubmitQueue orchestrator	`svc-submitqueue-orchestrator`	`sq-test-svc-submitqueue-orchestrator-xyz789-orchestrator-service-1`
Stovepipe	`svc-stovepipe`	`sq-test-svc-stovepipe-abc123-stovepipe-service-1`
SubmitQueue storage extension	`ext-submitqueue-storage-mysql`	`sq-test-ext-submitqueue-storage-mysql-2ce1d0-mysql-1`
Counter extension (shared)	`ext-counter-mysql`	`sq-test-ext-counter-mysql-…-mysql-1`
Stovepipe storage extension	`ext-stovepipe-storage-mysql`	`sq-test-ext-stovepipe-storage-mysql-…-mysql-1`
Shared queue extension	`ext-messagequeue-sql`	`sq-test-ext-messagequeue-sql-a1b2c3-mysql-1`
SubmitQueue consumer (core)	`core-submitqueue-consumer`	`sq-test-core-submitqueue-consumer-…-mysql-1`
SubmitQueue e2e (full stack)	`e2e-submitqueue`	`sq-test-e2e-submitqueue-def456-gateway-service-1`

The same shape covers the other suites, including e2e-stovepipe, e2e-runway, e2e-submitqueue-git, and ext-messagequeue-vitess.

Parallel execution

Each suite normally gets a distinct project name ({context}-{shortid}), where the short suffix is derived from the low 24 bits of the current nanosecond timestamp. It is useful for separating concurrent runs but is not a guaranteed unique identifier. Every compose service publishes ephemeral host ports (- "3306", - "8080"), so suites can run in parallel. make integration-test runs suites concurrently via --test_output=errors (--test_output=streamed would force Bazel to serialize them). The domain-qualified context keeps container names understandable when many run at once.

Debugging with Container Names

Container logs are automatically streamed to stderr during test runs, so you'll see service output (startup messages, errors, zap logs) in real time — both locally and in CI.

For additional manual inspection:

# See what tests are currently running
docker ps --format "table {{.Names}}\t{{.Status}}" | grep sq-test

# Find all containers from the SubmitQueue gateway integration test
docker ps | grep sq-test-svc-submitqueue-gateway

# Inspect a specific test's MySQL
docker exec -it sq-test-ext-counter-mysql-2ce1d0-mysql-1 \
  mysql -uroot -proot submitqueue -e "SHOW TABLES;"

Manual Testing

Quick Start

# Start the full workflow stack (Gateway + Orchestrator + Runway + 2 MySQL DBs)
make local-submitqueue-start

# See running containers and endpoints
make local-submitqueue-ps

# View logs
make local-submitqueue-logs

# Stop all services
make local-stop

Testing Individual Services

Gateway Only:

# Start Gateway in isolation (Gateway + 2 MySQL DBs)
make local-submitqueue-gateway-start

# Test Ping API (port shown by make local-submitqueue-ps)
grpcurl -plaintext -d '{"message": "hello"}' localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Ping

# Test Land API
grpcurl -plaintext -d '{
  "queue": "test-queue",
  "change": {"uris": ["github://github.com/owner/repo/pull/123/0123456789abcdef0123456789abcdef01234567"]},
  "strategy": "REBASE"
}' -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \
  localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Land

# Stop
make local-submitqueue-gateway-stop

Orchestrator Only:

# Start Orchestrator in isolation (Orchestrator + 2 MySQL DBs)
make local-submitqueue-orchestrator-start

# Test Ping API (port shown by make local-submitqueue-ps)
grpcurl -plaintext -d '{"message": "hello"}' localhost:<PORT> uber.submitqueue.orchestrator.SubmitQueueOrchestrator/Ping

# Stop
make local-submitqueue-orchestrator-stop

Note: All ports are ephemeral (randomly assigned). Use make local-submitqueue-ps to see the actual port mappings.

After Code Changes

# Rebuild and restart all services
make local-submitqueue-restart

# Or stop and start fresh
make local-stop
make local-submitqueue-start

Inspecting the Databases

# Find the port (shown by make local-submitqueue-ps)
make local-submitqueue-ps

# Connect to application DB
mysql -h127.0.0.1 -P<APP_PORT> -uroot -proot submitqueue

# Connect to queue DB
mysql -h127.0.0.1 -P<QUEUE_PORT> -uroot -proot submitqueue

Using grpcurl

# Install grpcurl if not already installed
brew install grpcurl  # macOS
# OR: go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest

# Gateway reflection currently cannot resolve one imported descriptor, so pass the local proto.
grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \
  -d '{"message": "test"}' \
  localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Ping

# Call Land
grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.proto \
  -d '{
  "queue": "test-queue",
  "change": {"uris": ["github://github.com/owner/repo/pull/456/0123456789abcdef0123456789abcdef01234567"]},
  "strategy": "REBASE"
}' localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Land

Available Commands

Command	Description
`make local-submitqueue-start`	Start all services (full stack)
`make local-submitqueue-gateway-start`	Start Gateway in isolation
`make local-submitqueue-orchestrator-start`	Start Orchestrator in isolation
`make local-submitqueue-ps`	Show running containers and ports
`make local-submitqueue-logs`	Follow logs from all services
`make local-submitqueue-restart`	Rebuild and restart all services
`make local-submitqueue-stop`	Stop the SubmitQueue stack (MySQL data does not survive; a `PROVIDER=git` sandbox is left in place)
`make local-stop`	Stop SubmitQueue, Stovepipe, and Runway
`make local-submitqueue-gateway-stop`	Stop Gateway service
`make local-submitqueue-orchestrator-stop`	Stop Orchestrator service
`make local-submitqueue-clean`	Stop and remove all services, volumes, and images

Troubleshooting

Docker Not Running

# Error: "Cannot connect to the Docker daemon"
# Solution: Start Docker Desktop or Docker daemon
docker ps  # Should not error

Services Not Starting

# Check logs
make local-submitqueue-logs

# Rebuild from scratch
make local-submitqueue-clean
make local-submitqueue-start

Port Already in Use

# Docker Compose uses ephemeral ports, so conflicts are rare.
# If a test left containers behind:
docker ps | grep sq-test
docker rm -f <container-id>

Database Schema Not Applied

# Re-apply schemas manually
make local-init-submitqueue-schemas

# Or recreate everything
make local-submitqueue-clean
make local-submitqueue-start

Tests Timing Out

# Clean Docker cache
docker system prune -a

# Clean Bazel cache
make clean

# Re-run tests
make integration-test

Containers Not Cleaning Up

Containers are torn down automatically after each test. Set SKIP_CLEANUP=true to keep them for inspection.

# List all test containers
docker ps -a | grep sq-test

# Remove all test containers
docker ps -a | grep sq-test | awk '{print $1}' | xargs docker rm -f

# Remove all test networks
docker network ls | grep sq-test | awk '{print $1}' | xargs docker network rm

Writing New Tests

Integration tests under test/integration/ use the directory name as the package (package gateway, package orchestrator, package stovepipe, package mysql), not an external *_test package.

End-to-end tests under test/e2e/ are the exception: every suite there is package e2e_test.

Adding Unit Tests

Create {file}_test.go next to production code
Use table-driven tests
Run: make test

Adding Integration Tests

Add test to test/integration/submitqueue/<area>/suite_test.go (e.g., test/integration/submitqueue/gateway/suite_test.go for Gateway, test/integration/submitqueue/orchestrator/suite_test.go for Orchestrator, or a subdirectory under test/integration/submitqueue/extension/ for extension tests).
Use suite's resources (s.client, s.db).
Run the matching Makefile target such as make integration-test-submitqueue-gateway.

Example:

func (s *GatewayIntegrationSuite) TestNewFeature() {
resp, err := s.client.NewAPI(s.ctx, &pb.Request{...})
require.NoError(s.T(), err)
assert.Equal(s.T(), "expected", resp.Value)
}

Adding E2E Tests

Add test to test/e2e/submitqueue/suite_test.go
Use all service clients
Use the suite's polling helper for async operations; it retries until the condition holds and relies on the Bazel test timeout rather than adding a second hardcoded deadline
Run: make e2e-test

Navigation Menu

Repository navigation

Testing

Prerequisites

Database Architecture

1. Application Database

2. Queue Database

Automated Testing

Quick Reference

Testing Levels

Live GitHub Tests

Use your own test repository

How Automated Tests Work

Container Naming

Naming Format

Context naming convention

Context reference

Parallel execution

Debugging with Container Names

Manual Testing

Quick Start

Testing Individual Services

After Code Changes

Inspecting the Databases

Using grpcurl

Available Commands

Troubleshooting

Docker Not Running

Services Not Starting

Port Already in Use

Database Schema Not Applied

Tests Timing Out

Containers Not Cleaning Up

Writing New Tests

Adding Unit Tests

Adding Integration Tests

Adding E2E Tests

See Also

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

Testing

Prerequisites

Database Architecture

1. Application Database

2. Queue Database

Automated Testing

Quick Reference

Testing Levels

Live GitHub Tests

Use your own test repository

How Automated Tests Work

Container Naming

Naming Format

Context naming convention

Context reference

Parallel execution

Debugging with Container Names

Manual Testing

Quick Start

Testing Individual Services

After Code Changes

Inspecting the Databases

Using grpcurl

Available Commands

Troubleshooting

Docker Not Running

Services Not Starting

Port Already in Use

Database Schema Not Applied

Tests Timing Out

Containers Not Cleaning Up

Writing New Tests

Adding Unit Tests

Adding Integration Tests

Adding E2E Tests

See Also