Database Migration Tool

A developer-first database migration engine for MySQL → PostgreSQL migrations

---

Overview

A robust, asynchronous database migration engine that automatically processes MySQL to PostgreSQL migrations through a job queue system. Submit database URLs via API, track migration progress, and monitor job status in real-time.

Key Features

• 🚀 Asynchronous Processing - BullMQ-powered job queue for non-blocking migrations
• 📊 Real-time Tracking - Monitor job progress and status via REST API
• 🔄 Retry Mechanism - Automatic retry with exponential backoff
• 🏗️ Schema Introspection - Complete MySQL schema analysis and JSON export
• 🔒 Type-safe - Full TypeScript implementation
• 🐳 Docker Ready - Redis setup included with docker-compose
• 📦 Minimal Dependencies - Lightweight, fast, and maintainable

---

Table of Contents

• Quick Start
• Configuration
• API Documentation
• Architecture
• Development
• Roadmap
• Troubleshooting
• Contributing
• License

---

Quick Start

Prerequisites

• Bun (latest version)
• Redis (v7+)
• Node.js 18+ (if not using Bun)

Installation

# Clone the repository
git clone https://github.com/yourusername/migration-tool.git
cd migration-tool

# Install dependencies
bun install

# Copy environment variables
cp .env.example .env

# Start Redis (using Docker)
docker compose up -d redis

# Run the application
bun run dev

The server will start at http://localhost:3000

---

Configuration

Create a .env file in the root directory:

# Server Configuration
PORT=3000
NODE_ENV=development

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=

---

API Documentation

Root Endpoint

GET /

Response:

{
  "name": "Migration Tool API",
  "version": "1.0.0",
  "endpoints": {
    "health": "/api/health",
    "jobs": "/api/jobs"
  }
}

Health Check

GET /api/health

Response:

{
  "status": "ok",
  "timestamp": "2026-02-05T10:30:00.000Z"
}

Create Migration Job

POST /api/jobs/migration
Content-Type: application/json

{
  "databaseUrl": "mysql://user:pass@localhost:3306/source_db"
}

Response:

{
  "message": "Migration job created successfully",
  "jobId": "abc123",
  "queue": "migration"
}

Get Job Status

GET /api/jobs/:jobId/status

Response:

{
  "jobId": "abc123",
  "status": "processing",
  "progress": 60,
  "name": "run-migration",
  "data": {
    "databaseUrl": "mysql://user:pass@localhost:3306/source_db"
  }
}

Get Job Details

GET /api/jobs/:jobId

Response:

{
  "jobId": "abc123",
  "name": "run-migration",
  "status": "completed",
  "progress": 100,
  "createdAt": "2026-02-05T10:30:00.000Z",
  "completedAt": "2026-02-05T10:32:00.000Z"
}

List All Jobs

GET /api/jobs

Response:

{
  "jobs": [
    {
      "jobId": "abc123",
      "name": "run-migration",
      "status": "completed",
      "progress": 100
    },
    {
      "jobId": "def456",
      "name": "run-migration",
      "status": "processing",
      "progress": 45
    }
  ],
  "total": 2
}

Get Queue Statistics

GET /api/jobs/stats

Response:

{
  "migration": {
    "name": "migration",
    "count": 5,
    "waiting": 2,
    "active": 1,
    "completed": 2,
    "failed": 0
  }
}

---

Architecture

System Overview

src/
├── config/              # Configuration (queue, redis, app settings)
├── types/               # TypeScript type definitions
├── models/              # Data models (Redis persistence)
├── controllers/         # HTTP request handlers
├── routes/              # API route definitions
├── services/            # Business logic orchestration
├── middlewares/         # Express middlewares
├── utils/               # Utility functions (database, logger, migration)
└── queues/              # Job queue system (BullMQ)
    ├── jobs/            # Job creation & definitions
    ├── processors/      # Business logic (core work)
    ├── workers/         # BullMQ worker instances
    └── events/          # Event handlers (monitoring)

For detailed architecture documentation, see ARCHITECTURE.md

Migration Flow

Below is the end-to-end flow when a migration job is submitted:

User Request                        Queue System                         Database
─────────────                       ────────────                         ────────

POST /api/jobs/migration
  { databaseUrl }
        │
        v
  JobController
  .createMigrationJob()
        │
        v
  QueueService
  .addMigrationJob()
        │
        ├──→ migrationQueue.add()  ──→  Redis (BullMQ)
        │                                    │
        └──→ JobModel.create()     ──→  Redis (job:{id})
        │                                    │
        v                                    v
  Return { jobId }                  Worker picks up job
                                    (concurrency: 2)
                                             │
                                             v
                                    processMigrationJob()
                                             │
                                             ├──→ listDatabaseTables() ──→ MySQL/PostgreSQL
                                             │    (information_schema)
                                             │
                                             ├──→ getTableRelationships()──→ MySQL/PostgreSQL
                                             │    (PKs, FKs, reverse refs)
                                             │
                                             └──→ Migration Steps:
                                                  1. Validating
                                                  2. Extracting
                                                  3. Transforming
                                                  4. Loading
                                                  5. Verifying
                                                  (progress: 0→100%)
                                             │
                                             v
                                    QueueEvents fires:
                                    - completed (on success)
                                    - failed (on error)
                                    - progress (during execution)
                                             │
                                             v
                                    JobModel.updateStatus()
                                    ──→ Redis: job:{id}

Step-by-step breakdown:

1. API Layer — Client sends POST /api/jobs/migration with a databaseUrl. The controller validates the request and passes it to the service layer.
2. Job Creation — QueueService adds the job to the BullMQ migration queue and persists job metadata to Redis via JobModel.
3. Worker Processing — The migration worker (concurrency: 2) picks up the job and delegates to processMigrationJob().
4. Schema Introspection — The processor connects to the source database, lists all tables, and analyzes relationships (primary keys, foreign keys, reverse references).
5. Migration Steps — The job executes through 5 stages (Validating → Extracting → Transforming → Loading → Verifying), updating progress at each step.
6. Completion — BullMQ QueueEvents fires a completed event, which updates the job status in Redis. Failed jobs retry with exponential backoff (3 attempts).

Clients can poll GET /api/jobs/:jobId/status at any time to check progress.

---

Development

Available Scripts

# Development with hot reload
bun run dev

# Production build
bun run start

# Run tests
bun test

Development Tools

Redis Commander - Web-based Redis management UI

# Start Redis with Commander
docker compose up -d

# Access at http://localhost:8081

Project Structure

migration-tool/
├── index.ts                   # Application entry point
├── src/
│   ├── app.ts                 # Express app setup
│   ├── config/                # Configuration files
│   ├── controllers/           # Request handlers
│   ├── models/                # Data models
│   ├── routes/                # API routes
│   ├── services/              # Business logic
│   ├── queues/                # BullMQ queues & workers
│   ├── middlewares/           # Express middlewares
│   ├── utils/                 # Utility functions
│   └── types/                 # TypeScript types
├── docker-compose.yml         # Docker services
├── .env.example               # Environment template
└── package.json               # Dependencies

Manual Testing

# 1. Start Redis (in a separate terminal)
docker compose up -d redis

# 2. Start the server
bun run dev

# 3. Create a migration job
curl -X POST http://localhost:3000/api/jobs/migration \
  -H "Content-Type: application/json" \
  -d '{
    "databaseUrl": "mysql://user:pass@localhost:3306/source_db"
  }'

# Response:
# {
#   "message": "Migration job created successfully",
#   "jobId": "abc123xyz",
#   "queue": "migration"
# }

# 4. Check job status
curl http://localhost:3000/api/jobs/abc123xyz/status

# 5. Get all jobs
curl http://localhost:3000/api/jobs

# 6. Get queue statistics
curl http://localhost:3000/api/jobs/stats

# 7. View Redis queue (using Redis CLI)
docker exec -it local_redis redis-cli
> KEYS bull:migration:*
> HGETALL bull:migration:1

---

Docker Deployment

Using Docker Compose

# Start all services (Redis + App)
docker compose up -d

# View logs
docker compose logs -f

# Stop services
docker compose down

Manual Docker Build

# Build image
docker build -t migration-tool .

# Run container
docker run -p 3000:3000 \
  -e REDIS_HOST=redis \
  -e REDIS_PORT=6379 \
  migration-tool

---

Roadmap

Current Features

• ✅ Asynchronous job queue system powered by BullMQ
• ✅ REST API for job management
• ✅ Real-time job status tracking and progress monitoring
• ✅ Schema introspection for MySQL databases
• ✅ Table relationship analysis (primary keys, foreign keys, reverse references)
• ✅ Automatic retry with exponential backoff (3 attempts)
• ✅ Redis-based job persistence and monitoring
• ✅ Support for hot-reloading in development (Bun)
• ✅ Docker support with Redis Commander UI
• ✅ TypeScript for type safety

Planned Features

• PostgreSQL schema generation and target database creation
• Data migration with batching and progress tracking
• Migration validation and verification
• Rollback capability for failed migrations
• Web UI dashboard for job monitoring and management
• Support for additional database types (SQL Server, Oracle, etc.)
• Migration templates and presets for common scenarios
• Webhook notifications for job completion/failure

---

Troubleshooting

Redis Connection Issues

# Check if Redis is running
docker ps | grep redis

# Test Redis connection
redis-cli ping

# Restart Redis
docker compose restart redis

# Check the app logs for Redis connection errors

Port Already in Use

# Change PORT in .env file
PORT=3001

# Or kill the process using the port
lsof -ti:3000 | xargs kill -9

Job Stuck in Processing

Jobs automatically retry with exponential backoff. To troubleshoot:

# View application logs
bun run dev

# Check Redis queue with Redis CLI
docker exec -it local_redis redis-cli

# List all migration queue keys
> KEYS bull:migration:*

# Inspect a specific job
> HGETALL bull:migration:{jobId}

# Check queue status
> LLEN bull:migration:waiting
> LLEN bull:migration:active

Common causes:

• Check if Redis is running and accessible
• Review application logs for error messages
• Verify the database connection string
• Look for jobs in the failed state using GET /api/jobs/stats
• Inspect failed job logs using the job ID endpoint

---

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Related Resources

• Job Queue Details: See README.md in queues
• Architecture Documentation: See ARCHITECTURE.md
• Code Style Guide: See CLAUDE.md
• Changelog: See CHANGELOG.md
• Security Policy: See SECURITY.md

---

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Acknowledgments

• BullMQ - Robust job queue system
• Express - Web framework
• Bun - Fast JavaScript runtime
• TypeScript - Type-safe JavaScript

---

Support & Community

• 🐛 Report Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📚 Documentation: See ARCHITECTURE.md for detailed technical information

---

Built with ❤️ by Soumik Sarkar

Made with TypeScript • Powered by BullMQ • Running on Bun