Production-grade distributed task orchestration with DAG execution, exactly-once semantics, and a cybernetic Next.js 14 dashboard.
TaskMesh is a full-stack distributed task orchestration engine that combines individual async task queuing with structured DAG (Directed Acyclic Graph) workflow execution. It accepts workflows via a REST API, validates the dependency graph using Kahn's Topological Sort, and automatically unblocks downstream tasks as their parents complete — all with exactly-once execution guarantees backed by a PostgreSQL idempotency ledger.
Key engineering challenges solved:
- 1,397,327 ops/min task enqueue throughput via Redis Streams
- Kahn's Topological Sort with
CycleDetectedError— rejects cycles before any task is created - Automated parent -> child dependency resolution:
BLOCKED -> QUEUEDonSUCCEEDED - PostgreSQL idempotency ledger with SHA-256 execution hashing — zero duplicate business logic
- Self-healing Dead-Letter Queue with audit-trail replay
Captured live from the running Next.js application — real-time DAG visualizer, metrics header, live status feeds, and interactive workflow trigger console.
Captured live from the FastAPI Swagger UI — complete OpenAPI 3.1 specification for DAG workflow submission, task management, DLQ inspection, and observability.
+------------------------------------------------------------------------+
| NEXT.JS 14 CYBERNETIC DAG DASHBOARD |
| DAGVisualizer | MetricsHeader | WorkflowSubmitModal | Live Status |
+-----------------------------------v------------------------------------+
| REST API
+-----------------------------------v------------------------------------+
| FASTAPI GATEWAY & DAG VALIDATOR |
| |
| POST /workflows/dag -> Validate DAG (Kahn's) -> Persist & Enqueue |
| GET /workflows/{id} -> Fetch full graph state & node statuses |
| POST /tasks -> Submit standalone asynchronous task |
| GET /tasks/{id} -> Task attempt history & result payload |
| GET /metrics/summary -> Throughput, latency, DLQ depth metrics |
+-----------------------------------v------------------------------------+
| XADD
+-----------------------------------v------------------------------------+
| REDIS STREAMS QUEUE LAYER |
| |
| taskmesh:tasks ──── Consumer Group ──── Worker Pool (N consumers) |
| |
| StaleJobReclaimer: XAUTOCLAIM idle > 60s messages from crashed worker |
+-----------------------------------v------------------------------------+
|
+-----------------------------------v------------------------------------+
| WORKER & DAG RESOLUTION ENGINE |
| |
| 1. Claim message from stream |
| 2. Check idempotency ledger (skip if already SUCCEEDED) |
| 3. Execute task handler |
| 4. Persist result & update ledger atomically |
| 5. Resolve DAG dependencies: |
| For each child node -> if all parents SUCCEEDED -> BLOCKED -> QUEUED|
| 6. ACK message in Redis stream |
+-----------------------------------v------------------------------------+
|
+-----------------------------------v------------------------------------+
| POSTGRESQL PERSISTENCE LAYER |
| workflows | workflow_nodes | tasks | idempotency_ledger | dlq |
+------------------------------------------------------------------------+
From scripts/run_taskmesh_benchmark.py:
| Metric | Measured Value | Target | Status |
|---|---|---|---|
| Ingestion Throughput | 1,397,327 ops/min | ≥ 2,400 | ✅ PASS |
| Idempotency Ledger Speed | 28.45 ms (1,250 entries) | — | ✅ |
| P95 Enqueue Latency | 18.42 ms | ≤ 22 ms | ✅ PASS |
| Test Suite | 10 / 10 Passing | 100% | ✅ PASS |
| Layer | Technology | Description |
|---|---|---|
| Backend | FastAPI (async) | High-throughput REST endpoints |
| Queue | Redis Streams | Persistent pub/sub queuing with consumer groups |
| Database | PostgreSQL + SQLAlchemy 2.0 | Async ORM sessions with strict schema constraints |
| Frontend | Next.js 14 + Tailwind | Cybernetic dark-mode DAG visualizer |
| Testing | Pytest + pytest-asyncio | 10/10 passing unit and workflow integration tests |
python -m uvicorn app.main:create_app --factory --port 8030 --reload
# → API live at http://localhost:8030/docscd frontend
npm install
npm run dev
# → DAG Dashboard live at http://localhost:3001MIT — portfolio demonstration of distributed task orchestration & DAG workflow engine architecture.