A production-style real-time aviation telemetry and analytics platform built as a polyglot software and data engineering portfolio project.
The system generates aircraft telemetry in Java, streams events through Apache Kafka, handles operational processing with Kotlin + Spring Boot, performs real-time analytics using Scala + Apache Spark, persists data in PostgreSQL, and presents live fleet operations through a React + TypeScript dashboard.
- Real-time event-driven telemetry pipeline with Apache Kafka
- Kotlin + Spring Boot REST API with JWT authentication
- Spark Structured Streaming with event-time processing and one-minute windows
- PostgreSQL persistence, analytical views, indexes and advanced SQL
- Interactive React + TypeScript operations dashboard
- Automated anomaly detection and alert resolution
- Prometheus + Grafana observability
- Docker Compose orchestration with optional service profiles
- Automated tests, ESLint, production builds and GitHub Actions CI
- End-to-end Java → Kafka → Spark → PostgreSQL streaming path validated locally
flowchart LR
P[Java Telemetry Producer] -->|JSON events| K[(Apache Kafka)]
K --> B[Kotlin Spring Boot API]
K --> S[Scala Spark Structured Streaming]
B -->|Telemetry + Alerts| DB[(PostgreSQL)]
S -->|Windowed Aggregates| DB
F[React + TypeScript Dashboard] -->|REST + JWT| B
B --> M[Prometheus]
M --> G[Grafana]
| Area | Technologies |
|---|---|
| Producer | Java 21 |
| Event Streaming | Apache Kafka |
| Backend | Kotlin, Spring Boot 3, REST |
| Security | Spring Security, JWT |
| Streaming Analytics | Scala 2.12, Spark Structured Streaming |
| Data | PostgreSQL 16, SQL, Flyway |
| Frontend | React, TypeScript, Vite, Recharts |
| Observability | Spring Actuator, Micrometer, Prometheus, Grafana |
| DevOps | Docker, Docker Compose, GitHub Actions |
| Quality | Maven tests, ESLint, TypeScript build |
- A Java producer maintains simulated state for eight aircraft and emits telemetry approximately every 1.2 seconds.
- Events are published to Kafka using
aircraftIdas the message key. - The Kotlin/Spring Boot backend consumes telemetry, persists operational data and generates anomaly alerts.
- Spark Structured Streaming independently consumes the telemetry stream and performs event-time, one-minute window aggregations per aircraft.
- Spark calculates engine-temperature, fuel-flow, vibration and ground-speed statistics and UPSERTs the results into PostgreSQL.
- The secured REST API exposes fleet state, telemetry, alerts, health analytics and dashboard KPIs.
- The React/TypeScript dashboard refreshes operational data every five seconds and supports alert resolution.
The frontend provides dedicated views for fleet overview, live telemetry, incident management and analytics.
Live aircraft state is refreshed from the backend every five seconds.
Operational rules detect abnormal engine temperature, vibration and fuel-flow conditions. Alerts are classified by severity and can be resolved through the API.
Fleet-health scores are calculated dynamically from recent telemetry and exposed through the analytics API.
The Spark service consumes the telemetry-events Kafka topic using Spark Structured Streaming.
The pipeline implements:
- explicit JSON schema parsing
- event-time processing
- two-minute watermarking
- one-minute streaming windows
- per-aircraft aggregation
- micro-batch processing
- transactional PostgreSQL writes
- UPSERT semantics
- Spark checkpointing
Each aircraft/window produces:
avg_engine_temperature_c
avg_fuel_flow_kg_h
max_vibration_g
avg_ground_speed_kt
event_count
The complete streaming path has been exercised successfully:
Java Telemetry Producer
│
▼
Apache Kafka
│
▼
Spark Structured Streaming
│
▼
1-minute Aggregations
│
▼
PostgreSQL
Spark micro-batches were verified processing live Kafka events and persisting aggregate rows into telemetry_aggregates.
Primary tables:
aircraft
flights
telemetry
alerts
telemetry_aggregates
The relational model includes foreign keys, unique constraints, composite indexes, analytical views, idempotency constraints and Flyway migrations.
database/analytics_examples.sql includes examples using:
- CTEs
LAG()DENSE_RANK()ROW_NUMBER()- analytical views
EXPLAIN ANALYZE
Fleet health is calculated from telemetry recorded during the most recent 30-minute period.
The PostgreSQL analytical view aggregates engine temperature, vibration, fuel flow and sample counts before applying threshold-based penalties to produce a bounded health score from 0 to 100.
This keeps analytical logic explicit and inspectable in SQL.
POST /auth/login
GET /actuator/health
GET /actuator/prometheus
GET /api/v1/dashboard
GET /api/v1/aircraft
GET /api/v1/aircraft/{id}/telemetry?limit=100
GET /api/v1/alerts?openOnly=true
PATCH /api/v1/alerts/{id}/resolve
GET /api/v1/analytics/fleet-health
Demo credentials:
username: demo
password: demo1234
- Docker
- Docker Compose
Create the environment file:
cp .env.example .envStart the core platform:
docker compose up --build postgres kafka backend-api telemetry-producer frontend-dashboardThen open:
Dashboard: http://localhost:5173
Backend health: http://localhost:8080/actuator/health
Spark is isolated behind the analytics profile:
docker compose --profile analytics up --build streaming-analyticsThe local Spark configuration uses local[2], a 512 MB driver and two shuffle partitions.
Start Prometheus and Grafana when required:
docker compose --profile observability up --buildPrometheus: http://localhost:9090
Grafana: http://localhost:3000
Grafana credentials:
admin / admin
Idempotent ingestion
Each telemetry event contains a UUID and telemetry.event_id is unique, preventing duplicate records after Kafka redelivery.
Kafka partition affinity
aircraftId is used as the Kafka record key, preserving per-aircraft ordering within a partition.
Operational vs analytical processing
Spring Boot handles low-latency ingestion, persistence and alerting, while Spark independently handles windowed analytical workloads.
Event-time streaming
Spark processes telemetry according to event timestamps and uses a two-minute watermark for bounded late-event handling.
Transactional analytical persistence
Spark foreachBatch processing writes aggregates through PostgreSQL transactions and UPSERTs aircraft/window results.
SQL-first analytics
Fleet-health calculations remain inspectable in PostgreSQL rather than being hidden entirely inside application logic.
Resource-aware development
Spark and observability components use Docker Compose profiles so resource-heavy services can be started only when required.
The system has been validated both at component level and as a running end-to-end platform.
| Component | Validation |
|---|---|
| Java Producer | Maven tests pass |
| Kotlin Backend | Maven tests pass |
| Backend Runtime | Spring Boot health UP |
| React Frontend | ESLint passes with zero errors |
| Frontend Build | TypeScript + Vite production build succeeds |
| Scala/Spark | Maven build succeeds |
| Spark Runtime | Structured Streaming runs successfully in Docker |
| Kafka → Spark | Live events consumed and windowed |
| Spark → PostgreSQL | Aggregate rows persisted successfully |
| Dashboard | Authentication, navigation and live data verified |
| Alerts | Alert resolution verified against running backend |
fleet-intelligence-platform/
├── backend-api/ # Kotlin + Spring Boot + JWT + Kafka
├── telemetry-producer/ # Java telemetry simulator + Kafka producer
├── streaming-analytics/ # Scala + Spark Structured Streaming
├── frontend-dashboard/ # React + TypeScript operations dashboard
├── database/ # SQL and analytical queries
├── monitoring/ # Prometheus + Grafana
├── docs/screenshots/ # Dashboard screenshots
├── .github/workflows/ # GitHub Actions CI
├── docker-compose.yml
└── README.md
Built an end-to-end real-time fleet intelligence platform using Java, Apache Kafka, Kotlin/Spring Boot, PostgreSQL, Scala/Spark and React/TypeScript. The system generates and ingests live aircraft telemetry, performs operational alerting and streaming window analytics, persists relational and analytical data, exposes secured REST APIs and presents fleet state through an interactive operations dashboard. The complete Java → Kafka → Spark → PostgreSQL streaming path was validated locally in Docker.
WebSocket/SSE push updates, Schema Registry with Avro or Protobuf, Kafka dead-letter topics, OpenTelemetry tracing, RBAC, contract/integration testing, partitioned telemetry storage, TimescaleDB, Kubernetes and cloud infrastructure with Terraform.
MIT