Observability Stack Guide

A complete observability platform with 4 instrumented microservices demonstrating metrics, caching, database monitoring, and inter-service communication tracking.

---

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                              OBSERVABILITY LAYER                             │
│  ┌─────────────────┐    ┌─────────────────┐                                 │
│  │   Prometheus    │───▶│     Grafana     │                                 │
│  │   :9090         │    │     :3000       │                                 │
│  └────────┬────────┘    └─────────────────┘                                 │
│           │ scrapes /metrics every 15s                                      │
└───────────┼─────────────────────────────────────────────────────────────────┘
            │
┌───────────┼─────────────────────────────────────────────────────────────────┐
│           ▼              APPLICATION LAYER                                   │
│  ┌─────────────────┐                                                        │
│  │   API Gateway   │  (Node.js/Express)                                     │
│  │     :3001       │  Entry point for external requests                     │
│  └─────────────────┘                                                        │
│                                                                              │
│  ┌─────────────────┐     HTTP calls      ┌─────────────────┐                │
│  │  Order Service  │────────────────────▶│  User Service   │                │
│  │     :8080       │                     │     :5000       │                │
│  │   (Go)          │                     │ (Python/Flask)  │                │
│  └────────┬────────┘                     └────────┬────────┘                │
│           │                                       │                          │
│           │ HTTP calls                            │ SQL queries              │
│           ▼                                       ▼                          │
│  ┌─────────────────┐                     ┌─────────────────┐                │
│  │ Product Service │                     │   PostgreSQL    │                │
│  │     :3002       │                     │     :5432       │                │
│  │   (Node.js)     │                     └─────────────────┘                │
│  └────────┬────────┘                                                        │
│           │                                                                  │
│           │ cache lookups                                                    │
│           ▼                                                                  │
│  ┌─────────────────┐                                                        │
│  │     Redis       │                                                        │
│  │     :6379       │                                                        │
│  └─────────────────┘                                                        │
└─────────────────────────────────────────────────────────────────────────────┘

---

Services Explained

1. API Gateway (Node.js/Express) - Port 3001

Purpose: Entry point for external API requests. Demonstrates basic RED metrics.

Endpoints:

• GET /health - Health check
• GET /metrics - Prometheus metrics
• GET /api/users/:id - Get user (simulated)
• POST /api/orders - Create order (simulated)

Metrics Exposed:

Metric	Type	Description
http_requests_total	Counter	Total HTTP requests by method, route, status
http_request_duration_seconds	Histogram	Request latency distribution
active_connections	Gauge	Current active connections

---

2. Product Service (Node.js + Redis) - Port 3002

Purpose: Product catalog with Redis caching. Demonstrates cache effectiveness monitoring.

How Caching Works:

1. Request comes in for /api/products/1
2. Check Redis for key product:1
3. Cache HIT: Return cached data immediately (fast!)
4. Cache MISS: Look up in "database", store in Redis with 60s TTL, return

Endpoints:

• GET /health - Health check (includes Redis connection status)
• GET /metrics - Prometheus metrics
• GET /api/products - List all products
• GET /api/products/:id - Get product by ID (uses cache)

Metrics Exposed:

Metric	Type	Description
cache_hits_total	Counter	Times data was found in Redis
cache_misses_total	Counter	Times data was NOT in Redis
http_requests_total	Counter	Total HTTP requests
http_request_duration_seconds	Histogram	Request latency

Why This Matters:

• High cache hit rate = good performance, less DB load
• Low cache hit rate = cache may be too small or TTL too short
• Alert: LowCacheHitRate fires if hit rate < 50% for 10 minutes

---

3. User Service (Python/Flask + PostgreSQL) - Port 5000

Purpose: User management with real PostgreSQL database. Demonstrates database monitoring.

How Connection Pool Works:

• Pool maintains 2-10 reusable database connections
• Instead of opening/closing connections per request (slow), we borrow from pool
• db_connections_active shows how many are currently in use
• If all 10 are in use, new requests must wait

Endpoints:

• GET /health - Health check (includes DB connection status)
• GET /metrics - Prometheus metrics
• GET /api/users - List all users
• GET /api/users/:id - Get user by ID
• POST /api/users - Create new user

Metrics Exposed:

Metric	Type	Description
db_connections_active	Gauge	Connections currently checked out from pool
db_query_duration_seconds	Histogram	SQL query execution time by type and table
http_requests_total	Counter	Total HTTP requests
http_request_duration_seconds	Histogram	Request latency

Why This Matters:

• db_connections_active approaching 10 = connection pool exhaustion imminent
• Slow db_query_duration_seconds = database performance issues
• Alert: HighDBConnectionUsage fires if active connections > 8

---

4. Order Service (Go) - Port 8080

Purpose: Order management that calls other services. Demonstrates inter-service communication monitoring.

How It Works: When you create an order:

1. Order service receives POST request
2. Calls user-service to validate user exists
3. Calls product-service to validate product exists
4. If both succeed, creates the order

Endpoints:

• GET /health - Health check
• GET /metrics - Prometheus metrics
• GET /api/orders - List all orders
• GET /api/orders/:id - Get order by ID
• POST /api/orders - Create order (triggers inter-service calls)

Metrics Exposed:

Metric	Type	Description
http_client_requests_total	Counter	Outbound calls to other services
http_client_request_duration_seconds	Histogram	Outbound call latency by target service
http_requests_total	Counter	Incoming HTTP requests
http_request_duration_seconds	Histogram	Incoming request latency

Why This Matters:

• If order-service is slow, http_client_request_duration_seconds tells you WHICH dependency is slow
• target_service="user-service" slow? Problem is in user-service
• target_service="product-service" slow? Problem is in product-service or Redis

---

Prometheus Alerts

Defined in Docker/prometheus/alerts.yml:

Alert	Condition	Duration	Severity
HighErrorRate	Error rate > 5%	5 minutes	warning
HighLatency	p95 latency > 1 second	10 minutes	warning
ServiceDown	Service unreachable	1 minute	critical
HighDBConnectionUsage	Active DB connections > 8	5 minutes	warning
SlowDBQueries	p95 query time > 500ms	5 minutes	warning
LowCacheHitRate	Cache hit rate < 50%	10 minutes	warning

---

Quick Start

Start Everything

cd Docker
docker compose up -d --build

Check Status

docker compose ps

Stop Everything

docker compose down

Stop and Remove Data

docker compose down -v

---

Testing Commands

Test 1: Cache Behavior (Product Service)

# First request - will be a CACHE MISS (slower, ~100ms)
curl http://localhost:3002/api/products/1
# Response includes: "_cache":"miss"

# Second request - will be a CACHE HIT (faster, ~1ms)
curl http://localhost:3002/api/products/1
# Response includes: "_cache":"hit"

# Try different products
curl http://localhost:3002/api/products/2  # miss
curl http://localhost:3002/api/products/2  # hit
curl http://localhost:3002/api/products/3  # miss

# Check cache metrics
curl -s http://localhost:3002/metrics | grep cache
# cache_hits_total{cache_name="products"} 2
# cache_misses_total{cache_name="products"} 3

Test 2: Database Queries (User Service)

# List all users (seeded with 5 sample users)
curl http://localhost:5000/api/users

# Get specific user
curl http://localhost:5000/api/users/1

# Create new user
curl -X POST http://localhost:5000/api/users \
  -H "Content-Type: application/json" \
  -d '{"username": "newuser", "email": "new@example.com", "full_name": "New User"}'

# Check database metrics
curl -s http://localhost:5000/metrics | grep db_
# db_connections_active 0
# db_query_duration_seconds_bucket{...}

Test 3: Inter-Service Communication (Order Service)

# Create an order (order-service calls user-service AND product-service)
curl -X POST http://localhost:8080/api/orders \
  -H "Content-Type: application/json" \
  -d '{"user_id": 1, "product_id": 2, "quantity": 3}'

# Response includes embedded user and product data from other services

# List all orders
curl http://localhost:8080/api/orders

# Check HTTP client metrics (shows calls to other services)
curl -s http://localhost:8080/metrics | grep http_client
# http_client_requests_total{method="GET",status_code="200",target_service="user-service"} 1
# http_client_requests_total{method="GET",status_code="200",target_service="product-service"} 1

Test 4: Error Handling

# Request non-existent user
curl http://localhost:5000/api/users/999
# {"error": "User 999 not found"}

# Request non-existent product
curl http://localhost:3002/api/products/999
# {"error": "Product 999 not found"}

# Create order with invalid user
curl -X POST http://localhost:8080/api/orders \
  -H "Content-Type: application/json" \
  -d '{"user_id": 999, "product_id": 1, "quantity": 1}'
# {"error": "User 999 not found"}

Test 5: Health Checks

# All services expose /health
curl http://localhost:3001/health  # api-gateway
curl http://localhost:3002/health  # product-service (shows redis_connected)
curl http://localhost:5000/health  # user-service (shows database_connected)
curl http://localhost:8080/health  # order-service

---

View Metrics in Prometheus

Open http://localhost:9090 and try these queries:

Request Rate

rate(http_requests_total[5m])

Error Rate Percentage

sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

Latency Percentiles

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Cache Hit Rate

rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m]))

Active DB Connections

db_connections_active

Outbound Call Duration by Service

histogram_quantile(0.95, sum(rate(http_client_request_duration_seconds_bucket[5m])) by (le, target_service))

---

View Dashboards in Grafana

1. Open http://localhost:3000
2. Login: admin / admin
3. Go to Dashboards → API Gateway Observability

The dashboard shows:

• Request Rate (requests per second)
• Error Rate (percentage of 5xx responses)
• Latency Percentiles (p50, p95, p99)

---

Generate Load for Testing

# Simple load test - 100 requests to product service
for i in {1..100}; do
  curl -s http://localhost:3002/api/products/$((i % 5 + 1)) > /dev/null &
done
wait

# Create 10 orders
for i in {1..10}; do
  curl -s -X POST http://localhost:8080/api/orders \
    -H "Content-Type: application/json" \
    -d "{\"user_id\": $((i % 5 + 1)), \"product_id\": $((i % 5 + 1)), \"quantity\": $i}" > /dev/null &
done
wait

# Check metrics after load
curl -s http://localhost:3002/metrics | grep -E "cache_(hits|misses)_total"
curl -s http://localhost:8080/metrics | grep http_client_requests_total

---

File Structure

OB_ST_M/
├── api-gateway/
│   ├── Dockerfile
│   ├── index.js          # Express server with RED metrics
│   └── package.json
├── product-service/
│   ├── Dockerfile
│   ├── index.js          # Express + Redis + cache metrics
│   └── package.json
├── user-service/
│   ├── Dockerfile
│   ├── app.py            # Flask + PostgreSQL + DB metrics
│   └── requirements.txt
├── order-service/
│   ├── Dockerfile
│   ├── go.mod
│   └── main.go           # Go + HTTP client instrumentation
├── Docker/
│   ├── compose.yml       # All services defined here
│   ├── prometheus/
│   │   ├── prometheus.yml    # Scrape configs
│   │   └── alerts.yml        # Alert rules
│   └── grafana/
│       └── provisioning/
│           ├── dashboards/
│           │   ├── dashboards.yml
│           │   └── api-gateway.json  # Pre-built dashboard
│           └── datasources/
│               └── datasources.yml   # Prometheus datasource
└── OBSERVABILITY_GUIDE.md  # This file

---

Troubleshooting

Service won't start

# Check logs
docker compose logs <service-name>

# Example
docker compose logs user-service

Prometheus not scraping

# Check targets status
curl http://localhost:9090/api/v1/targets | grep health

# Or open http://localhost:9090/targets in browser

Database connection failed

# Check if postgres is running
docker compose ps postgres

# Check postgres logs
docker compose logs postgres

Redis connection failed

# Check if redis is running
docker compose ps redis

# Test redis directly
docker compose exec redis redis-cli ping
# Should return: PONG

Rebuild a specific service

docker compose build <service-name> --no-cache
docker compose up -d <service-name>

---

Key Concepts Demonstrated

1. RED Metrics - Rate, Errors, Duration for every HTTP endpoint
2. Cache Monitoring - Track hit/miss rates to optimize caching
3. Database Pool Monitoring - Prevent connection exhaustion
4. Distributed Tracing Prep - HTTP client instrumentation shows dependency latency
5. Health Checks - Every service reports its status and dependencies
6. Alerting - Pre-configured alerts for common issues

---

Next Steps (from plan.txt)

• Step 5: Add Loki for structured logging
• Step 6: Add Tempo for distributed tracing with OpenTelemetry
• Step 7: Configure full correlation between metrics, logs, and traces
• Step 8: Add Alertmanager for alert routing