Fix observability: route metrics through Cloud Monitoring

[shlbatra]

Summary

The FastAPI server on Cloud Run was pushing OTLP metrics to otel-collector:4317 — a Docker-internal hostname unreachable from Cloud Run. Metrics were silently dropped. This PR unifies all production metrics through a single path: Cloud Monitoring → stackdriver-exporter → Prometheus → Grafana.

Changes

• FastAPI server — environment-aware exporter: CloudMonitoringMetricsExporter in production (Cloud Run), OTLPMetricExporter when OTEL_EXPORTER_OTLP_ENDPOINT is set (local dev)
• Dockerfile.fastapi — removed hardcoded OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 so production defaults to Cloud Monitoring
• requirements.fastapi.txt (both copies) — added opentelemetry-exporter-gcp-monitoring
• docker-compose.observability.yml — switched stackdriver-exporter from env vars to CLI flags (--monitoring.metrics-prefixes), added GCP ADC credential mount, moved OTel Collector behind profiles: ["local-dev"], added custom.googleapis.com/dataflow and workload.googleapis.com metric prefixes
• observability_plan.md — updated architecture diagrams, collection patterns table, metric name mapping, implementation phases, and docker-compose snippets
• README.md — updated observability section with Cloud Monitoring architecture, new start commands, and production metric names

How to Review

1. Start with fastapi_server.py — the core fix (environment-aware exporter)
2. docker-compose.observability.yml — CLI flags fix for stackdriver-exporter v0.19+
3. observability_plan.md — architecture diagrams and updated plan
4. Readme.md — updated observability section

Architecture (before → after)

BEFORE (broken):
  Cloud Run FastAPI → OTLP → otel-collector:4317 → ✖ unreachable

AFTER (unified):
  Cloud Run FastAPI ──→ Cloud Monitoring ──→ stackdriver-exporter ──→ Prometheus ──→ Grafana
  Dataflow Beam     ──→ Cloud Monitoring ──↗
  GCP platform      ──→ Cloud Monitoring ──↗

Testing

• stackdriver-exporter starts and Prometheus scrapes it successfully
• Beam custom metrics (33 metrics — parse_success, fetch_latency, prediction_latency, etc.) appear in Prometheus
• Cloud Run (61), Dataflow (61), Pub/Sub (53) platform metrics appear in Prometheus
• Syntax check (python -m py_compile) on new/modified files
• Deploy updated FastAPI Docker image to Cloud Run — verify Using Cloud Monitoring exporter in startup logs
• Verify workload_googleapis_com:fastapi_predictions_total appears after sending predictions
• Update Grafana dashboard queries from fastapi_fastapi_* to production metric names

Dependencies

• opentelemetry-exporter-gcp-monitoring>=1.9.0a0 (new — production metrics exporter)

🤖 Generated with Claude Code