Fix observability: route metrics through Cloud Monitoring

[shlbatra]

Summary

The FastAPI server on Cloud Run was pushing OTLP metrics to otel-collector:4317 — a Docker-internal hostname unreachable from Cloud Run. Metrics were silently dropped. This PR unifies all production metrics through a single path: Cloud Monitoring → stackdriver-exporter → Prometheus → Grafana.

Changes

• FastAPI server — environment-aware exporter: CloudMonitoringMetricsExporter in production (Cloud Run), OTLPMetricExporter when OTEL_EXPORTER_OTLP_ENDPOINT is set (local dev)
• Dockerfile.fastapi — removed hardcoded OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 so production defaults to Cloud Monitoring
• requirements.fastapi.txt (both copies) — added opentelemetry-exporter-gcp-monitoring
• docker-compose.observability.yml — switched stackdriver-exporter from env vars to CLI flags (--monitoring.metrics-prefixes), added GCP ADC credential mount, moved OTel Collector behind profiles: ["local-dev"], added custom.googleapis.com/dataflow and workload.googleapis.com metric prefixes
• observability_plan.md — updated architecture diagrams, collection patterns table, metric name mapping, implementation phases, and docker-compose snippets
• README.md — updated observability section with Cloud Monitoring architecture, new start commands, and production metric names

Architecture (before → after)

BEFORE (broken):
  Cloud Run FastAPI → OTLP → otel-collector:4317 → ✖ unreachable

AFTER (unified):
  Cloud Run FastAPI ──→ Cloud Monitoring ──→ stackdriver-exporter ──→ Prometheus ──→ Grafana
  Dataflow Beam     ──→ Cloud Monitoring ──↗
  GCP platform      ──→ Cloud Monitoring ──↗

Verified locally

• stackdriver-exporter UP, scraping 212 GCP metrics including:
  • 33 Beam custom metrics (parse_success, fetch_latency, prediction_latency, etc.)
  • 61 Dataflow system metrics
  • 61 Cloud Run metrics
  • 53 Pub/Sub metrics

Test plan

• stackdriver-exporter starts and Prometheus scrapes it successfully
• Beam custom metrics (custom.googleapis.com/dataflow) appear in Prometheus
• Cloud Run / Dataflow / Pub/Sub platform metrics appear in Prometheus
• Deploy updated FastAPI Docker image to Cloud Run — verify Using Cloud Monitoring exporter in startup logs
• Verify workload_googleapis_com:fastapi_predictions_total appears after sending predictions
• Update Grafana dashboard queries from fastapi_fastapi_* to production metric names

🤖 Generated with Claude Code