docs(ops): production deployment guide (COW-1030)

[brunota20]

Summary

Adds docs/production.md — the operator handbook the
production-hardening milestone has been pointing at since M2.
Sister doc to docs/06-production-hardening.md: that existing
file is the architecture / design rationale; this is the
concrete operator handbook (unit files, backup recipes, alert
rules, runbook procedures). Cross-referenced both ways so an
operator reading either lands on the right next step.

Stacks on #44 (COW-1064 E2E scaffold). Linear: COW-1030.

Sections (12 total)

1. Pre-flight checklist — release binary, persistent state
   dir, metrics on loopback, paid RPC, observability pipeline.
2. systemd unit — full shepherd.service with dedicated
   user, SIGINT for graceful shutdown (30 s timeout covers
   the COW-1072 last-block-persistence path), filesystem
   hardening, 2 G memory cap (defence in depth over wasmtime's
   per-module 64 MiB).
3. Docker Compose — interim Dockerfile + compose stack
   with bundled Prometheus, loopback-only port mapping,
   stop_signal: SIGINT, /metrics-based healthcheck. Marked
   interim because the official Dockerfile is tracked
   separately.
4. redb backup — three operationally-supported paths
   (cold/hot/restore + integrity check) with the redb 2.6
   limitations stated honestly. Retention policy: 7 daily / 4
   weekly / 12 monthly ≈ 2.3 GiB on a 100 MiB store.
5. Logs — JSON shape, two-tier retention model (7 d hot
   full debug + 90 d cold INFO-only on S3 Glacier), Vector
   config sample for the split. Sizing based on the E2E run
   shape.
6. RPC selection — provider tier recommendations, per-chain
   capacity sizing, why public nodes are non-starters.
7. Metrics + scraping — complete metric surface table
   verified against grep metrics::(counter|histogram|gauge)!:
   • shepherd_event_latency_seconds{module, event_kind}
   • shepherd_module_errors_total{module, error_kind}
   • shepherd_module_restarts_total{module}
   • shepherd_module_poisoned{module}
   • shepherd_chain_request_total{chain_id, method, outcome}
   • shepherd_cow_api_submit_total{chain_id, outcome}
   • shepherd_stream_reconnects_total{kind, chain_id, module?}
8. Workload-class tuning — light indexer / TWAP-style /
   multi-chain swarm classes with concrete fuel + memory
   numbers. Honest that limits are compile-time constants
   today and per-module overrides via [engine.limits] are a
   0.3 follow-up.
9. Alert rules — full prometheus-rules.yml (7 alerts,
   page vs ticket severity). ShepherdModulePoisoned +
   ShepherdDown page; the rest ticket.
10. Operational runbook — five common tasks: tail-per-module,
    reset poisoned module, add module to running deploy,
    inspect local-store, bump log level. Each carries a
    concrete shell snippet.
11. Pre-upgrade checklist — staged-binary smoke before
    promoting.
12. References — links to the architecture doc, ADRs,
    runbooks, sister Linear issues.

Drive-by fix

The COW-1064 e2e report template (committed in #44) had two
metric-label mistakes I caught while writing this PR's section
7 metric surface table:

• result=\"ok|err\" -> outcome=\"ok|err\" (the actual label
  name in cow_api.rs + chain.rs).
• reason=\"trap\" -> error_kind=\"trap\" (the actual label
  name in supervisor.rs).

Both labels appear in 4 places in the template; fixed
in-place rather than as a separate PR.

Workspace impact

• No code changes (pure documentation).
• No new build dependencies.
• cargo fmt --all --check clean.

AI assistance disclosure

Claude (Opus 4.7, 1M-context) drafted docs/production.md
working from: the issue's spec (systemd / Docker Compose /
redb backup / log retention / workload tuning / alert
templates), the existing docs/06-production-hardening.md
shape, the actual metric surface (verified via grep metrics::(counter|histogram|gauge)! over the engine crate),
the actual runtime constants (DEFAULT_FUEL_PER_EVENT,
DEFAULT_MEMORY_LIMIT, POISON_MAX_FAILURES,
POISON_WINDOW, RESTART_MAX_BACKOFF), and the existing
testnet runbooks for shape consistency. I reviewed every
section, verified each shell snippet would work as written
(systemd directives, docker compose schema, journalctl + jq
filter syntax, redb integrity API), and the drive-by metric-
label fix in the e2e report template came from cross-checking
the table against the source.

[linear-code]

COW-1030