[matteius]

lightNVR now keeps a durable record of each object detection, with a stored photo, and delivers it over MQTT at least once. That delivery survives broker outages and restarts. A client can also write its own classification back onto the detection.

This works for any detector label. Each camera's observation_labels setting picks the labels to keep (for example person,car,package), and an empty list keeps every label. The API and events have no bird-specific fields. Any client with the observations.annotate permission can write classifications under its own namespace.

YA-WAMF is the first consumer and the integration we tested end to end: it picks the bird observations, identifies the species, and writes it back as yawamf/species. The PRD for that integration is docs/prd/INT_01_Event_Enrichment_YAWAMF.md, and the contract is docs/integrations/OBSERVATIONS_CONTRACT.md.

The matching YA-WAMF change lives on feature/lightnvr-provider in a separate repo and isn't part of this PR.

What it adds

• Durable observations. Migrations 0095 and 0096 add the tables. Every change is logged in the same transaction as the write, and a relay publishes the log through the existing outbox. Message IDs are deterministic.
• Evidence photos. Detection threads hand the inference frame for each eligible label to a bounded queue without blocking. A worker encodes the JPEG off-thread. Each photo is pending, ready, unavailable or expired, and retention runs per camera.
• Detectors covered. Local detectors, HTTP API detectors and ONVIF events (ONVIF observations have no photo). Startup cleanup removes leftover partial writes.
• API. Observations, cursor-based history, evidence download, capabilities, and PUT /api/observations/{id}/classifications/{namespace}/{name}. The PUT needs the new observations.annotate permission. A cursor that has gone stale, or that came from before a database restore, gets 410 cursor_expired.
• Settings, diagnostics and UI. Evidence is opt-in per camera, with per-camera budgets. Pipeline health appears in metrics and system info. Detection results show the linked classification.
• Docs. docs/integrations/YA_WAMF.md has the deployment recipe and the supported-version matrix.

Testing

• Unit tests: new tests with ASan/UBSan builds for each module.
• Contract and crash tests: a contract and crash-boundary suite, plus Playwright specs for history, reliability and the pipeline (L11).
• End-to-end: tests/integration/yawamf_e2e/ runs real Mosquitto, lightNVR and YA-WAMF as host processes. All 10 scenarios pass (A1–A8, A12, A13, A7-annotate):
  • publish and admit an observation;
  • check that the photo is the frame the detector saw;
  • handle two birds in one frame;
  • replay without creating duplicates;
  • survive the broker down for 100 observations;
  • recover after each process is killed mid-burst;
  • expire evidence and recover from a database restore;
  • write the species back, with owner corrections locked.
• Performance gate (A11): results are in tests/performance/observations/results/.
  • Seven 600 s runs: 3 with evidence off, 2 at the default limits, and 2 with the queue deliberately saturated.
  • No recording gaps in any run. Segment-interval p95 changed by less than 0.1%.
  • Saturation drops all carry a reason.
  • Submit p99 is at most 50 µs. A rejected frame was blocking the detection thread for 1.5–2.6 ms; 9ed02bb4 fixes that.
  • The budgets are now final.

Also in this PR

• WebRTC External IP list (3ddcc03b).
  • The go2rtc External IP setting now takes up to 8 entries, separated by commas, semicolons or whitespace. Each entry becomes its own WebRTC candidate: an IP or hostname with an optional :port, IPv6, * or auto for auto-detect, or stun:host[:port].
  • Previously the setting held one address, so a user reaching the server over both the LAN and Tailscale only got WebRTC and two-way audio on one of those paths; the other fell back to MSE.
  • An invalid list is rejected with a 400 instead of producing a broken or YAML-breaking candidate.
• ONVIF connection test (1c0d0712).
  • POST /api/onvif/device/test used to return a generic 500 for every failure. It now returns a 502 that says what failed: the camera rejected the credentials (HTTP 401/403 or a NotAuthorized SOAP fault), the ONVIF service couldn't be reached, the camera returned no usable profile, or the stream couldn't be opened. Credentials are removed from the message.
  • Some cameras advertise an rtsps:// stream address, for example on port 322, while secure RTSP is turned off. When such an address can't be opened, the test now retries plain rtsp:// on port 554 with the same path.
• Review fixes:
  • Atomic bool flags in the new code are now atomic_int, so the C-to-Rust conversion builds.
  • The CodeQL time-of-check alert in the evidence startup cleanup is fixed by checking and deleting relative to the open directory.
  • The end-to-end harness now closes its log files.
  • The evidence settings text no longer reads as bird-only.

Known gaps and follow-ups

• Database lock latency. The DB-lock probe is advisory in the gate, because two evidence-off runs differ by up to 5% on it. Pooled across all runs, its p95 rises about 10% with evidence on, while the median is unchanged. The likely cause is SQLite checkpoints running while the global lock is held. The fix is to checkpoint outside the lock or batch the worker's commits; that's a follow-up.
• Detection↔observation match. The link between a detection and its observation is a heuristic. Two objects with the same label in the same second both link to the earlier observation.
• Multi-engine API detectors. Those engines never run, so they produce no evidence.
• Not yet run:
  • the Docker mode of the end-to-end suite, in CI;
  • a real HTTP or TFLite bird model;
  • the A14 pilot on a real camera (tests/integration/yawamf_e2e/A14_PILOT_CHECKLIST.md).
• Default permission. Only the Administrator role has observations.annotate by default.

Reviewing

• rust/ isn't touched; CI regenerates it.
• Most of the line count is generated OpenAPI output (web/generated/api/validators.js, include/web/openapi_document_generated.h) and committed performance results.
• int01-event-enrichment-yawamf-plan.md holds the per-task logs.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!