fix(waf): detect stale seed config and report version skew (#312, #316)

CHANGELOG.md

@@ -9,8 +9,29 @@ spelling of the same version (e.g. `0.1.0b2` for `0.1.0-beta.2`).
 
 ## [Unreleased]
 
+### Added
+
+- `GET /runtime/status` reports `version_agreement` (`app_version`,
+  `configured_image_tag`, `crs_version`) and warns when the tag in
+  `GUARD_PROXY_IMAGE_TAG` and the backend's own version disagree, so a stack
+  mixing images from two releases is visible instead of assumed absent.
+- The generated CRS metadata (`ver:'OWASP_CRS/...'` and
+  `tx.crs_setup_version`) comes from one constant that a test pins to the
+  `configs/coraza/crs` submodule, replacing the hardcoded `4.25.0` / `4250` in
+  the template. Bumping the submodule without the constant now fails a test
+  instead of silently mislabelling the rules that load.
+
 ### Fixed
 
+- A tag-only upgrade (new images, old bind-mounted `haproxy/` files) no longer
+  runs silently with the `default` Coraza application instead of each vhost's
+  `policy_<id>`. The shipped `haproxy/haproxy.cfg` and `haproxy/coraza.cfg`
+  carry a `# guard-proxy-config-version: N` stamp; the backend compares the
+  mounted `coraza.cfg` with the version its image expects at startup, in
+  `GET /health` (`degraded`, naming the file), in `GET /ready` (fails, so the
+  release kit's backend healthcheck fails too; the path stays in the server
+  log like the other readiness checks) and in `POST /config/apply` (`409`,
+  naming the file). See `release/README.md`.
 - Certificate renewal registers its ACME account with the same contact the
   certificate was issued with. The renewal job used to invent
   `admin@<domain>` for every vhost, which Let's Encrypt refuses for reserved

configs/haproxy/coraza.cfg

@@ -1,3 +1,4 @@
+# guard-proxy-config-version: 1
 # SPOE engine configuration for the Coraza SPOA.
 #
 # This file is referenced from haproxy.cfg via:

configs/haproxy/haproxy.cfg

@@ -1,3 +1,4 @@
+# guard-proxy-config-version: 1
 # Guard Proxy HAProxy configuration.
 # Generated from the policy database; manual edits are overwritten on the
 # next config apply.

docs/architecture.md

@@ -380,19 +380,30 @@ The backend exposes two probe endpoints, both unauthenticated:
 
 | Endpoint | Type | Returns | Checks |
 |----------|------|---------|--------|
-| `GET /health` | Liveness | 200 always | None — proves the process is alive |
-| `GET /ready` | Readiness | 200 / 503 | DB connectivity (`SELECT 1`), runtime config volume writable |
+| `GET /health` | Liveness | 200 always | Process aliveness, plus the seed config version stamp (issue #312) |
+| `GET /ready` | Readiness | 200 / 503 | DB connectivity (`SELECT 1`), runtime config volume writable, seed config version stamp |
 
 `/ready` returns a JSON body with per-check status so operators can tell which dependency is down:
 
 ```json
 // 200 — all clear
-{"status": "ready", "checks": {"database": {"status": "ok"}, "runtime_config": {"status": "ok"}}}
+{"status": "ready", "checks": {"database": {"status": "ok"}, "runtime_config": {"status": "ok"}, "seed_config": {"status": "ok"}}}
 
 // 503 — one or more dependencies unavailable
-{"status": "not ready", "checks": {"database": {"status": "ok"}, "runtime_config": {"status": "error", "detail": "/var/lib/guard-proxy/generated is not a writable directory"}}}
+{"status": "not ready", "checks": {"database": {"status": "ok"}, "runtime_config": {"status": "error", "detail": "runtime config directory is not writable"}, "seed_config": {"status": "ok"}}}
 ```
 
+Both probes also compare the bind-mounted seed `coraza.cfg` with the
+`guard-proxy-config-version` stamp this image expects. A missing or older
+stamp — the tag-only upgrade of issue #312, where HAProxy would silently
+inspect every request with the `default` Coraza application instead of the
+vhost's `policy_<id>` — makes `/health` report `degraded` with the stale file
+named and `/ready` fail. The readiness body stays generic like its other
+checks (the path is logged server-side); the file name remains available to an
+authenticated admin through `/health`, `GET /runtime/status` and the
+`POST /config/apply` error, which refuses with `409`. Outside a release kit the
+file is absent and the check is skipped.
+
 The Docker Compose `healthcheck` for the `backend` service targets `/ready`. Almost every other service in the stack (`frontend`, `haproxy`, `coraza`, `log-shipper`) depends on `backend` with `condition: service_healthy`, so `/ready` must pass before dependents start. This prevents dependents from launching while the database is still initialising or the config volume is unavailable.
 
 `/health` is retained as a lightweight liveness probe for orchestrators that separately track process aliveness (e.g. a future Kubernetes `livenessProbe`).

release/README.md

@@ -119,10 +119,33 @@ it — the static `haproxy.cfg` seed is replaced at that point.
    breaking-change callouts.
 2. Back up the `pgdata` volume (`docker compose exec postgres pg_dump ...`
    or a volume snapshot) before upgrading, out of caution during beta.
-3. Update `GUARD_PROXY_IMAGE_TAG` in `.env` to the new version.
+3. Update `GUARD_PROXY_IMAGE_TAG` in `.env` to the new version **and** replace
+   the `haproxy/` directory (`haproxy.cfg` and `coraza.cfg`) with the one from
+   that release. The tag and the files are one unit: the config files are
+   bind-mounted into the containers, not shipped in the images.
 4. `docker compose pull && docker compose up -d`. The backend runs
    `alembic upgrade head` automatically on startup.
 
+### A tag-only upgrade is refused
+
+The shipped `haproxy/haproxy.cfg` and `haproxy/coraza.cfg` carry a
+`# guard-proxy-config-version: N` stamp, and the backend knows the version its
+image expects. If you bump the tag but keep the old files, `GET /health`
+reports `degraded` naming the stale file, `GET /ready` — which the backend
+container's healthcheck calls — fails, and `POST /config/apply` answers `409`
+naming the file. The readiness body itself stays generic; the `backend`
+container log carries the path. A tag-only upgrade therefore leaves the stack
+reporting unhealthy and refusing to start: `frontend`, `haproxy`, `coraza` and
+`log-shipper` wait for `backend` to become healthy, so nothing runs until the
+`haproxy/` files from the same release are put in place and the backend is
+restarted.
+
+The check exists because the failure it catches is otherwise invisible: an old
+`coraza.cfg` tells HAProxy to inspect every request with the `default` Coraza
+application instead of the vhost's `policy_<id>`. Traffic is still inspected
+and attacks are still blocked, so nothing looks broken — the vhost's own
+paranoia level, thresholds and rule overrides are simply not applied.
+
 ### What an upgrade does not carry over
 
 Everything you configure in the admin UI survives an upgrade: it lives in

release/haproxy/coraza.cfg

@@ -1,3 +1,4 @@
+# guard-proxy-config-version: 1
 # SPOE engine configuration for the Coraza SPOA.
 #
 # This file is referenced from haproxy.cfg via:

release/haproxy/haproxy.cfg

@@ -1,3 +1,4 @@
+# guard-proxy-config-version: 1
 # Guard Proxy HAProxy configuration.
 # Generated from the policy database; manual edits are overwritten on the
 # next config apply.

src/backend/app/config.py

@@ -107,6 +107,12 @@ class Settings(EnvFileSettings):
     haproxy_reload_timeout_seconds: int = 10
     haproxy_stats_socket_path: str = "/var/run/haproxy/admin.sock"
     haproxy_stats_timeout_seconds: int = 10
+    # Bind-mounted seed SPOE config from the release kit, checked against
+    # EXPECTED_CONFIG_VERSION (issue #312). Absent outside a container.
+    seed_coraza_config_path: str = "/usr/local/etc/haproxy/coraza.cfg"
+    # Image tag the operator pulled, compared with this backend's own version
+    # in /runtime/status. Set by the release kit's .env; unset in dev.
+    guard_proxy_image_tag: str | None = None
     log_retention_days: int = 30
     # Log level written into the generated coraza-spoa.yaml (debug, info,
     # warn, error). The debug compose overlay sets CORAZA_LOG_LEVEL=debug.

src/backend/app/constants/versions.py

@@ -0,0 +1,18 @@
+"""Versions this backend expects the rest of the stack to agree with.
+
+`EXPECTED_CONFIG_VERSION` is the stamp every shipped `haproxy.cfg` /
+`coraza.cfg` carries; bump it together with those files (see
+`configs/haproxy/coraza.cfg`) when the contract they implement changes. The
+backend refuses to report healthy for a bind-mounted seed config that does not
+carry the expected stamp (issue #312).
+
+`CRS_VERSION` must match the `configs/coraza/crs` submodule; bump both
+together, because the generated CRS metadata is derived from it and the
+submodule supplies the actual rules (issue #316).
+"""
+
+from __future__ import annotations
+
+EXPECTED_CONFIG_VERSION = 1
+
+CRS_VERSION = "4.25.0"

src/backend/app/main.py

@@ -1,10 +1,7 @@
 import logging
 import os
-import tomllib
 from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
-from importlib.metadata import PackageNotFoundError
-from importlib.metadata import version as _package_version
 from pathlib import Path
 
 from fastapi import Depends, FastAPI
@@ -47,37 +44,13 @@
 )
 from app.services.config_apply import seed_runtime_config
 from app.services.config_generator import generate
+from app.services.config_version import check_seed_config
 from app.services.scheduler import start_scheduler, stop_scheduler
+from app.version import APP_VERSION
 
 logger = logging.getLogger(__name__)
 
 
-def _resolve_app_version() -> str:
-    """Return the app version, reading from a single source of truth.
-
-    Prefers installed package metadata (present in dev venvs where the
-    project is installed editable). The production Docker image installs
-    only dependencies as a distinct layer and copies app source on top
-    without installing the project itself, so no dist-info exists there —
-    fall back to parsing `pyproject.toml` directly in that case.
-    """
-    try:
-        return _package_version("guard-proxy-backend")
-    except PackageNotFoundError:
-        pyproject_path = Path(__file__).resolve().parent.parent / "pyproject.toml"
-        try:
-            with pyproject_path.open("rb") as f:
-                version = tomllib.load(f)["project"]["version"]
-            if not isinstance(version, str):
-                raise KeyError("project.version is not a string")
-            return version
-        except (OSError, KeyError):  # pragma: no cover - defensive fallback
-            return "0.0.0+unknown"
-
-
-APP_VERSION = _resolve_app_version()
-
-
 class _HealthcheckAccessFilter(logging.Filter):
     def filter(self, record: logging.LogRecord) -> bool:
         args = record.args
@@ -105,6 +78,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
             "sent over plain HTTP. Enable it once the admin UI is served over "
             "HTTPS."
         )
+    seed_config = check_seed_config()
+    if not seed_config.ok:
+        # Log once at startup; the probes re-read the file, so replacing it
+        # clears the degraded state without a restart.
+        logger.error("Refusing to report healthy: %s", seed_config.detail)
     _seed_runtime_config()
     start_scheduler()
     yield
@@ -183,6 +161,7 @@ def _reconcile_runtime_operation_checksum(
 
 
 
+
 app = FastAPI(
     title=settings.app_name,
     version=APP_VERSION,
@@ -219,10 +198,22 @@ def _reconcile_runtime_operation_checksum(
 
 @app.get("/health")
 def health_check() -> dict[str, str]:
-    """Liveness probe — always returns 200 if the process is running."""
+    """Liveness probe — always returns 200 if the process is running.
+
+    Reports `degraded` when the bind-mounted seed config does not match this
+    image (issue #312): the process is alive, but HAProxy may be enforcing a
+    different policy set than the one this release expects.
+    """
+    seed_config = check_seed_config()
+    if seed_config.ok:
+        return {
+            "status": "healthy",
+            "version": APP_VERSION,
+        }
     return {
-        "status": "healthy",
+        "status": "degraded",
         "version": APP_VERSION,
+        "detail": seed_config.detail or "",
     }
 
 
@@ -262,6 +253,21 @@ def readiness_check(db: Session = Depends(get_db)) -> JSONResponse:
         }
         all_ok = False
 
+    # Check 3: the bind-mounted seed config matches this image (issue #312).
+    # The release kit's backend healthcheck calls this endpoint, so a half-done
+    # upgrade leaves the stack unhealthy instead of silently enforcing the
+    # wrong Coraza application.
+    seed_config = check_seed_config()
+    if seed_config.ok:
+        checks["seed_config"] = {"status": "ok"}
+    else:
+        logger.error("Readiness check failed: %s", seed_config.detail)
+        checks["seed_config"] = {
+            "status": "error",
+            "detail": "seed config version does not match this image",
+        }
+        all_ok = False
+
     return JSONResponse(
         content={
             "status": "ready" if all_ok else "not ready",

src/backend/app/routers/config.py

@@ -1,5 +1,6 @@
 """Config apply router — triggers generate + apply pipeline."""
 
+import logging
 import threading
 
 from fastapi import APIRouter, Depends, HTTPException, status
@@ -24,6 +25,9 @@
 from app.services.config_apply import ApplyResult, ApplyStatus
 from app.services.config_apply import apply as _apply
 from app.services.config_generator import generate
+from app.services.config_version import check_seed_config
+
+logger = logging.getLogger(__name__)
 
 router = APIRouter(prefix="/config", tags=["config"])
 
@@ -92,6 +96,17 @@ def apply_config(
             detail="Another config apply is already in progress",
         )
     try:
+        seed_config = check_seed_config()
+        if not seed_config.ok:
+            # HAProxy reads the bind-mounted seed config at runtime, so a
+            # stale one applies the wrong Coraza application no matter what
+            # this release generates (issue #312).
+            logger.error("Refusing config apply: %s", seed_config.detail)
+            raise HTTPException(
+                status_code=status.HTTP_409_CONFLICT,
+                detail=f"Refusing to apply: {seed_config.detail}",
+            )
+
         vhosts = db.query(VHost).options(selectinload(VHost.backends)).all()
         policies = db.query(Policy).all()
         rule_overrides = db.query(RuleOverride).all()

src/backend/app/schemas/runtime_status.py

@@ -43,6 +43,21 @@ class ActiveRuntimeConfig(BaseModel):
     warning: str | None = None
 
 
+class VersionAgreement(BaseModel):
+    """Versions the stack pieces must agree on (issue #316).
+
+    `configured_image_tag` is the tag the operator wrote into `.env`; it is
+    None when the backend is not run from the release kit. `warning` is set
+    when that tag and this backend's own version disagree, which means the
+    stack mixes images from different releases.
+    """
+
+    app_version: str
+    configured_image_tag: str | None
+    crs_version: str
+    warning: str | None = None
+
+
 class RuntimeOperationSnapshot(BaseModel):
     """Latest persisted status for a specific runtime operation type."""
 
@@ -62,5 +77,6 @@ class RuntimeStatusResponse(BaseModel):
     deployment_state: DeploymentState
     generated_config: RuntimeGeneratedConfigStatus
     active_config: ActiveRuntimeConfig
+    version_agreement: VersionAgreement
     latest_validation: RuntimeOperationSnapshot | None = None
     latest_reload: RuntimeOperationSnapshot | None = None

src/backend/app/services/config_renderer.py

@@ -8,6 +8,7 @@
 
 from jinja2 import Environment, PackageLoader, StrictUndefined
 
+from app.constants.versions import CRS_VERSION, EXPECTED_CONFIG_VERSION
 from app.coraza_syntax import VARIABLES_PATTERN, quoted_value_error
 from app.haproxy_syntax import host_error
 from app.models.custom_rule import RuleOperator, RulePhase
@@ -483,7 +484,17 @@ def render_haproxy_cfg_multi(vhost_contexts: list[HaproxyRenderContext]) -> str:
 
 def _render_haproxy_routes(routes: tuple[HaproxyRoute, ...]) -> str:
     template = _ENVIRONMENT.get_template("haproxy.cfg.j2")
-    return template.render(routes=routes, default_waf_app=CORAZA_DEFAULT_APP)
+    return template.render(
+        routes=routes,
+        default_waf_app=CORAZA_DEFAULT_APP,
+        config_version=EXPECTED_CONFIG_VERSION,
+    )
+
+
+def _crs_setup_version(version: str) -> int:
+    """CRS setup version integer derived from a release version (`4.25.0` -> 4250)."""
+    major, minor, patch = (int(part) for part in version.split("."))
+    return major * 1000 + minor * 10 + patch
 
 
 def render_crs_setup(policy: CrsPolicyRenderContext) -> str:
@@ -494,7 +505,12 @@ def render_crs_setup(policy: CrsPolicyRenderContext) -> str:
         if policy.enforcement_mode == PolicyEnforcementMode.detect_only
         else "On"
     )
-    return template.render(policy=policy, sec_rule_engine=sec_rule_engine)
+    return template.render(
+        policy=policy,
+        sec_rule_engine=sec_rule_engine,
+        crs_version=CRS_VERSION,
+        crs_setup_version=_crs_setup_version(CRS_VERSION),
+    )
 
 
 def render_coraza_spoa_yaml(app_names: tuple[str, ...], log_level: str) -> str: