test: characterize pg-erd partial upstream response

CHANGELOG.md

@@ -22,7 +22,8 @@ All notable changes are tracked here. No release has been published yet.
 - Added dedicated compiled pg-erd traffic acceptance for streamed/chunked body overflow and routed in-flight saturation/recovery: the migration process must return 413 above the shared body budget, return 503 in less than one second above the in-flight budget, keep `/readyz` observable, expose the exact single-rejection Prometheus sample, and admit a later routed request after capacity is released.
 - Added dedicated compiled pg-erd refused-origin recovery acceptance using a Linux TCP socket bound to the characterized backend address without entering LISTEN state. The fixture first proves direct `ECONNREFUSED` while retaining exclusive port ownership, then requires the migration gateway to return 502 within a conservative one-second envelope around the configured 200/400 ms connection budgets, keep `/readyz` 200, expose the exact single-error Prometheus sample, and allow a later independent frontend route to recover. Connected read stall, TCP reset, partial-response/streaming failure, retry and failover behavior remain separate gaps.
 - Added dedicated compiled pg-erd connected read-stall acceptance with `read_ms=100`: the backend accepts the routed request and remains open without response bytes until the gateway has already failed it, preventing fixture closure from faking timeout behavior. The contract requires 502 inside a conservative one-second envelope, preserved `/readyz`, the exact single-error Prometheus sample, and independent frontend recovery. Pingora `read_timeout` remains a per-read inactivity budget, not a whole-response lifetime; reset, partial-response and slow-drip cases remain open.
-- Added dedicated routed pg-erd graceful-drain acceptance: a characterized `/api/held` backend request is held in flight, SIGTERM is sent only after the backend has accepted it, the response is released during the shared grace period, the downstream must still receive HTTP 200, and the migration process must exit successfully inside the external termination budget. Generic drain evidence is not transferred to this composition root; the contract remains source-defined until the unchanged exact head reaches terminal hosted GREEN.
+- Added dedicated compiled pg-erd post-header partial-response acceptance: the backend commits HTTP 200 with exactly one `Content-Length: 20` field, writes only `partial`, and closes only after the downstream has observed that committed header/body prefix. The downstream must retain the committed status and the single exact framing field, then terminate before body completion instead of receiving an invented second status or silent failover; `/readyz`, exact low-cardinality request-error telemetry, and an independent `frontend` route must remain usable. `X-Content-Length` and duplicate/conflicting `Content-Length` fields cannot satisfy the framing oracle. Explicit TCP reset, broader streaming/upgraded failure, and slow-drip/whole-response lifetime remain separate gaps.
+- Added dedicated routed pg-erd graceful-drain acceptance: a characterized `/api/held` backend request is held in flight, SIGTERM is sent only after the backend has accepted it, the response is released during the shared grace period, the downstream must still receive HTTP 200, and the migration process must exit successfully inside the external termination budget measured from SIGTERM. Generic drain evidence is not transferred to this composition root.
 - Added optional per-upstream absolute PEM trust-bundle consumption without taking ownership of certificate issuance/rotation; trust material is loaded fail-closed before listeners open.
 - Added an executable local-CA TLS test through the compiled gateway that holds CA trust constant and proves SNI/hostname mismatch is rejected.
 - Added a focused transport-adapter regression proving an upstream without a custom trust bundle leaves Pingora's platform trust roots selected rather than replacing the CA store.
@@ -31,11 +32,11 @@ All notable changes are tracked here. No release has been published yet.
 - Added request-body limits and a distrust-by-default forwarded-header policy.
 - Added low-cardinality metrics plus credential/cookie-safe access logging through the production path.
 - Overrode Pingora framework retry/drain defaults with one total upstream attempt, a 5-second SIGTERM grace period, and a 30-second graceful-shutdown timeout.
-- Added non-root/read-only-root OCI packaging with an explicit build-time allowlist for the generic and bounded pg-erd process identities. Exact-head OCI acceptance now builds and starts each image under uid/gid 65532, dropped capabilities and `no-new-privileges`; the supply-chain lane builds and vulnerability-scans both candidate images. This is unreleased source-defined acceptance until the current head reaches terminal hosted GREEN.
+- Added non-root/read-only-root OCI packaging with an explicit build-time allowlist for the generic and bounded pg-erd process identities. Exact-head OCI acceptance builds and starts each image under uid/gid 65532, dropped capabilities and `no-new-privileges`; the supply-chain lane builds and vulnerability-scans both candidate images.
 - Extended the dedicated pg-erd OCI acceptance so the least-privilege migration container is not accepted until its process-health `/livez` endpoint answers and the separately published `/metrics` listener identifies the Pingora Prometheus service through its `text/plain` response media type. The check deliberately does not require a metric family before application traffic has emitted one, and it preserves the current one-binary-per-image packaging boundary.
 - Added a committed dependency lock, fail-closed license/source/advisory policy, exact-source SBOM and image-vulnerability evidence.
 - Added an exact-head owned-production coverage gate that requires 100% lines and regions without filename/function/branch exclusions; repaired compiler-generated generic startup coverage and structurally impossible literal-header error regions rather than weakening the gate.
 - Added missing-public-rustdoc enforcement and documentation builds with warnings denied.
 - Added DDD, product, technical, security, threat, test, operability, configuration, migration-gap, and primary-source traceability documentation.
 
-Release remains blocked on the organization decision for the exact Pingora release versus `RUSTSEC-2026-0253` and the separate time-bounded disposition of unmaintained `derivative 2.2.0` / `RUSTSEC-2024-0388` (`ContextualWisdomLab/.github#1605`), restoration of authoritative public non-fork Dependency Review evidence (`ContextualWisdomLab/.github#810`), terminal exact-current-head CI/supply-chain/security/review evidence, representative compiled-binary pg-erd route/header/forwarding/body/backpressure/failure/concurrency/drain and benchmark evidence, an immutable registry digest with provenance and rehearsed rollback, and protected-branch integration. Routed graceful drain now has source acceptance but remains uncredited until terminal exact-head execution. No consumer migration, canary, cutover, or legacy removal is claimed before those release and traffic-contract gates are satisfied.
+Release remains blocked on the organization decision for the exact Pingora release versus `RUSTSEC-2026-0253` and the separate time-bounded disposition of unmaintained `derivative 2.2.0` / `RUSTSEC-2024-0388` (`ContextualWisdomLab/.github#1605`), restoration of authoritative public non-fork Dependency Review evidence (`ContextualWisdomLab/.github#810`), terminal exact-current-head CI/supply-chain/security/review evidence, representative compiled-binary pg-erd route/header/forwarding/body/backpressure/failure/concurrency/drain and benchmark evidence, an immutable registry digest with provenance and rehearsed rollback, and protected-branch integration. Parent #20 has moved by ordinary repair to `ab70a6c75da1677374597d2302a18a59fe4c3850` and must reacquire its own exact-head evidence; the changed #21 partial-response candidate must independently reacquire exact-head execution/review evidence as well. No predecessor GREEN transfers across either movement. No consumer migration, canary, cutover, or legacy removal is claimed before those release and traffic-contract gates are satisfied.

TEST_STRATEGY.md

@@ -16,12 +16,14 @@ The bounded Admin Config transition has its own executable contract. `tests/pg_e
 
 `tests/pg_erd_read_stall_traffic.rs` separates connected upstream inactivity from refusal. The characterized backend accepts `/api/read-stall`, reads the request headers, then remains connected and sends no response bytes until the test explicitly releases it after the gateway has already returned. With `read_ms=100`, the gateway must fail as HTTP 502 inside a conservative one-second outer envelope, preserve `/readyz`, expose the exact Prometheus sample `cwl_pingora_gateway_request_errors_total 1`, and leave an independent `frontend` route usable. Keeping the fixture connection open prevents origin closure from masquerading as the timeout. Because Pingora's `read_timeout` is per successful `read()` rather than a whole-response lifetime, reset, post-commit partial response, slow-drip and whole-response deadline behavior remain separate contracts.
 
-On Unix, `tests/pg_erd_graceful_shutdown.rs` proves the bounded pg-erd composition root consumes the shared drain policy instead of borrowing generic-binary evidence. The test routes `/api/held` to the characterized backend, holds that response open until the backend has confirmed the in-flight request, then delivers SIGTERM and releases the response during the shared grace period. The downstream must still receive HTTP 200 and the migration process must exit successfully inside the external termination budget measured from SIGTERM. This is source-defined drain acceptance until the unchanged exact head executes it to terminal GREEN.
+`tests/pg_erd_partial_response_traffic.rs` covers the distinct post-header failure phase. The characterized backend sends HTTP 200 with `Content-Length: 20` and the body prefix `partial`, then remains open until the downstream reader has observed a complete response header block and that exact body prefix. Only after this acknowledgement does the fixture release the backend to close normally; EOF/reset before acknowledgement fails the test. This prevents scheduler/socket-buffer timing from masquerading as post-commit evidence. Acceptance requires the downstream to retain the committed status/framing and terminate before all 20 bytes arrive rather than receiving an invented second status or silent route failover. The framing oracle parses individual header lines from the original header block, matches `Content-Length` field names case-insensitively, trims field-value optional whitespace, requires exactly one value equal to `20`, and includes a focused regression proving `X-Content-Length` plus duplicate/conflicting Content-Length fields cannot false-pass. `/readyz` must stay HTTP 200, the shared request-error counter must expose exactly `cwl_pingora_gateway_request_errors_total 1`, and an independent `frontend` request must still complete successfully. Traffic and metrics ports are reserved simultaneously before process startup so ephemeral-port reuse cannot manufacture an invalid listener collision. This orderly-close contract does not transfer evidence to explicit TCP reset, WebSocket/Upgrade, broader streaming failure, or slow-drip/whole-response lifetime.
+
+On Unix, `tests/pg_erd_graceful_shutdown.rs` proves the bounded pg-erd composition root consumes the shared drain policy instead of borrowing generic-binary evidence. The test routes `/api/held` to the characterized backend, holds that response open until the backend has confirmed the in-flight request, then creates one absolute termination deadline, delivers SIGTERM, and releases the response during the shared grace period. The downstream must still receive HTTP 200 and the migration process must exit successfully before that SIGTERM-relative external termination deadline. Its traffic and metrics reservation sockets are held simultaneously before startup so the fixture cannot false-fail through ephemeral-port reuse. Current parent #20 exact `ab70a6c75da1677374597d2302a18a59fe4c3850` is an ordinary ancestry/single-writer repair and must independently reacquire hosted evidence; historical `d4d4565...` GREEN is predecessor evidence only and does not transfer.
 
 The `oci-runtime` job separately validates artifact composition instead of inferring it from compiled-process tests. The Dockerfile admits only `cwl-pingora-gateway` and `cwl-pingora-pg-erd-migration` as build-time process identities, normalizes the selected executable to one fixed distroless runtime path, and CI builds both image profiles. Each exact candidate must declare uid/gid `65532`, start under a read-only root filesystem with all capabilities dropped and `no-new-privileges`, and consume only a read-only configuration mount. The generic profile must expose local `/livez`. The pg-erd profile must expose local `/livez` on the traffic listener and its separately published `/metrics` listener must identify the Pingora Prometheus service by a `text/plain` response media type before the container is accepted. `tests/pg_erd_oci_metrics_workflow_contract.rs` prevents a bare HTTP 200 from false-greening a mistakenly bound proxy service while deliberately avoiding a metric-family requirement before routed application traffic has emitted one. `examples/pg-erd-migration.yaml` is deliberately an origin-independent OCI smoke fixture; this gate proves the dedicated binary is actually packaged and both process/observability listeners start under the required container isolation but does not claim routed pg-erd parity, origin health, or performance. The supply-chain job additionally builds and vulnerability-scans both image profiles and binds both local image IDs plus per-image scan outputs to the exact source SHA. These workflow contracts count only after terminal success on the unchanged exact head.
 
 `tests/load/gateway_smoke.js` is a separate concurrent traffic contract executed with checksum-pinned k6 2.2.0 against the release-mode generic gateway binary and `tests/load/upstream_fixture.py`, a deterministic local HTTP/1.1 upstream. The contract sends 400 requests across four virtual users, requires every response to preserve the expected status/body, requires zero failed HTTP requests, and gates loopback `http_req_duration` p95 below 20 ms. Its gateway fixture supplies explicit downstream in-flight and upstream keepalive-pool budgets rather than relying on Pingora framework defaults. This is a regression bound for the minimal local generic path, not evidence that the pg-erd multi-route path or an Internet, TLS, multi-hop, consumer production path satisfies a 20 ms p95 SLO. A dedicated pg-erd routed load contract remains required before adopting the 20 ms objective for that migration.
 
 Every behavioral migration should begin with characterization against the old owned edge behavior, then add equivalent production-path evidence for Pingora. Static-serving consumers must cover route precedence, SPA fallback, MIME, ETag/cache, Range/HEAD/304/416, redirects/security headers and compression as applicable. Proxy consumers must cover Host/SNI/TLS, forwarding trust, WebSocket/upgrade, limits, timeout/retry behavior, streaming/uploads, saturation/backpressure, errors, health/readiness and drain. Characterized response policies must additionally be exercised through the compiled proxy path before any parity or canary claim.
 
-Release-quality gaps remain: no property/fuzz tests yet; no downstream TLS listener contract; no HTTP/2 or HTTP/3 parity evidence; no tracing evidence; no immutable published registry digest/provenance and rehearsed rollback; and no benchmark against replaced Nginx/Traefik traffic. For the pg-erd candidate specifically, the dedicated source now covers streamed body overflow, in-flight saturation/recovery, refused-origin recovery, connected silent-origin read timeout, routed graceful drain, and OCI process/Prometheus-listener identity, but TCP reset, post-commit partial-response/streaming failure, slow-drip/whole-response lifetime, representative routed origin-capacity load, terminal current-head traffic/OCI/supply-chain execution, shadow/canary and rollback still lack evidence. OCI non-root/read-only-root source acceptance covers both admitted process images, while owned-production 100% line/region coverage, public rustdoc, SBOM/image vulnerability, local-CA upstream TLS, explicit backpressure/recovery, and the minimal k6 loopback path remain exact-head gates. None can be transferred to a changed or consumer-specific head. Production performance claims remain forbidden until representative deployment measurements exist.
+Release-quality gaps remain: no property/fuzz tests yet; no downstream TLS listener contract; no HTTP/2 or HTTP/3 parity evidence; no tracing evidence; no immutable published registry digest/provenance and rehearsed rollback; and no benchmark against replaced Nginx/Traefik traffic. For the pg-erd candidate specifically, source now covers streamed body overflow, in-flight saturation/recovery, refused-origin recovery, connected silent-origin read timeout, orderly post-header truncation, routed graceful drain, and OCI process/Prometheus-listener identity. Explicit TCP reset, broader streaming/upgraded failure, slow-drip/whole-response lifetime, representative routed origin-capacity load, shadow/canary and rollback remain open. Current parent #20 and this changed #21 head must each independently pass their exact-head formatting/compile/test/Clippy/rustdoc/100%-coverage/load/OCI/Supply Chain gates before their respective contracts are credited. OCI non-root/read-only-root source acceptance covers both admitted process images, while public rustdoc, SBOM/image vulnerability, local-CA upstream TLS, explicit backpressure/recovery, and the minimal k6 loopback path remain exact-head gates. None can be transferred to a changed or consumer-specific head. Production performance claims remain forbidden until representative deployment measurements exist.

TRD.md

@@ -30,6 +30,8 @@ Non-health requests acquire the process `max_in_flight_requests` budget before u
 
 Generic v1 makes one prevalidated upstream peer available per request. The pg-erd migration adapter selects only peers already bound by `MigrationDeliveryPlan`; neither path performs request-controlled service discovery. Domain retries, failover, and idempotency policy are not invented by this runtime.
 
+Failure handling is phase-aware. Before an upstream response header is committed downstream, transport failure may still be represented by the gateway's fail-closed error response under the one-attempt policy. After a valid response header has been committed, a later upstream framing/body failure cannot be rewritten into a second HTTP status or silently failed over: the incomplete downstream response terminates, low-cardinality error telemetry records the failed request, process readiness remains available, and independent routes must remain usable. This is an edge transport invariant, not product retry authority.
+
 ## Health and observability
 
 `GET /livez` and `/readyz` return HTTP 200 with an empty, non-cacheable response through the process-local Pingora health boundary. Readiness proves validated configuration plus an active serving path, not product dependency health. In the pg-erd migration profile, consumer `/healthz` remains ordinary routed application traffic and is not confused with process liveness/readiness.