diff --git a/CHANGELOG.md b/CHANGELOG.md index ba3ae95f..13cc91f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ All notable changes are tracked here. No release has been published yet. - Kept migration admin parsing side-effect free for custom TLS trust material: exact transport-authority and upstream-contract validation happens during parse, while peer/trust-bundle materialization occurs once during `build_proxy` before listener creation. This removes an avoidable validate-then-reload trust-file window. - Added a separate Ingress forwarding-policy boundary for the pg-erd migration. Request-controlled `Forwarded`, `X-Forwarded-*` and `X-Real-IP` values are removed, then the compatibility `X-Forwarded-For`, `X-Real-IP`, `X-Forwarded-Host`, `X-Forwarded-Port` and `X-Forwarded-Proto` fields are rebuilt from accepted transport metadata. The current characterized Traefik `web` entryPoint remains explicitly HTTP; TLS-derived scheme behavior is not claimed before a TLS listener contract exists. - Added a shared `observability` bounded context so both `GatewayProxy` and `MigrationGatewayProxy` use the same low-cardinality request/error/body/backpressure counters and coarse access-log shape instead of duplicating telemetry. The public observation vocabulary contains only response status, `ok`/`error`, and observed request-body bytes; paths, query strings, headers/cookies, credentials, customer payloads and product identifiers stay out of the shared telemetry contract. +- Added dedicated compiled pg-erd payload-free access-log acceptance. A routed request carries unique URI/query, Host, Authorization, Cookie and product-context sentinels; the backend must receive those exact fields so the fixture is non-vacuous, while shared gateway stderr must emit only the bounded completion vocabulary and none of the sentinels. Test oracles use exact case-insensitive HTTP field matching and exact Prometheus sample-line matching so `X-Forwarded-Host` cannot satisfy `Host` and counter value `10` cannot satisfy the expected value `1`. - Added mandatory positive `max_in_flight_requests` and `upstream_keepalive_pool_size` capacity budgets; Pingora's framework keepalive default is overridden from the validated edge contract. - Added process-local fail-fast backpressure: non-health requests above the in-flight budget receive HTTP 503, health remains observable, rejection telemetry increments, and capacity is released after request completion or failure. - Added dedicated compiled pg-erd traffic acceptance for streamed/chunked body overflow and routed in-flight saturation/recovery: the migration process must return 413 above the shared body budget, return 503 in less than one second above the in-flight budget, keep `/readyz` observable, expose the exact single-rejection Prometheus sample, and admit a later routed request after capacity is released. @@ -40,4 +41,4 @@ All notable changes are tracked here. No release has been published yet. - Added missing-public-rustdoc enforcement and documentation builds with warnings denied. - Added DDD, product, technical, security, threat, test, operability, configuration, migration-gap, and primary-source traceability documentation. -Release remains blocked on the exact Pingora supplier disposition, including unmaintained `derivative 2.2.0` / `RUSTSEC-2024-0388`, restoration of authoritative dependency-review evidence, terminal exact-current-head CI/supply-chain/security/review evidence, representative pg-erd TLS/protocol/failure/concurrency performance, immutable registry/package identity with release-bound SBOM/provenance/reproducibility, rollback rehearsal, and protected-branch integration. Current parent #21 is `4d3cf712b89a0b607db1e5f80db2d60ed29f6e1c` after ordinary ancestry/single-writer repair and must independently reacquire its exact-head evidence; the changed #22 routed-load candidate must independently acquire unchanged exact-head CI/Supply Chain and review evidence before routed p95 credit is granted. Historical `51f1242...` and `3db4fe0...` GREEN receipts do not transfer. No consumer migration, canary, cutover, rollback, or legacy removal is claimed before those release and traffic-contract gates are satisfied. +Release remains blocked on the exact Pingora supplier disposition, including unmaintained `derivative 2.2.0` / `RUSTSEC-2024-0388`, restoration of authoritative dependency-review evidence, terminal exact-current-head CI/supply-chain/security/review evidence, representative pg-erd TLS/protocol/failure/concurrency performance, immutable registry/package identity with release-bound SBOM/provenance/reproducibility, rollback rehearsal, and protected-branch integration. Current parent #22 is `d74207828f3889393f8099948101e033721f7d7c` after ordinary ancestry/single-writer repair and must independently reacquire exact-head evidence; the current #23 payload-free observability child must independently acquire exact-head hosted and review evidence as well. Historical `3db4fe0...` and `5b58bcf...` GREEN receipts do not transfer. No consumer migration, canary, cutover, rollback, or legacy removal is claimed before those release and traffic-contract gates are satisfied. diff --git a/TEST_STRATEGY.md b/TEST_STRATEGY.md index 146b4e8a..a0c0950b 100644 --- a/TEST_STRATEGY.md +++ b/TEST_STRATEGY.md @@ -18,11 +18,13 @@ The generic `load-contract` is deliberately separate from functional production- Runtime and failure traffic are split by causal phase. `tests/pg_erd_runtime_isolation_traffic.rs` covers streamed body overflow plus in-flight saturation/recovery and exact rejection telemetry. `tests/pg_erd_upstream_failure_traffic.rs` creates deterministic `ECONNREFUSED`, requires bounded 502 recovery, exact request-error telemetry and later frontend success. `tests/pg_erd_read_stall_traffic.rs` keeps an accepted origin connection silent and open so read inactivity cannot be confused with origin closure. `tests/pg_erd_graceful_shutdown.rs` holds a routed request in flight, fixes one SIGTERM-relative external termination deadline, releases the response during the shared grace period, requires downstream 200 and clean process exit before that same deadline. -`tests/pg_erd_partial_response_traffic.rs` covers the post-header phase. The backend sends HTTP 200 with `Content-Length: 20` and only the seven-byte `partial` prefix, then waits until the downstream has observed the complete header block plus that exact prefix before closing. Acceptance requires the committed status/framing to remain visible without a fabricated second status or silent failover, `/readyz` to stay 200, exact `cwl_pingora_gateway_request_errors_total 1`, and an independent frontend recovery request. The framing oracle parses header lines, matches `Content-Length` field identity case-insensitively, trims field-value whitespace, requires exactly one value equal to `20`, and rejects lookalike or duplicate/conflicting fields. Current #21 exact `4d3cf712b89a0b607db1e5f80db2d60ed29f6e1c` is an ordinary ancestry/single-writer repair and must independently reacquire its exact-head CI/Supply Chain/review evidence; historical `51f1242...` GREEN does not transfer. +`tests/pg_erd_partial_response_traffic.rs` covers the post-header phase. The backend sends HTTP 200 with `Content-Length: 20` and only the seven-byte `partial` prefix, then waits until the downstream has observed the complete header block plus that exact prefix before closing. Acceptance requires the committed status/framing to remain visible without a fabricated second status or silent failover, `/readyz` to stay 200, exact `cwl_pingora_gateway_request_errors_total 1`, and an independent frontend recovery request. The framing oracle parses header lines, matches `Content-Length` field identity case-insensitively, trims field-value whitespace, requires exactly one value equal to `20`, and rejects lookalike or duplicate/conflicting fields. Current #21 exact `4d3cf712b89a0b607db1e5f80db2d60ed29f6e1c` must independently reacquire exact-head CI/Supply Chain/review evidence after its ordinary repair; no historical GREEN transfers. + +`tests/pg_erd_payload_free_observability.rs` covers the shared gateway observability boundary through the compiled migration process. A real routed request carries unique URI/query, Host, Authorization, Cookie and product-context sentinels; the backend must receive those values so the test cannot pass vacuously, while the `cwl_pingora_gateway::observability` target may emit only the bounded completion vocabulary and none of the sentinels. Traffic and metrics listener reservations are held concurrently until process start, origin header reads are bounded by five seconds and 64 KiB, HTTP field identity is matched case-insensitively by exact field name with OWS trimming, and the Prometheus oracle requires a complete exact sample line so a counter value such as `10` cannot satisfy an expected value of `1`. This proves only the shared gateway observability contract; it does not claim authority over product-owned logging, tracing, identity or third-party logger configuration. ## Routed performance acceptance -#22 adds the first dedicated routed pg-erd concurrency/latency gate on the current #21 ancestry. `tests/load/load_origin.rs` is the only measured origin implementation: bounded std-only Rust, finite worker/queue capacity, 64 KiB header bound, deterministic Content-Length framing and direct parser/framing tests. The CI load lane compiles both admitted gateway binaries, runs `rustfmt` and `rustc -D warnings --test` on the origin, then builds the optimized fixture used by measured traffic. +PR `#22` adds the first dedicated routed pg-erd concurrency/latency gate on the current #21 ancestry. `tests/load/load_origin.rs` is the only measured origin implementation: bounded std-only Rust, finite worker/queue capacity, 64 KiB header bound, deterministic Content-Length framing and direct parser/framing tests. The CI load lane compiles both admitted gateway binaries, runs `rustfmt` and `rustc -D warnings --test` on the origin, then builds the optimized fixture used by measured traffic. `tests/load/pg_erd_gateway_smoke.js` runs four VUs for 400 total iterations. It alternates `/api/load-contract` and `/load-contract`, tags each request `backend` or `frontend`, requires exact 200/body identity and zero HTTP failures, gates aggregate plus each route independently at p95 `<20 ms`, and requires at least 198 measured requests per route. `tests/pg_erd_routed_latency_contract.rs` freezes the route/path/body/tag binding and per-route thresholds/sample floors. `tests/rust_load_origin_workflow_contract.rs` prevents regression to interpreted measured-origin execution by inspecting only the `load-contract` job and requiring the bounded Rust build/test/start commands with no Python invocation. @@ -38,4 +40,4 @@ Supply-chain evidence must remain exact-source-bound and include dependency audi ## Remaining gaps -Open acceptance still includes explicit TCP reset, broader streaming and WebSocket/Upgrade failure behavior, slow-drip/whole-response lifetime, downstream TLS/H2, H2→H1 Cookie handling, Extended CONNECT, explicit H3/QUIC disposition, tracing, property/fuzz testing, representative routed TLS/origin-capacity load, shadow/canary and rollback. Current #21 and #22 exact heads must independently reacquire their hosted evidence after the ordinary restack; no historical receipt transfers. Nginx/OpenResty/legacy removal is permitted only after parity, immutable release, canary/cutover and rollback evidence are all current on protected ancestry. +Open acceptance still includes explicit TCP reset, broader streaming and WebSocket/Upgrade failure behavior, slow-drip/whole-response lifetime, downstream TLS/H2, H2→H1 Cookie handling, Extended CONNECT, explicit H3/QUIC disposition, tracing, property/fuzz testing, representative routed TLS/origin-capacity load, shadow/canary and rollback. Current #22 and #23 exact heads must independently reacquire hosted/review evidence after ordinary restack; no historical receipt transfers. Nginx/OpenResty/legacy removal is permitted only after parity, immutable release, canary/cutover and rollback evidence are all current on protected ancestry. diff --git a/tests/pg_erd_payload_free_observability.rs b/tests/pg_erd_payload_free_observability.rs new file mode 100644 index 00000000..2e31a4b4 --- /dev/null +++ b/tests/pg_erd_payload_free_observability.rs @@ -0,0 +1,346 @@ +//! Real-listener payload-free logging acceptance for the dedicated pg-erd migration binary. +//! +//! The shared observability bounded context promises that request paths, query strings, headers, +//! cookies, credentials, customer payloads, and product identifiers never enter its access-log +//! vocabulary. This contract proves that boundary through the compiled migration process while +//! sensitive request material is actually present on the proxied request path. + +use std::fs; +use std::io::{Read, Write}; +use std::net::{SocketAddr, TcpListener, TcpStream}; +use std::process::{Child, Command, Stdio}; +use std::thread; +use std::time::{Duration, Instant}; + +use tempfile::NamedTempFile; + +const MAX_ORIGIN_REQUEST_HEADER_BYTES: usize = 64 * 1024; + +struct GatewayProcess { + child: Option, + stderr: NamedTempFile, +} + +impl GatewayProcess { + fn wait_until_stderr_contains(&mut self, needle: &str) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + let captured = fs::read_to_string(self.stderr.path()) + .expect("gateway stderr capture should remain readable"); + if captured.contains(needle) { + return; + } + if let Some(status) = self + .child + .as_mut() + .expect("gateway child should still be owned") + .try_wait() + .expect("gateway process state should be readable") + { + panic!( + "gateway exited before expected log {needle:?}: {status}; stderr={captured:?}" + ); + } + assert!( + Instant::now() < deadline, + "gateway did not emit expected log {needle:?} within 10s; stderr={captured:?}" + ); + thread::sleep(Duration::from_millis(10)); + } + } + + fn capture_stderr(mut self) -> String { + let mut child = self + .child + .take() + .expect("gateway child should still be owned"); + child + .kill() + .expect("gateway should be terminable after traffic"); + child + .wait() + .expect("gateway should terminate after traffic capture"); + fs::read_to_string(self.stderr.path()).expect("gateway log output should be UTF-8") + } +} + +impl Drop for GatewayProcess { + fn drop(&mut self) { + if let Some(child) = self.child.as_mut() { + let _ = child.kill(); + let _ = child.wait(); + } + } +} + +/// Reserves both process listeners at once so sequential bind-and-drop cannot +/// reuse the first ephemeral port and manufacture an Admin Config collision. +fn reserve_gateway_addresses() -> (TcpListener, TcpListener, SocketAddr, SocketAddr) { + let gateway = TcpListener::bind("127.0.0.1:0").expect("gateway port should be reservable"); + let metrics = TcpListener::bind("127.0.0.1:0").expect("metrics port should be reservable"); + let gateway_address = gateway + .local_addr() + .expect("gateway reservation should expose an address"); + let metrics_address = metrics + .local_addr() + .expect("metrics reservation should expose an address"); + assert_ne!( + gateway_address, metrics_address, + "traffic and metrics reservations must remain distinct" + ); + (gateway, metrics, gateway_address, metrics_address) +} + +fn write_config( + listener: SocketAddr, + metrics_listener: SocketAddr, + backend: SocketAddr, + frontend: SocketAddr, +) -> NamedTempFile { + let mut file = NamedTempFile::new().expect("temporary config should be writable"); + writeln!( + file, + "version: 1\nlistener: {listener}\nmetrics_listener: {metrics_listener}\nmax_request_body_bytes: 8\nmax_in_flight_requests: 8\nupstream_keepalive_pool_size: 4\nupstreams:\n - name: backend\n address: {backend}\n tls: false\n timeouts:\n connection_ms: 200\n total_connection_ms: 400\n read_ms: 1000\n write_ms: 1000\n idle_ms: 5000\n - name: frontend\n address: {frontend}\n tls: false\n timeouts:\n connection_ms: 200\n total_connection_ms: 400\n read_ms: 1000\n write_ms: 1000\n idle_ms: 5000" + ) + .expect("migration config should be written"); + file +} + +fn wait_until_listening(address: SocketAddr, process: &mut Child) { + let deadline = Instant::now() + Duration::from_secs(10); + loop { + if let Some(status) = process + .try_wait() + .expect("gateway process state should be readable") + { + panic!("gateway exited before accepting traffic: {status}"); + } + if TcpStream::connect_timeout(&address, Duration::from_millis(100)).is_ok() { + return; + } + assert!( + Instant::now() < deadline, + "gateway did not start within 10s" + ); + thread::sleep(Duration::from_millis(25)); + } +} + +fn start_gateway( + config: &NamedTempFile, + gateway_address: SocketAddr, + metrics_address: SocketAddr, +) -> GatewayProcess { + let stderr = NamedTempFile::new().expect("gateway stderr capture should be writable"); + let stderr_writer = stderr + .reopen() + .expect("gateway stderr capture should be reopenable for the child"); + let mut child = Command::new(env!("CARGO_BIN_EXE_cwl-pingora-pg-erd-migration")) + .args(["--config", config.path().to_str().expect("UTF-8 temp path")]) + .env("RUST_LOG", "cwl_pingora_gateway::observability=info") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::from(stderr_writer)) + .spawn() + .expect("compiled pg-erd migration binary should start"); + wait_until_listening(gateway_address, &mut child); + wait_until_listening(metrics_address, &mut child); + GatewayProcess { + child: Some(child), + stderr, + } +} + +fn raw_request(address: SocketAddr, request: &[u8]) -> String { + let mut downstream = TcpStream::connect(address).expect("gateway should accept traffic"); + downstream + .set_read_timeout(Some(Duration::from_secs(5))) + .expect("downstream timeout should be configurable"); + downstream + .write_all(request) + .expect("downstream request should be writable"); + let mut response = String::new(); + downstream + .read_to_string(&mut response) + .expect("gateway response should be readable"); + response +} + +/// Reads one origin-side request header block under a finite timeout and byte +/// budget so a broken forwarding path fails deterministically instead of hanging CI. +fn read_request_headers(stream: &mut TcpStream) -> String { + stream + .set_read_timeout(Some(Duration::from_secs(5))) + .expect("origin request timeout should be configurable"); + let mut bytes = Vec::new(); + let mut buffer = [0_u8; 1024]; + loop { + let read = stream + .read(&mut buffer) + .expect("origin request should be readable before the fixture deadline"); + assert!( + read > 0, + "gateway closed origin request before headers completed" + ); + bytes.extend_from_slice(&buffer[..read]); + assert!( + bytes.len() <= MAX_ORIGIN_REQUEST_HEADER_BYTES, + "gateway origin request headers exceeded the fixture bound" + ); + if bytes.windows(4).any(|window| window == b"\r\n\r\n") { + return String::from_utf8_lossy(&bytes).into_owned(); + } + } +} + +/// Selects exact HTTP field names case-insensitively while rejecting lookalike +/// fields such as `X-Forwarded-Host` that contain `Host` only as a suffix. +fn header_values<'a>(request: &'a str, name: &str) -> Vec<&'a str> { + request + .split("\r\n") + .skip(1) + .take_while(|line| !line.is_empty()) + .filter_map(|line| line.split_once(':')) + .filter_map(|(field_name, value)| { + field_name + .eq_ignore_ascii_case(name) + .then_some(value.trim()) + }) + .collect() +} + +/// Requires a complete Prometheus sample line so a value such as `10` cannot +/// satisfy an oracle that expects the exact counter value `1`. +fn contains_exact_metric_sample(metrics: &str, sample: &str) -> bool { + metrics + .lines() + .any(|line| line.trim_end_matches('\r') == sample) +} + +#[test] +fn exact_header_matching_rejects_forwarded_host_lookalikes() { + let request = "GET / HTTP/1.1\r\nX-Forwarded-Host: tenant-secret.example:8080\r\nhOsT: expected.example\r\n\r\n"; + assert_eq!(header_values(request, "Host"), vec!["expected.example"]); +} + +#[test] +fn exact_metric_sample_rejects_numeric_prefix_lookalikes() { + let metrics = "# TYPE cwl_pingora_gateway_requests_total counter\ncwl_pingora_gateway_requests_total 10\n"; + assert!(!contains_exact_metric_sample( + metrics, + "cwl_pingora_gateway_requests_total 1" + )); +} + +#[test] +fn compiled_pg_erd_shared_access_log_excludes_request_sensitive_material() { + let backend = TcpListener::bind("127.0.0.1:0").expect("backend fixture should bind"); + let backend_address = backend.local_addr().expect("backend address should exist"); + let backend_origin = thread::spawn(move || { + let (mut stream, _) = backend + .accept() + .expect("routed request should reach the characterized backend authority"); + let request = read_request_headers(&mut stream); + let request_line = request + .split("\r\n") + .next() + .expect("origin request should contain a request line"); + assert_eq!( + request_line, "GET /api/log-contract?customer=query-secret HTTP/1.1", + "request target and query sentinel must be preserved exactly" + ); + assert_eq!( + header_values(&request, "Host"), + vec!["tenant-secret.example:8080"] + ); + assert_eq!( + header_values(&request, "Authorization"), + vec!["Bearer authorization-secret"] + ); + assert_eq!( + header_values(&request, "Cookie"), + vec!["session=cookie-secret"] + ); + assert_eq!( + header_values(&request, "X-Product-Context"), + vec!["product-secret"] + ); + stream + .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok") + .expect("backend response should be writable"); + }); + + let frontend = TcpListener::bind("127.0.0.1:0").expect("frontend fixture should bind"); + let frontend_address = frontend + .local_addr() + .expect("frontend address should exist"); + + let (gateway_reservation, metrics_reservation, gateway_address, metrics_address) = + reserve_gateway_addresses(); + let config = write_config( + gateway_address, + metrics_address, + backend_address, + frontend_address, + ); + drop(gateway_reservation); + drop(metrics_reservation); + let mut process = start_gateway(&config, gateway_address, metrics_address); + + let response = raw_request( + gateway_address, + b"GET /api/log-contract?customer=query-secret HTTP/1.1\r\nHost: tenant-secret.example:8080\r\nAuthorization: Bearer authorization-secret\r\nCookie: session=cookie-secret\r\nX-Product-Context: product-secret\r\nConnection: close\r\n\r\n", + ); + assert!( + response.starts_with("HTTP/1.1 200"), + "sensitive-material fixture request should proxy successfully: {response:?}" + ); + backend_origin + .join() + .expect("backend sensitive-material fixture should complete"); + + let metrics = raw_request( + metrics_address, + b"GET /metrics HTTP/1.1\r\nHost: metrics\r\nConnection: close\r\n\r\n", + ); + assert!( + contains_exact_metric_sample(&metrics, "cwl_pingora_gateway_requests_total 1"), + "metrics scrape should prove exactly one proxied request reached shared completion recording: {metrics:?}" + ); + process + .wait_until_stderr_contains("gateway_request status=200 outcome=ok request_body_bytes=0"); + + let stderr = process.capture_stderr(); + let request_logs: Vec<_> = stderr + .lines() + .filter(|line| line.contains("gateway_request")) + .collect(); + assert_eq!( + request_logs.len(), + 1, + "the shared observability target should emit one completion record: {stderr:?}" + ); + let access_log = request_logs[0]; + let completion = access_log + .split_once("gateway_request ") + .expect("shared access log should contain the completion message") + .1; + assert_eq!( + completion, "status=200 outcome=ok request_body_bytes=0", + "shared access logging should contain only bounded transport facts: {access_log:?}" + ); + + for forbidden in [ + "/api/log-contract", + "query-secret", + "tenant-secret.example", + "authorization-secret", + "cookie-secret", + "product-secret", + ] { + assert!( + !stderr.contains(forbidden), + "shared observability target leaked request-sensitive material {forbidden:?}: {stderr:?}" + ); + } +}