Skip to content

Commit 94195e7

Browse files
committed
Merge TASK-191: Repair independent HTTP/3 drain observation
2 parents ed3b144 + 16c0f32 commit 94195e7

11 files changed

Lines changed: 444 additions & 14 deletions

File tree

‎docs/task-174-http3-semantic-matrix.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,41 @@ output. Evidence directories are private; TLS secrets have mode 0600.
7979
`packets.pcap` contains real application-boundary datagrams with synthesized
8080
IPv4/UDP envelopes, rather than an interface capture.
8181

82+
## TASK-191 drain scheduling clarification
83+
84+
A successful drain requires the accepted response FIN and required control/QPACK
85+
information to retire through actual QUIC ACK completions. A clean peer close is
86+
not proof of receipt of that FIN. If the peer closes first, the ticket reports an
87+
immutable incomplete outcome, even when its application close code is H3_NO_ERROR.
88+
The fixture's historical `drain deadline expired` diagnostic also covers this
89+
interrupted outcome; it does not establish that five wall-clock seconds elapsed.
90+
91+
The pinned quic-go H3 client automatically closes after GOAWAY when its last
92+
tracked request reaches EOF. Its application can consume that EOF before the
93+
native delayed ACK for the server FIN is sent. The original immediate-EOF schedule
94+
is therefore a legal negative case, retained in the protected owner regression.
95+
A naive ACK-before-read observer is also invalid: remote CONNECTION_CLOSE makes
96+
this stack discard unread response data.
97+
98+
The positive client schedule preopens one additional client-only stream (ID 8)
99+
before drain, sends no bytes or FIN on it, and retains it through the actual native
100+
FIN ACK and remote server closure. It is not server work or a drain dependency.
101+
The original accepted stream 0 and preopened rejected stream 4 remain unchanged.
102+
The native tracer observes exact packet ACK coverage; qlog validation independently
103+
requires FIN → covering ACK → authenticated server application close 256. It also
104+
rejects any wire frame or server event naming the local hold. After remote closure,
105+
the client cancels each half of that local stream once. The observation and closure
106+
share the original five-second drain budget; no sleeps or synthetic ACKs are used.
107+
This scheduling amendment was explicitly authorized for TASK-191.
108+
109+
Final fixture `traffic` records are teardown capture accounting. The live runner
110+
validates their bounded integer shape and uniqueness, then waits for the actual
111+
process exit. Nonzero fixture exit and its captured causal stderr remain the
112+
primary error. Accounting cannot acknowledge semantic work or replace a receipt;
113+
unknown lifecycle events, duplicate/malformed accounting and lifecycle output after
114+
accounting fail closed. Successful runs still require exactly one accounting record
115+
and exact packet/byte reconciliation against the unchanged capture.
116+
82117
## Provisioning and reproduction
83118

84119
Local tools: Clang 18 (C++20 in actual compiler output), Python 3.12, Go 1.26.9,

‎scripts/run-v3-http3-gates.py‎

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,37 @@ def flow_witness(events, stream, sibling):
296296
return {'exhausted_event':min(exhausted),'sibling_fin_event':min(healthy),'grant_event':min(grants),'response_fin_event':min(finished)}
297297

298298

299+
def drain_ack_witness(events, case):
300+
need(case.get('fin_acknowledged') is True and case.get('local_hold_stream') == 8,
301+
'missing native drain synchronization')
302+
need(case.get('accepted_body_hex') == '', 'accepted drain response payload mismatch')
303+
fins = set()
304+
acknowledged = None
305+
closed = False
306+
for event in events:
307+
name = event.get('name')
308+
data = event.get('data', {})
309+
for frame in data.get('frames', []):
310+
kind = frame.get('frame_type')
311+
need(frame.get('stream_id') != case['local_hold_stream'], 'local hold appeared on wire')
312+
if name == 'transport:packet_received':
313+
if kind == 'stream' and frame.get('stream_id') == case['accepted_stream'] and frame.get('fin'):
314+
fins.add(data['header']['packet_number'])
315+
if kind == 'connection_close':
316+
need(frame.get('error_space') == 'application' and frame.get('raw_error_code') == 256,
317+
'wrong native drain close')
318+
need(acknowledged is not None, 'server close preceded accepted FIN ACK')
319+
closed = True
320+
elif name == 'transport:packet_sent':
321+
need(kind != 'connection_close' or closed, 'client closed before server drain')
322+
if kind == 'ack' and not closed:
323+
for bounds in frame.get('acked_ranges', []):
324+
for number in fins:
325+
if bounds[0] <= number <= bounds[-1]: acknowledged = number
326+
need(closed and acknowledged is not None, 'missing accepted FIN ACK/server close')
327+
return {'fin_packet': acknowledged, 'local_hold_stream': case['local_hold_stream'], 'server_close_code': 256}
328+
329+
299330
def semantic_artifacts(events, receipt, qlogs, directory):
300331
cases={c['case']:c for c in receipt['cases']}
301332
connection=cases['concurrency']['connection']
@@ -319,13 +350,21 @@ def semantic_artifacts(events, receipt, qlogs, directory):
319350
need(not any(e.get('connection')==d['connection'] and e.get('stream')==d['late_stream'] and e.get('event') in ('get','entered') for e in events),'late drain request entered a handler')
320351
need(sum(e.get('event')=='drained' and e.get('connection')==d['connection'] for e in events)==1,'missing completed drain ticket')
321352
witnesses=[]
353+
drain_witnesses=[]
354+
if receipt['client']=='quic-go':
355+
need(not any(e.get('connection')==d['connection'] and e.get('stream')==d.get('local_hold_stream') for e in events), 'local hold admitted by server')
322356
for path in qlogs:
323357
need(path.stat().st_size<=16*1024*1024,'qlog diagnostic bound')
324358
if path.suffix=='.qlog':native=json.loads(path.read_text())['traces'][0]['events']
325359
else:native=[json.loads(line.lstrip('\x1e')) for line in path.read_text().splitlines() if line.lstrip('\x1e')]
326360
need(len(native)<=100000,'qlog event bound')
361+
if receipt['client']=='quic-go' and any(e.get('name')=='transport:connection_closed' and e.get('data',{}).get('owner')=='remote' and e.get('data',{}).get('application_code')==256 for e in native):
362+
drain_witnesses.append({'qlog':path.name, **drain_ack_witness(native,d)})
327363
if any(f.get('stream_id')==c['stream'] and e.get('name')=='transport:packet_received' for e in native for f in e.get('data',{}).get('frames',[])):
328364
witnesses.append({'qlog':path.name,**flow_witness(native,c['stream'],c['sibling_stream'])})
365+
if receipt['client']=='quic-go':
366+
need(len(drain_witnesses)==1,'missing or ambiguous native drain ACK witness')
367+
(directory/'drain-ack-witness.json').write_text(json.dumps(drain_witnesses[0],indent=2)+'\n')
329368
need(len(witnesses)==1,'missing or ambiguous native flow-control witness')
330369
(directory/'flow-witness.json').write_text(json.dumps(witnesses[0],indent=2)+'\n')
331370

@@ -378,13 +417,29 @@ def run_client(name, fixture_path, client_command, credentials, directory, matri
378417
command = client_command + ['--port',str(port),'--ca',str(credentials/'ca.pem'),'--wrong-ca',str(credentials/'wrong-ca.pem'),'--log-dir',str(directory)]
379418
client = Process(command,directory,'client',messages)
380419
deadline = time.monotonic()+15
381-
waiting = None; receipt = None; active = set(); connection_id = None; drain_deadline = None
420+
waiting = None; receipt = None; active = set(); connection_id = None; drain_deadline = None; accounting = None
382421
while receipt is None:
383422
process,line = next_message(messages,deadline)
423+
if line is None and process is fixture:
424+
fixture.process.wait(timeout=3)
425+
for thread in fixture.threads: thread.join(timeout=2)
426+
diagnostic = (directory/'fixture.stderr').read_text().strip()
427+
prefix = 'fixture exited nonzero' if fixture.process.returncode else 'fixture exited'
428+
raise GateFailure(prefix + ' before all receipts' + (': ' + diagnostic if diagnostic else ''))
384429
need(line is not None, 'process exited before all receipts')
385430
try: item=json.loads(line)
386431
except json.JSONDecodeError as error: raise GateFailure('malformed JSON process receipt') from error
387432
if process is fixture:
433+
if item.get('event') == 'traffic':
434+
need(accounting is None, 'duplicate fixture traffic accounting')
435+
keys = ('received_count', 'sent_count', 'received_bytes', 'sent_bytes')
436+
need(set(item) == {'event', 'port', *keys} and type(item.get('port')) is int
437+
and item['port'] == port
438+
and all(type(item.get(key)) is int and item[key] >= 0 for key in keys),
439+
'malformed fixture traffic accounting')
440+
accounting = item
441+
continue # Teardown accounting never acknowledges semantic work.
442+
need(accounting is None, 'fixture lifecycle event after final accounting')
388443
allowed = ('held','released','cancelled','get','post','health')
389444
if semantic:
390445
allowed += ('entered','read','write_started','write_completed','finished','snapshot','drain_started','drained')
@@ -394,6 +449,7 @@ def run_client(name, fixture_path, client_command, credentials, directory, matri
394449
else:
395450
need(process is client,'unknown process')
396451
if 'receipt' in item:
452+
need(accounting is None, 'fixture accounting before semantic receipt')
397453
receipt=item['receipt'];break
398454
command=item.get('command')
399455
need(waiting is None,'overlapping runner commands')
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# TASK-191 implementation evidence
2+
3+
Manual Groundwork implementation in `/home/etr/progs/libhttpserver/.worktrees/TASK-191`, branch `task/TASK-191`, from `v3` at `ed3b1441df190d3972c0944b0daa12aa3089f331`. The caller owns validation, finalization, merge and cleanup. Task tracking remains In Progress. No executor memory store was configured; reusable setup facts are retained here and in the journal.
4+
5+
## Verified cause and authorized scheduling amendment
6+
7+
Evidence root: `build/task191`. Invocation-unique journal: `.debug/h3-semantic-drain-1fn2v4xl/journal.md`. Immutable TASK-185 logs, manifests, process receipts, qlogs and captures were read and hashed in `retained-identities.json`. The retained capture shows accepted FIN packet 32, ACK only through 31, then authenticated client H3_NO_ERROR close 256. Fresh unchanged-source attempts 02–09 passed; bounded attempt 10 reproduced the fault: accepted FIN packet 33, ACK only through 32, then client close 256. Its bounded authenticated decoder slice is `reproduction-10/quic-go/causal-wire.tsv`. Original source copies and the original independently rebuilt client binary are retained with hashes.
8+
9+
The pinned quic-go v0.55.0 native H3 `Conn.clearStream` closes after GOAWAY when its final tracked request reaches EOF. Native application consumption can precede the delayed ACK for the server FIN. The server correctly interrupts its still-incomplete ticket upon authenticated peer closure. The fixture's historical `drain deadline expired` message describes its incomplete result; it does not establish five elapsed wall-clock seconds. Treating a clean close as an ACK would violate required receipt retirement and immutable incomplete outcomes. Forcing server PING timing would leave the race unresolved.
10+
11+
The first observation discriminator, `reproduction-green-01`, waited for native FIN ACK and the ticket before reading the response. It established successful server drain but failed the client response because this pinned stack discards unread buffered data at remote close. This failed attempt remains retained.
12+
13+
The caller explicitly authorized a client-only scheduling amendment after fresh reproduction. The positive schedule preopens stream 8 locally before drain and sends **no bytes or FIN** on it. Native H3 tracking then stays alive while the accepted response reaches EOF. A connection-specific tracer requires real outgoing ACK coverage of an actual accepted stream-0 FIN packet, followed by the original completed-ticket event and remote authenticated application close 256. The client cancels each half of the local hold once after closure. All observation/closure work shares the original five-second drain budget; there is no sleep, fabricated ACK, protocol-library change or server dependency on that local hold.
14+
15+
The original accepted stream 0, exact response status 200 and empty payload, preopened late stream 4 and rejection 267, ordered GOAWAY IDs 4611686018427387900 and 4, original deadlines, capture reconciliation, H1/H2 parity and all existing semantic predicates remain. `drain-ack-witness.json` identifies the actual qlog and FIN packet. Independent qlog validation rejects missing/gapped/out-of-order FIN ACKs, client closure before server closure, any wire frame naming stream 8, and any server observation naming that hold. The unchanged aioquic 1.3.0 client remains the independent control.
16+
17+
No production engine, recovery, transport, budget, cancellation or public API code changes. PRD-V3N-REQ-007/008/025/031/032/033 and architecture §§3.3–3.5 / DR-V3-001/006/008 continue to require one semantic exchange, connection-owned compression/recovery, serialized owner cancellation, reserved critical capacity and deadline-bound tickets. This private fixture work establishes no new public H3 listener capability.
18+
19+
## Deterministic contracts and causal RED/GREEN
20+
21+
`runner-red.log` contains five expected failing assertions on the original runner: final accounting masked the causal fixture failure, and unsolicited/duplicate/malformed accounting was not classified correctly. The repair treats accounting as a bounded teardown record, never as a semantic acknowledgement. It preserves stdout lifecycle ordering, rejects unknown lifecycle events, duplicate/malformed accounting and lifecycle output after accounting, and reports actual nonzero fixture exit with fully captured causal stderr. Cleanup nonzero status remains in `exits.json`. Successful artifact validation still requires exactly one final record and unchanged exact packet/byte reconciliation.
22+
23+
`go-red.log` fails at the missing native FIN-ACK observation assertion. `go-green.log` and `go-refactor-green.log` establish exact accepted-stream ownership, ACK gaps, retransmitted FIN coverage, nil/duplicate ACK stability. The final Go tests additionally cover bounded cancellation and exactly-once terminal notification, with `go-race.log` and `go-vet-final.log` retaining race-enabled tests and vet results. Refactoring isolates the connection-specific `drainACKWitness` from the existing flow observer.
24+
25+
`native-witness-red.log` fails the missing independent qlog predicate. Its GREEN verifies FIN → covering ACK → server close, rejects absent/reordered/gapped ACK evidence and a transmitted local hold. Final runner contract logs preserve semantic/full/WebSocket/QUIC checks; QUIC contract tests explicitly skip four endpoint-dependent cases when that unit invocation lacks `QUIC_INTEROP_ENDPOINT`. Actual endpoint/capture interoperability remains required separately by the aggregate gate.
26+
27+
The protected owner regression runs two explicit authenticated schedules over the same held request, GOAWAY output and late-stream rejection. FIN ACK followed by peer close completes with remaining 0; immediate peer close before that ACK stays incomplete with remaining 2. Both preserve exactly-once terminal claim, application-key disposal on silent draining, aliases until the protocol endpoint, final route detachment and immutable result across subsequent ticks/waits. It adds 60 checks to the existing owner suite: 34 cases / 393 checks. Existing drain tests retain unrelated completion rejection, partial/lost/reordered control output, QPACK retirement, multiple waiters, real deadline expiry, reset cancellation and pending-send lifetime. Existing owner tests retain protected cached close output, closing keys, pending UDP storage/debit and endpoint arbitration.
28+
29+
## Required local verification and receipts
30+
31+
Fresh normal and ASan/nonrecovering UBSan C++20 TLS-off/on lanes use Clang 18.1.3, `--disable-examples`, `-std=c++20 -pthread -O1 -g -fno-omit-frame-pointer`. TLS-on uses the retained genuine OpenSSL 3.5.9 prefix `/home/etr/progs/libhttpserver/build/tooling/provider`, its include/lib directories and `-lssl -lcrypto`. Sanitizers add `-fsanitize=address,undefined -fno-sanitize-recover=undefined -fsanitize-address-use-after-scope` and retain `ASAN_OPTIONS=detect_leaks=1:detect_stack_use_after_return=1:halt_on_error=1`, `UBSAN_OPTIONS=halt_on_error=1:print_stacktrace=1` outside ptrace. No sanitizer diagnostics are disabled or suppressed. An initial sandboxed configure hit the explicit LeakSanitizer ptrace failure; those logs were archived before unsandboxed execution. An initial owner test execution from the build root had test-root lookup errors; the corrected test-directory execution is the baseline evidence.
32+
33+
`run_lanes.py` records exact configure/library/scoped-test commands, environment, exits and durations. Library build precedes tests; TESTS/check_PROGRAMS/noinst_PROGRAMS are scoped together. The unchanged full-runner selections cover all H3/QUIC framing, stream, cancellation, drain, QPACK, lifecycle, scheduling, recovery, flow, pacing, identity, sizing and WebSocket suites; TLS-on adds handshake/provider/network-owner/interop suites and explicitly builds network/parity/QUIC fixtures.
34+
35+
| Lane | Native programs | Cases | Checks | Skips |
36+
|---|---:|---:|---:|---:|
37+
| normal-off | 48 | 431 | 12270 | 0 |
38+
| normal-on | 55 | 498 | 13561 | 0 |
39+
| asan-off | 48 | 431 | 12270 | 0 |
40+
| asan-on | 55 | 498 | 13561 | 0 |
41+
42+
Each off lane includes 46 littletest programs plus two standalone corpus/replay programs; each on lane adds seven littletest programs. The cases/checks columns count littletest assertions, with standalone corpus records validated independently by the full runner.
43+
44+
The authoritative completion evidence is the task-local lane `receipt.json` and `final-summary.json`; no missing or failing required gate can be replaced by this table. Normal/ASan independent semantic gates are retained at `semantic-normal-current` and `semantic-asan-final`. They require both original pinned stacks, executed H1/H2 parity, fresh exits/captures/qlogs and the original received GOAWAY/closure predicates. The complete nine-stage runner is retained at `full`, with its source/inventory/exclusion/tool/provider identities, authenticated QUIC interop, semantic matrix, RFC 9220 WebSocket matrix and actual fuzz-off/fuzz-on receipts. A successful local acceptance requires its `manifest.json` status `passed` and all nine stages; an aggregate stopping earlier is incomplete.
45+
46+
Tool identities: aioquic 1.3.0, quic-go v0.55.0, Go 1.25.1 from the retained tools, tshark 4.6.4 and the existing pinned upstream QUIC runner commit 740c05a10b61d65e8abd3ad38d60898004d335d9. Reusable caches remain at `/tmp/task185-tools/{gopath,gocache}`. Actual executable, provider, source, binary, library and receipt hashes are captured in aggregate/final task manifests. No new installed public-runtime witness is claimed because no public runtime path changed. No production concurrency change requires a new TSan lane; the client observer is race-tested separately. TASK-200 retains timer-teardown race ownership.
47+
48+
Changed C++ cpplint, Python compilation, Go vet/race tests, littletest result/skip contract and `git diff --check` are required local checks. Two retained runs of the unchanged external decoder's 50 ms absence contract hit process-start scheduling timeout under compiler load; unchanged rerun receipts record final outcome. This does not change the original witness deadline or substitute another gate. Nonlocal BSD/macOS/Windows/backend checks remain unexecuted and CI/v3-PR owned under AGENTS.md.
49+
50+
Archive `build/task191` and the unique `.debug` journal through the caller before cleanup. Preserve TASK-185 artifacts/status, TASK-200 ownership, unrelated FEATURE worktrees and the clean primary checkout. This executor performs no validation-loop, merge, cleanup, push, tag or publication.

0 commit comments

Comments
 (0)