Skip to content

Fix response decoding, framing, and OPTIONS replay #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mandatoryprogrammer merged 3 commits into from
Oct 5, 2026
Merged

Fix response decoding, framing, and OPTIONS replay #52

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
752e22c
Fix captured response bytes and downstream framing
mandatoryprogrammer Oct 4, 2026
bd5ae03
Finalize response framing after hooks
mandatoryprogrammer Oct 5, 2026
b8c4deb
Handle repeated response metadata in after-request hook
mandatoryprogrammer Oct 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions _readme/internal-notes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,37 @@ Rebuild and recreate the Chrome service to install the policy: `docker compose u
Explicit HTTPS requests, certificate validation, HSTS, and server-issued redirects remain enabled. An HTTP URL covered by HSTS can still return Chrome's 307 upgrade redirect. These settings do not guarantee HTTP for HSTS hosts or measure TLS/HTTP fingerprint parity.

Policy references: [HttpsUpgradesEnabled](https://chromeenterprise.google/policies/#HttpsUpgradesEnabled) and [HttpsOnlyMode](https://chromeenterprise.google/policies/#HttpsOnlyMode).

## Captured response bytes and framing

Both CDP capture paths decode base64 directly into a `Buffer`, with UTF-8 for protocol text. Converting base64 through `atob()` and then UTF-8 re-encodes bytes above `0x7f`; it does not preserve binary or UTF-8 response bytes. Chrome also decompresses captured bodies, so upstream compression and transfer framing cannot describe the returned buffer.

The shared header formatter removes content encoding, every casing of content length, hop-by-hop fields (including fields named by `Connection`), and trailer declarations. Ordinary responses receive one length for the captured bytes; repeated headers remain separate values, including `Set-Cookie`. HEAD and 304 omit the optional representation length because the decoded representation size is unknown; 1xx and 204 omit the forbidden length, and 205 returns an empty body with length zero. Redirects retain their status and `Location` but return an empty body because CDP does not expose redirect bodies.

The proxy finalizes framing again after `AFTER_REQUEST_HOOK` has completed. Hooks still receive the captured body's initial length, but the send path discards all casings of `Content-Length`, `Transfer-Encoding`, and `Trailer`, converts the final body to bytes, and sets one length for those exact bytes where permitted. The same final buffer is sent to mockttp. Bodyless rules are reapplied to the final status, so a hook cannot accidentally send a body with HEAD, 204, 205, or 304. The handler's generated 502 fallback uses the same framing path, and TRACE reflects the finalized response.

The downstream HTTP/1 connection policy is derived from the client's HTTP version and `Connection` options, independently of the browser's upstream connection. This explicit header is needed because mockttp removes Node's default connection header. HTTP/2 receives no connection-specific headers. All final responses have a byte length or bodyless semantics; HTTP/1 persistence follows the client's version and connection options.

The existing synthetic OPTIONS preflight policy only applies when both `Origin` and `Access-Control-Request-Method` identify a real preflight. `Access-Control-Request-Headers` is optional; when absent, no `Access-Control-Allow-Headers` field is emitted. Ordinary same-origin OPTIONS requests, and requests whose preflight is cached, continue to the server. Chrome response-stage network errors fail promptly instead of being treated as new requests until timeout.

### Manual verification, 2026-10-04

Chrome 152.0.7977.64 and Node 18.20.4 were used with existing repository dependencies. The browser had a disposable profile and the repository's managed policies mounted into an isolated process. All upstream targets were loopback HTTP fixtures; the actual proxy used a temporary CA and generated credentials. No user Chrome profile or repository CA material was used.

The review-only harness was run with `CHECK_TIMEOUTS=1 node /tmp/thermoptic-response-fix-20261004/verify.mjs`; its source, `run.log`, and `results.json` are local review artifacts in that directory, not repository dependencies or a committed test suite. It sends ordinary requests through `get_http_proxy()` and `requestengine.process_request()` with an HTTP/1 keep-alive client.

All 38 end-to-end checks passed; 32 requests reused an existing downstream connection. Coverage includes navigation and fetch for ASCII/Unicode, gzip/Brotli/deflate, mixed header casing, chunked responses with trailers and connection-specific fields, empty bodies, redirects, 204/205/304, and partial responses; binary fetch, compressed HEAD, same-origin OPTIONS, cross-origin OPTIONS with and without custom headers, caller cookie forwarding, network errors, timeout expiry, and a successful request after timeouts. Body bytes and wire header lengths are checked, along with header uniqueness, removal of conflicting transfer framing, and connection reuse. Repeated `Set-Cookie` preservation is checked at the formatter boundary: this Chrome build omits those headers from the observed Fetch response events, so the live cookie check covers caller cookies reaching the upstream server.

Syntax checks: `node --check cdp.js`, `node --check utils.js`, and `node --check proxy.js`; whitespace check: `git diff --check`. These changes do not alter Chrome launch parameters, navigation mechanisms, TLS settings, or generated ordinary request headers. Fingerprint parity was not measured by the loopback checks.

### Post-hook verification, 2026-10-05

With the same installed Node, Chrome, and repository dependencies, `node /tmp/thermoptic-hook-framing-20261005/verify.mjs` passed 36 local wire checks over HTTP/1.1 and TLS HTTP/2. This temporary harness invokes an asynchronous hook through `utils.run_hook_file()` inside the actual proxy callback. It covers growing/shrinking UTF-8 bodies, empty/null bodies, binary buffers, typed-array slices and ArrayBuffers, object/Map/array header formats, replacement transfer/trailer headers, HEAD and 204/205/304, a hook restoring a body after 204, missing-status fallback, and the generated 502 response. Assertions check exact returned bytes, one correct length or its required omission, removal of transfer/trailer headers, repeated cookies, and HTTP/1 connection reuse. Sources, `run.log`, and `results.json` remain local review artifacts in that directory.

`node /tmp/thermoptic-response-fix-20261004/verify.mjs` also passed all 36 normal Chrome navigation/fetch checks again, including network-error recovery, with two downstream sockets. This rerun excludes the two 30-second timeout cases already verified above. Its log is `post-hook-run.log`; the earlier timeout run is preserved in `run-with-timeouts.log` and `results-with-timeouts.json`. Syntax and whitespace checks passed. No dependencies were installed and no test suite was added to the repository.

### Repeated-header hook compatibility, 2026-10-05

The bundled after-request hook reads metadata from both strings and arrays of strings. Its case-insensitive header reader creates a comma-separated string view for metadata checks and ignores non-string entries; it leaves the response header object and repeated `Set-Cookie` values untouched. This prevents repeated `Server`, `Content-Type`, or `CF-Mitigated` fields from causing `.toLowerCase()` failures after capture starts preserving duplicates.

`node /tmp/thermoptic-hook-framing-20261005/duplicate-headers.mjs` passed eight focused checks using the real formatter and bundled hook, covering ordinary and repeated metadata, mixed header casing, non-HTML and empty values, and unchanged cookies. The fixtures have no target URL and use a CDP sentinel to ensure no browser operations occur. `node /tmp/thermoptic-hook-framing-20261005/verify-bundled-hook.mjs` then passed all 36 HTTP/1.1 and TLS HTTP/2 wire checks with repeated `Server` fields passed through the bundled hook before the existing body-changing fixture hook. Results are in `duplicate-results.json` and `bundled-hook-wire-results.json`; the wire log is `bundled-hook-wire.log`. Hook syntax and whitespace checks passed; no dependencies or committed tests were added.
143 changes: 74 additions & 69 deletions cdp.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1239,6 +1239,47 @@ export async function set_each_file_input_via_input_api(cdp, file_paths) {
}
}

async function read_captured_response_body(tab, request_id, method, status_code) {
if (utils.response_has_no_body(method, status_code) || REDIRECT_STATUS_CODES.includes(status_code)) {
// Fetch.getResponseBody does not expose redirect bodies. Other empty
// responses need no body read (which can fail for HEAD/204/304).
return Buffer.alloc(0);
}
const response = await tab.Fetch.getResponseBody({ requestId: request_id });
if (response.base64Encoded) {
return Buffer.from(response.body, 'base64');
}
return Buffer.from(response.body, 'utf8');
}

function cors_preflight_response_headers(request) {
if (request.method.toUpperCase() !== 'OPTIONS') {
return null;
}
const headers = fetchgen.convert_headers_map_to_array(request.headers);
const origin = fetchgen.get_header_value_ignore_case('Origin', headers);
const method = fetchgen.get_header_value_ignore_case('Access-Control-Request-Method', headers);
if (!origin || !method) {
// Ordinary OPTIONS requests (including when preflight is cached) must
// reach the upstream server instead of being swallowed by the mock.
return null;
}
const requested_headers = fetchgen.get_header_value_ignore_case('Access-Control-Request-Headers', headers);
const response_headers = [
{ name: 'Access-Control-Allow-Origin', value: origin },
{ name: 'Access-Control-Allow-Methods', value: method },
{ name: 'Access-Control-Allow-Credentials', value: 'true' },
// Keep synthetic preflight permission scoped to this request chain.
{ name: 'Access-Control-Max-Age', value: '0' },
{ name: 'Content-Type', value: 'text/plain; charset=utf-8' },
{ name: 'Content-Length', value: '0' }
];
if (requested_headers) {
response_headers.push({ name: 'Access-Control-Allow-Headers', value: requested_headers });
}
return response_headers;
}

/*
Serves the HTML on the URL specified and captures and returns the raw HTTP response.
*/
Expand Down Expand Up @@ -1294,7 +1335,13 @@ async function intercept_navigation_and_capture(tab, url_to_host_on, html_to_ser
responseHeaders
}) => {
try {
const { url } = request;
if (settled) {
return;
}
if (responseErrorReason) {
reject(new Error(responseErrorReason));
return;
}
const is_request = (responseStatusCode === undefined);

cdp_logger.debug('Intercepted request during navigation capture.', {
Expand All @@ -1304,41 +1351,23 @@ async function intercept_navigation_and_capture(tab, url_to_host_on, html_to_ser
frame_id: frameId
});

// Is this a preflight OPTIONS request?
// If so, we'll just mock a response that allows the next request to continue
// TODO: Make this configurable as some users may *want* the requests to occur
// as normal for anti-fingerprinting reasons. It's a tricky tradeoff that we
// should allow the user to configure.
if (is_request && request.method && request.method.toLowerCase() === 'options' && !is_cors_preflight_handled) {
cdp_logger.debug('Pre-flight OPTIONS request detected, mocking a passing response.', {
url: request.url
});

const headers = fetchgen.convert_headers_map_to_array(request.headers);
// Pull Origin header so we now what to reply with to pass the CORS check
const cors_origin = fetchgen.get_header_value_ignore_case('Origin', headers);
const cors_method = fetchgen.get_header_value_ignore_case('Access-Control-Request-Method', headers);
const cors_headers = fetchgen.get_header_value_ignore_case('Access-Control-Request-Headers', headers);

await Fetch.fulfillRequest({
requestId,
responseCode: 200,
responseHeaders: [
{ name: 'Access-Control-Allow-Origin', value: cors_origin },
{ name: 'Access-Control-Allow-Methods', value: cors_method },
{ name: 'Access-Control-Allow-Headers', value: cors_headers },
{ name: 'Access-Control-Allow-Credentials', value: 'true' },
// Never cache CORS preflight, we need to make sure the request
// chain happens in the same order each time in case the user is doing
// an OPTIONS request through the proxy.
{ name: 'Access-Control-Max-Age', value: '0' },
{ name: 'Content-Type', value: 'text/plain; charset=utf-8' },
{ name: 'Content-Length', value: String(Buffer.byteLength('')) },
],
body: Buffer.from('').toString('base64'),
});
is_cors_preflight_handled = true;
return;
// Preserve the existing OPTIONS replay policy, but only mock
// real preflights. Access-Control-Request-Headers is optional.
if (is_request && !is_cors_preflight_handled) {
const preflight_headers = cors_preflight_response_headers(request);
if (preflight_headers) {
cdp_logger.debug('Pre-flight OPTIONS request detected, mocking a passing response.', {
url: request.url
});
await Fetch.fulfillRequest({
requestId,
responseCode: 200,
responseHeaders: preflight_headers,
body: ''
});
is_cors_preflight_handled = true;
return;
}
}

// Catch the base request and swap it
Expand Down Expand Up @@ -1378,20 +1407,10 @@ async function intercept_navigation_and_capture(tab, url_to_host_on, html_to_ser
status_code: responseStatusCode
});

let body = '';
if (!REDIRECT_STATUS_CODES.includes(responseStatusCode)) {
const response_tmp = await Fetch.getResponseBody({ requestId });
body = response_tmp.body;
if (response_tmp.base64Encoded) {
body = atob(body);
}
}
const raw_body = Buffer.from(body);

let formatted_headers = {};
responseHeaders.forEach(header_pair => {
formatted_headers[header_pair.name] = header_pair.value;
});
const raw_body = await read_captured_response_body(tab, requestId, request.method, responseStatusCode);
const formatted_headers = utils.fetch_headers_to_proxy_response_headers(
responseHeaders, raw_body, request.method, responseStatusCode
);

const response_string = fetchgen.get_blank_response();
await Fetch.fulfillRequest({
Expand Down Expand Up @@ -1528,7 +1547,7 @@ async function _manual_browser_visit(tab, url) {
patterns: [{ urlPattern: capture_url, requestStage: 'Response' }]
});

Fetch.requestPaused(async({ requestId, responseStatusCode, responseHeaders, responseErrorReason }) => {
Fetch.requestPaused(async({ requestId, request, responseStatusCode, responseHeaders, responseErrorReason }) => {
try {
if (responseErrorReason) {
await resolve({
Expand All @@ -1541,15 +1560,7 @@ async function _manual_browser_visit(tab, url) {
return;
}

let raw_body = Buffer.alloc(0);
if (!REDIRECT_STATUS_CODES.includes(responseStatusCode)) {
const response_tmp = await Fetch.getResponseBody({ requestId });
if (response_tmp.base64Encoded) {
raw_body = Buffer.from(response_tmp.body, 'base64');
} else {
raw_body = Buffer.from(response_tmp.body, 'utf8');
}
}
const raw_body = await read_captured_response_body(tab, requestId, request.method, responseStatusCode);

await Fetch.fulfillRequest({
requestId,
Expand All @@ -1558,15 +1569,9 @@ async function _manual_browser_visit(tab, url) {
body: Buffer.from(fetchgen.get_blank_response()).toString('base64'),
});

const normalized_headers = utils.fetch_headers_to_proxy_response_headers(responseHeaders);
if (typeof normalized_headers['content-length'] !== 'undefined') {
normalized_headers['content-length'] = String(raw_body.length);
}
if (typeof normalized_headers['Content-Length'] !== 'undefined') {
normalized_headers['Content-Length'] = String(raw_body.length);
}
delete normalized_headers['content-encoding'];
delete normalized_headers['Content-Encoding'];
const normalized_headers = utils.fetch_headers_to_proxy_response_headers(
responseHeaders, raw_body, request.method, responseStatusCode
);

cdp_logger.debug('Captured response headers.', {
headers: normalized_headers,
Expand Down
5 changes: 4 additions & 1 deletion hooks/afterrequest.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,10 @@ function get_header_value_ignore_case(headers, header_name) {
const target = header_name.toLowerCase();
for (const [key, value] of Object.entries(headers)) {
if (typeof key === 'string' && key.toLowerCase() === target) {
return value;
// Use a string view for metadata checks without changing the stored
// header values, especially separate Set-Cookie fields.
const values = Array.isArray(value) ? value : [value];
return values.filter(header_value => typeof header_value === 'string').join(', ');
}
}
return '';
Expand Down
Loading
Loading