feat(deploy): stage streamed payloads to a temp file for replication

[kriszyp]

Stacked on #531 — please review #530 and #531 first; this PR will be retargeted to `main` after both land.

Summary

Origin-side payload staging needed for the direct-HTTPS peer relay (harper-pro slice 3b). When a `deploy_component` request will be replicated to peers, the origin now writes the streamed payload to a temp file before local extraction so it can re-stream the same bytes to each peer afterwards. Without this, the payload Readable is consumed once by local extract and gone by the time replication runs.

Third slice of #524.

Where to look

• `components/payloadStaging.ts` — single-purpose utility that pipelines a Readable into a temp `payload.tar.gz` and returns a cleanup callback. The trade-off comment in the file header explains why we write-then-read instead of teeing in flight (simpler code, identical backpressure for extraction; disk speed isn't the bottleneck).
• `components/operations.js` — staging gate in `deployComponent`. Skipped when:
  • the deploy is package-identifier-based (`req.package`, no `payload` stream),
  • `req.replicated === false` (the operation is itself a replicated deploy that the peer received, so no further fan-out),
  • or there are no peers (`server.nodes.length === 0` — single-node deploys keep their zero-disk-copy property).
• Cleanup runs in a `finally` after `server.replication.replicateOperation(req)`, plus a defensive cleanup if `prepareApplication` fails before replication.

Backward compatibility

• The staging is invisible to the operations API contract: `req.payload` is replaced internally with a `createReadStream` over the temp file before the `Application` is constructed, so every downstream code path (`Application.extractApplication`, the SSE phase emitter, anything reading `application.payload`) keeps seeing a `Readable` exactly as before.
• Single-node and package-identifier deploys: zero change in behavior, no temp file written.

Test plan

• `npx mocha unitTests/components/payloadStaging.test.js` — 5/5 pass (round-trip contents, cleanup removes file+dir, double-cleanup is safe, path-traversal sanitization keeps the staged dir under tmpdir, source-stream errors propagate).
• `npx mocha unitTests/server/fastifyRoutes/operations.test.js` — 21/21 pass (existing deployComponent tests; the staging branch is only entered when nodes are present, which these tests don't simulate).
• `npx mocha unitTests/components/packageComponent.test.js unitTests/server/serverHelpers/multipartParser.test.js unitTests/bin/multipartBuilder.test.js unitTests/server/serverHelpers/progressEmitter.test.js unitTests/bin/sseConsumer.test.js` — 32/32 pass (slice 1 + slice 2 unaffected).
• End-to-end replicated deploy — exercised in harper-pro's slice 3b integration test, not here.

🤖 Generated with Claude Code

[claude]

2 blockers unchanged from the prior run.

1. Staged payload leaks on load-phase failure

File: components/operations.js (load-phase throw lastError)
What: The throw lastError that fires after loadComponent exits deployComponent before the replicateOperation try/finally block — the only place stagedPayload?.cleanup() is called. The prepareApplication catch block doesn't cover this code path.
Why it matters: A multi-node streaming deploy that fails the load check accumulates unbounded temp files.
Suggested fix: Add stagedPayload?.cleanup().catch(() => {}) immediately before throw lastError in the load-error branch.

2. stagePayloadToTempFile leaks temp dir on pipeline failure

File: components/payloadStaging.ts:30 (inline comment posted)
What: mkdtemp creates dir before pipeline runs. If pipeline rejects, the function throws without returning a cleanup handle — the caller has no way to delete the orphaned directory. The new test covers error propagation but not the resulting disk leak.
Suggested fix:

try {
    await pipeline(source, createWriteStream(path));
} catch (err) {
    await rm(dir, { recursive: true, force: true }).catch(() => {});
    throw err;
}

[claude]

This catches prepareApplication failures, but there's a gap between this catch and the replicateOperation finally: the load-phase throw lastError path (the if (lastError) throw lastError block that runs when the pseudo-resource load fails, after line ~452) is not covered by either cleanup handler. If a multi-node streaming deploy reaches the load phase and the component fails to load, stagedPayload leaks.

Suggested change

	stagedPayload?.cleanup().catch(() => {});
	stagedPayload?.cleanup().catch(() => {});

Same one-liner before throw lastError in the load-error branch (or restructure with a single outer try/finally wrapping both phases).

[claude]

Temp dir leaks on pipeline failure (still open from prior run)

mkdtemp already created dir before pipeline runs. If pipeline rejects (source stream error), the function throws without returning a cleanup handle — the caller has no way to delete the orphaned directory. The error-propagation test asserts the rejection propagates but does not assert the temp dir is removed.

Suggested change

	await pipeline(source, createWriteStream(path));
	const path = join(dir, 'payload.tar.gz');
	try {
	await pipeline(source, createWriteStream(path));
	} catch (err) {
	await rm(dir, { recursive: true, force: true }).catch(() => {});
	throw err;
	}

Also add a test assertion: after stagePayloadToTempFile rejects, stat(path.dirname(...)) should throw ENOENT.

[claude]

Temp dir leaks on pipeline failure (carry from prior run)

mkdtemp already created dir before pipeline runs. If pipeline rejects, the caller never receives a cleanup handle — the directory is orphaned. The error-propagation test asserts the rejection propagates but doesn't assert the temp dir is removed afterward.

Suggested change

	await pipeline(source, createWriteStream(path));
	try {
	await pipeline(source, createWriteStream(path));
	} catch (err) {
	await rm(dir, { recursive: true, force: true }).catch(() => {});
	throw err;
	}

[claude]

Temp dir leaks on pipeline failure (carry from prior run — still open)

mkdtemp already created dir before pipeline runs. If pipeline rejects (source stream error), the function throws without returning a cleanup handle — the caller has no way to delete the orphaned directory.

Suggested change

	await pipeline(source, createWriteStream(path));
	const path = join(dir, 'payload.tar.gz');
	try {
	await pipeline(source, createWriteStream(path));
	} catch (err) {
	await rm(dir, { recursive: true, force: true }).catch(() => {});
	throw err;
	}

[claude]

Temp dir leaks on pipeline failure (carry from prior run — still open)

mkdtemp already created dir before pipeline runs. If pipeline rejects (source stream error), the function throws without returning a cleanup handle — the caller has no way to delete the orphaned directory.

Suggested change

	await pipeline(source, createWriteStream(path));
	const path = join(dir, 'payload.tar.gz');
	try {
	await pipeline(source, createWriteStream(path));
	} catch (err) {
	await rm(dir, { recursive: true, force: true }).catch(() => {});
	throw err;
	}

[claude]

Still leaks dir on pipeline failure. mkdtemp already ran at line 28; if pipeline throws here the function exits without returning a cleanup handle, so the caller can never delete the orphaned temp directory.

Suggested change

	await pipeline(source, createWriteStream(path));
	try {
	await pipeline(source, createWriteStream(path));
	} catch (err) {
	await rm(dir, { recursive: true, force: true }).catch(() => {});
	throw err;
	}

[kriszyp]

Pausing this PR — converting to draft.

This work is being rebuilt on top of the design in #641: a replicated hdb_deployment system table that owns the deploy lifecycle, audit trail, and (because Harper's BLOB_CHUNK replication already supports chunked, back-pressured blob transfer) the payload delivery channel.

In the new design:

• The ProgressEmitter from this PR becomes one of two subscribers — the second being a DeploymentRecorder that writes the persistent record. SSE remains the live channel for the CLI; get_deployment becomes content-negotiated to serve the same stream to Studio.
• _stagedPayloadPath (feat(deploy): stage streamed payloads to a temp file for replication #536) and the direct-HTTPS peer relay (harper-pro#146) both go away — peers read the payload directly from the replicated blob attribute on the row.

This work resumes as part of Slice B in #641 once Slice A (table + blob-backed multipart receive) lands.

— Claude