Skip to content

Fix Product host first-sound blockers - #115

Merged
knzeng-e merged 21 commits into
devfrom
fix/product-host-first-sound-blockers
Aug 3, 2026
Merged

Fix Product host first-sound blockers#115
knzeng-e merged 21 commits into
devfrom
fix/product-host-first-sound-blockers

Conversation

@knzeng-e

Copy link
Copy Markdown
Owner

Outcome

Three blockers found while deploying Dotify to Product DevNet, plus one defect
found reviewing them. Each is a first-sound problem: something between opening
the app and hearing audio.

Stacked on #114. Refs #85.

The blockers

1. Cold API machines sat directly in front of first sound

services/api/fly.toml ran min_machines_running = 0. A stopped machine
cold-starts on the first content-key request, and that request is the last hop
before playback: 8.2s to /health cold against 0.11s warm. Scaling to zero
saved nothing a listener would trade eight silent seconds for.

Now keeps one machine warm. This has a standing cost - it is a deliberate trade
of idle spend against the single worst latency spike in the listening path.

2. One slow IPFS gateway blocked every gateway behind it

The gateway reader awaited each URL in turn with no timeout, so an unresponsive
gateway cost the browser's full connection timeout before the next candidate was
even attempted - and a browser will stall a request far longer than a listener
will wait for sound.

web/src/services/gatewayRace.ts replaces that with a bounded, hedged race:

  • 8s budget per attempt, time-to-headers only - once a gateway starts
    answering, the body streams at its own pace, because cutting a healthy
    download short trades a slow track for a broken one;
  • a 1.2s hedge delay before racing the next gateway alongside the current one,
    so a healthy primary still serves nearly every read alone;
  • at most 3 attempts in flight;
  • losers aborted the moment a winner is known.

Racing is safe here because every URL addresses the same immutable CID -
whichever answers first is the same object. This is the whole-object sibling of
audioV2Gateway, which already does this for byte ranges.

3. A refused room connection said nothing actionable

Socket.IO reports connect_error identically for a CORS rejection, a stopped
server, and a wrong URL. signalDiagnostics.ts reads the signaling server's
unauthenticated /health, which already echoes its live origin allowlist
(signaling.mjs:153), and upgrades the message in place:

Situation Message
/health unreachable server did not answer; may be starting or offline
origin not in allowedOrigins names the origin and says to add it to SIGNAL_ORIGINS
origin allowed, or * generic reason - the fault is elsewhere, so do not blame configuration

This matters most inside the Product host, where the app is served from a DotNS
origin an operator has to add deliberately. It only ever widens a message; room
access stays decided by the server.

Defect found in review

The hedged race broke cancellation of the winning read.

makeAttempt detaches its parent-abort listener when the fetch settles - which
is when headers arrive, before the body streams. The winner was returned
already disconnected from the caller's signal, so aborting afterwards no longer
stopped the download.

That is a regression against the serial reader it replaced, which passed the
caller's signal straight to fetch. And it bites where it matters: useCatalog
passes a signal to every audio and asset read (useCatalog.ts:700, 856-857),
so a listener skipping tracks left the previous audio downloading to completion,
unread.

Fixed by re-linking the winner to the caller's signal before returning it, and
by aborting the winner on the late-abort path - it has already been removed from
active, so the cleanup block would not otherwise reach it. Confirmed with a
regression test that fails against the previous implementation.

Review guide

  1. web/src/services/gatewayRace.ts - the race loop, the hedge/launch guard on
    caller abort, and the winner re-link.
  2. web/src/features/rooms/signalDiagnostics.ts - check the message matrix
    never over-claims.
  3. services/api/fly.toml - the warm-machine trade is a cost decision.

Verify carefully

  • A caller abort stops in-flight reads and does not walk the remaining list.
  • The winner's body is still readable after the losers are aborted.
  • The hedge cannot exceed 3 parallel attempts.
  • Diagnostics never claim an origin problem when the origin is allowed.
  • The warm machine is worth its standing cost.

Validation

Evidence Result
cd web && npm run test:unit 237 pass (+21 over #114: gateway race 12, signal diagnostics 8, plus the regression test)
cd web && npm run test:signal pass
cd web && npm run lint 0 errors; the 3 pre-existing App.tsx/ArtistShell.tsx warnings remain
cd web && npm run build pass - 4.4 MB
cd web && npm run build:product-devnet pass - 4.4 MB
cd web && npm run smoke:production-env pass
cd web && npm run smoke:devnet 6/6 pass against the live chain
cd services/api && npm run typecheck && npm test pass, 92 tests
node scripts/backlog-sync.mjs --check --offline pass; pre-existing warnings remain
git diff --check clean

Known limitations and follow-ups

The 8s timeout and 1.2s hedge delay are fixed constants chosen from the observed
DevNet gateway behaviour, not measured optima. If DevNet gateway latency shifts,
they are the first knobs to revisit.

min_machines_running = 1 removes the cold start but not the underlying cause -
the content-key request is synchronous in front of playback. A warmed read path
or an earlier key fetch would be the structural fix.

The signaling diagnosis costs one extra /health read on a failed connection.
It runs only after a failure, and never blocks the generic error already shown.

knzeng-e and others added 21 commits July 26, 2026 21:22
The Host signRaw wire format is not pinned by the SDK: HostSignPayloadResponse
carries an untagged signature, and a Substrate host may sign a raw payload
verbatim or inside the conventional <Bytes> envelope. Verification assumed one
shape, so a wrong guess would have failed every Product key request with an
error indistinguishable from a wrong signer.

Accept a bounded set instead: the canonical message verbatim or <Bytes>-wrapped,
and a bare 64-byte or MultiSignature-tagged 65-byte sr25519 signature. Every
variant carries the identical domain-bound message, so this adds no replay,
cross-app, cross-chain, or cross-track surface; a non-sr25519 tag still fails
closed. Route schemas widen to 128 or 130 hex so the tag is checked by the
verifier rather than rejected before it.

Reject EVM-derived account ids for product-sr25519-v1. A 20-byte H160 padded
with 0xee derives back to the H160 it contains, so accepting that shape let a
caller name any paying EVM listener as the requester and rested the boundary on
the curve check alone. A real Product account is a native AccountId32.

A key that parses and derives to the requester but verifies under no variant now
returns PRODUCT_SIGNATURE_REJECTED, kept distinct from SIGNATURE_INVALID so an
envelope problem is separable from a wrong-account problem in logs.

Pin @scure/sr25519 exactly, matching @noble/hashes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wiring the Product key signatures changed the behaviour but three places still
described the old one. The wallet modal told Product-host users that protected
playback required an EVM signer, the runbook asked operators to confirm no key
is released through the Product identity, and the architecture matrix said the
shipped UI used EIP-191 or a session token - contradicted by its own prose two
sections later.

All three now describe what ships: a connected Product account requests
protected keys through product-sr25519-v1, and paid access plus artist
publishing remain on the EVM signer. This matters beyond tidiness - the stale
runbook step would have had an operator sign off on a denial as correct
behaviour, hiding a real signing failure.

Record the signing envelope decision and the EVM-derived key rejection, and turn
the runbook step into an evidence capture that names which envelope the live
host actually produced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Track and split counts come from contract storage and the directory enumerates
runtimes Dotify does not control, so Array.from({ length: Number(count) })
allocated before anything could reject a malformed or hostile value. Both
adapters now validate counts first and throw rather than truncate, since a
silent cap would present a partial catalog as complete. The catalog loader
already isolates per-runtime failures, so one bad runtime degrades to a missing
artist.

Mark the two unverified spots in the CDM adapter that must be settled before it
can be selected: waitForTransaction returns immediately where the viem writer
awaits a receipt, and the payForAccess value-transfer shape is inferred rather
than confirmed against generated contract types.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dotify has no CDM-registered packages and no `cdm install`, but it does not
need them. Its Solidity contracts are deployed through Asset Hub's eth-rpc,
which is a compatibility layer over pallet-revive - the same pallet the Product
SDK contract helpers target - so the deployed H160 addresses are already
reachable without PolkaVM recompilation or a registry entry.

CdmJsonContract needs only version, address, and abi for getContract(), and
`new ContractManager(...)` is documented as snapshot-only. The generator emits
exactly that snapshot from the same Hardhat artifacts the viem bindings come
from, so the two adapters cannot disagree about an ABI.

Artist runtimes are deliberately absent from the manifest: a diamond is
deployed per artist, so its address is known at call time, not build time, and
a placeholder would misrepresent the deployment. Their merged facet ABI is
emitted separately and bound to an address by createContract.

The generator lives in web/ because it needs the SDK's codegen, which is a
frontend dependency; adding the Product SDK tree to contracts/evm just to emit
types would be a worse trade. Unnamed Solidity getter params are named
positionally for codegen only - generateContractTypes interpolates the name
into a tuple label and emits `args: [: HexString]`, which does not parse. The
manifest ABI stays byte-faithful to the artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Implements the contract resolver the CDM adapter was missing. Fixed-address
contracts resolve from the generated manifest through ContractManager;
per-artist runtime diamonds bind the merged facet ABI to their call-time
address through createContract.

Two SDK constraints shape this. First, createChainClient routes exclusively
through the Product host provider with no direct-WebSocket fallback, so Product
mode cannot work in a standalone build - validateProductionEnvironment now
rejects product-cdm unless the host mode is enabled. Second, the host decides
which chain an environment resolves to, and Dotify's runtimes live on Polkadot
Hub TestNet, reached through the `paseo` preset rather than `devnet`.
verifyDeployment queries the directory before any catalog read so a wrong-chain
connection fails with a named error instead of looking like artists with no
releases.

Selection is build-time rather than runtime, for two reasons. Switching the
authority for access policy is a deployment decision made with evidence, not
something a page should flip. And Vite inlines the value, so a viem build
tree-shakes the whole Product graph away: 4.4 MB against 10 MB when opted in.
The difference is @parity/product-sdk-descriptors, whose shared descriptors
module references every chain's metadata - only one chunk is ever fetched, but
all are published, and Bulletin storage is a finite quota. Both shipped builds
stay at 4.4 MB.

Reads only. Writes stay on the viem signer path in every mode, because routing
a payment or a publication through a signer with no host transaction evidence
is not a reasonable default. A failed Product setup rejects every read rather
than falling back to viem: the adapter in use must never be ambiguous.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The blocking unknown for Product contract mode is no longer the manifest or
generated types - both now exist and are wired. It is whether the Product host
serves a chain that holds Dotify's runtimes, since the host controls that
mapping and the contracts are on Polkadot Hub TestNet rather than Product
DevNet Asset Hub.

Also records the measured build-size trade-off, so an operator weighs it
against the Bulletin quota before enabling product-cdm for a .dot deployment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous default was wrong in a way that would have produced an empty
catalog. Product DevNet is not a separate network: it is a preset over the
Paseo system parachains - Asset Hub (1000), People (1004), Bulletin (1010) -
at EVM chain 420420417. That is exactly where Dotify is already deployed.

Verified read-only against both endpoints for the ArtistDirectory at
0xcf1534c6e2b0e43b9436c1e86a076466dc0f2108: eth-rpc-testnet.polkadot.io and
paseo-assethub-rpc.laissez-faire.trade both report chain id 0x190f1b41, blocks
one apart, and byte-identical contract code. They are two providers for one
chain, so no contract redeploy is needed to port Dotify to DevNet.

The SDK's `paseo` preset is the trap: it targets Paseo Next (Asset Hub Next
1500 / People Next 1502), which the Product docs call a different network where
"funds sent there will not appear on this Devnet". Dotify has no deployment
there, so ProductChainEnvironment now admits only `devnet` - selecting a chain
that cannot hold the catalog is a bug, not a configuration option.

Also realigns the environment reference tables, clearing the markdownlint
MD060 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Porting Dotify to Product DevNet turned out to be a configuration question
rather than a migration: DevNet is a preset over the Paseo system parachains
(Asset Hub 1000, People 1004, Bulletin 1010) at EVM chain 420420417, which is
where Dotify's contracts already are. This makes that claim checkable instead
of asserted.

`npm run smoke:devnet` reads web/.env.product-devnet and deployments.json and
verifies, read-only, that the configured Asset Hub reports chain 420420417, is
producing blocks past the 2026-07 halt, still serves bytecode for the
ArtistDirectory and ArtistRuntimeFactory, and that the Bulletin RPC and IPFS
gateway respond. It sends no transaction, reads no secret, and prints no
credential. Network-dependent, so it stays out of the unit test path.

The build profile needed no endpoint changes - the configured Bulletin and IPFS
gateway were already the DevNet ones. What it needed was honest comments: the
previous note framed the Asset Hub endpoint as a DevNet-compatible stopgap when
it is in fact the DevNet chain. Adds the second DevNet IPFS gateway as a read
fallback, and a warning against Asset Hub Next (1500) and People Next (1502),
which are a different network holding none of Dotify's contracts.

Verified: 6/6 checks pass against the live chain at head 11546553.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The known-limits bullet still said Product contract mode was blocked on
CDM-installed packages and generated contract types. Both now exist, so the
list overstated what is missing. Only pallet-revive account mapping and
host-signed transaction evidence remain.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fetchIpfsCid and fetchAssetRef walked 4-6 gateways serially with no
per-gateway timeout, so one unresponsive gateway cost the listener the
browser's full connection timeout before the next candidate was tried.
Covers, manifests, and v1 audio all read through that path.

Race the candidates instead, the way audioV2Gateway already does for byte
ranges: bound time-to-headers, hedge onto the next gateway after a delay,
cap concurrency at 3, and abort the losers once a winner answers. Every
URL addresses the same immutable CID, so racing them cannot diverge.

The winner's own controller is never aborted, so its body stays readable,
and a caller abort stops the queue where it is rather than walking the
rest of the list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
min_machines_running = 0 let the API scale to zero, and the first content
key request after an idle period paid the cold start. Measured against the
running deployment: 8.17s to /health cold (uptime 0), 0.11s warm.

That request sits directly in front of first sound, so the saving was
being taken out of the listening experience.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Socket.IO reports a CORS rejection, a stopped server, and a wrong URL
identically, so every one of them surfaced as "Room service unavailable."
That is the vague failure the product invariants rule out, and it hid a
real deployment gap: the running signaling server allows only
muzinga.netlify.app, so opening a room from the Polkadot Product host
origin is refused with a 403 the browser will not explain.

On connect_error, read the server's unauthenticated /health, compare the
page origin against the allowlist it reports, and upgrade the message in
place. The generic reason stands if health cannot be read or if the origin
is allowed, so this only ever widens an error - room access stays decided
by the server.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploying with `flyctl deploy -c services/api/fly.toml` from the repository
root fails at `COPY src ./src`: -c selects the config file, but the build
context stays the shell's working directory, and the Dockerfile is written
against services/api. It also uploads a ~1.3 GB context, because Docker reads
.dockerignore from the context root and only the service directories have one.

A cached `npm ci` layer from an earlier correct build hides the cause, so the
error surfaces at the first uncached step rather than the first wrong one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`makeAttempt` detaches its parent-abort listener as soon as the fetch settles,
which is when headers arrive - before the body has streamed. The winner was
therefore returned already disconnected from the caller's signal, so aborting
afterwards no longer stopped the download.

That is a regression against the serial reader this replaced, which passed the
caller's signal straight to `fetch`. It bites where it matters: `useCatalog`
passes a signal to every audio and asset read, so a listener skipping tracks
left the previous audio downloading to completion, unread.

Re-link the winner to the caller's signal before returning it, and abort the
winner on the late-abort path too - it has already been removed from `active`,
so the cleanup block would not otherwise reach it.

Covered by a regression test that failed before this change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inside the Product host the app is served from polkadot://app.dotify-test01.dot,
not the DotNS web gateway. SIGNAL_ORIGINS already carried it; API_ORIGINS did
not, so a container would have had working rooms and no content keys - catalog
and key delivery both go to the API, so free and protected playback would fail
CORS while the room layer looked healthy.

Deliberately does not add a bare `null`. `polkadot:` is a non-special scheme, so
its origin is opaque and a browser may send `Origin: null` instead of the
literal value. Allowing that would admit every sandboxed iframe and file:// page
on the web to the authenticated upload and content-key routes. If a host request
is still refused, the actual Origin header from the Fly log is the evidence to
act on. Two regression tests pin both halves: the custom-scheme origin is
answered, a null origin is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@knzeng-e
knzeng-e changed the base branch from feat/product-devnet-endpoint-truth to dev August 3, 2026 13:33
@knzeng-e
knzeng-e marked this pull request as ready for review August 3, 2026 13:34
@knzeng-e
knzeng-e merged commit f801619 into dev Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant