feat: per-connection backend authentication via OIDC federation

[alukach]

Adapts the proxy's backend connections to make authenticated requests, replacing the always-unsigned path. The proxy side is complete end-to-end; production activation is gated on the Source API surfacing the role (see Production status).

Model

DataConnection gains an authentication field — a sibling of details, matching the Source API's shape:

pub enum BackendAuth {            // #[serde(tag = "type")], default Unsigned
    Unsigned,                                  // public bucket — unsigned requests
    S3WebIdentityRole { role_arn: String },    // federate proxy OIDC identity -> AWS role
    Unsupported,                               // #[serde(other)] — unknown/unimplemented type
}

resolve_product branches via apply_backend_auth, replacing the long-standing // TODO: provide real backend credentials forced skip_signature insert:

• Unsigned (default) → skip_signature (current behavior).
• S3WebIdentityRole → auth_type=oidc + oidc_role_arn + a per-connection subject scv1:conn:{id}, leaving signing on so the middleware injects the federated credentials.
• Unsupported → fails closed with BackendAuthError rather than silently serving unsigned.

End-to-end federation

lib.rs wires MaybeOidcAuth(AwsBackendAuth(OidcCredentialProvider)) into the gateway via .with_middleware, backed by a reqwest FetchHttpExchange for the worker. For an auth_type=oidc connection the middleware mints the proxy's RS256 assertion (iss = OIDC_PROVIDER_ISSUER, aud = sts.amazonaws.com, sub = scv1:conn:{id}), exchanges it at AWS STS (AssumeRoleWithWebIdentity) over fetch, and injects the temporary credentials so the backend request is signed. A no-op for unsigned/public connections.

Resilience

• Lenient deserialization — the proxy parses the whole data-connection list in one serde_json::from_str, so a deserialize_with degrades a present-but-malformed authentication (null, wrong shape, missing role_arn, unknown type) to Unsigned/Unsupported instead of failing the entire parse and breaking resolution for every product.
• Fail closed on Unsupported (the app-side gcp_workload_identity / azure_workload_identity variants this build doesn't implement, or malformed config) — a connection that can't be authenticated isn't served at all, rather than falling back to an anonymously-readable unsigned request.

Performance

The gateway is rebuilt per fetch(), so the credential provider is held in an isolate-level OnceLock and cloned per request; cloning shares the cache (enabled by the pinned multistore feat/shareable-credential-cache), so repeat federated requests reuse cached temporary credentials instead of re-minting a JWT and re-running AssumeRoleWithWebIdentity every call.

Observability

BackendAuth::kind() (unsigned / s3_web_identity_role / unsupported — no secrets, no ARN) is recorded on the resolve_product span, so an operator can see which backend-auth path a request took and correlate a fail-closed Unsupported or an STS 403 without leaking the role ARN.

Testing

backend_auth is extracted into a wasm-free module so it can be natively unit-tested despite the lib being cdylib + test = false — tests/backend_auth.rs includes it via #[path], mirroring tests/pagination.rs. Covered: serde round-trips (unknown type → Unsupported, malformed → degrade, the s3_web_identity_role variant) and the option-set translation for each variant, including fail-closed. An env-gated tests/test_federation.py smoke test exercises the full live path and skips unless FEDERATION_TEST_ACCOUNT/PRODUCT/KEY are set.

Temporary multistore pin

The consolidated backend-auth work (oidc-provider owning the credential exchange, the shareable credential cache) isn't on crates.io yet, so [patch.crates-io] pins all five multistore crates to an exact rev (not branch=main, which cargo update could silently float) for reproducible builds. Drop the patch and bump the versions once multistore ships.

Production status

Non-breaking and currently inert: the Source API still redacts authentication, so every connection resolves Unsigned and behaves exactly as before. Remaining go-live steps (outside this PR):

1. The app surfacing the connection's role_arn to the proxy (source.coop #327 / #329).
2. The customer-side IAM OIDC provider + role trust policy (conditioned on aud = sts.amazonaws.com, sub = scv1:conn:{id}) and a federated test product in staging to activate the smoke test.

🤖 Generated with Claude Code

[github-actions]

🚀 Latest commit deployed to https://source-data-proxy-pr-147.source-coop.workers.dev

• Date: 2026-06-19T16:00:53Z
• Commit: 21c5330

[claude]

Claude finished @alukach's task in 2m 0s —— View job

---

✅ No blocking issues — safe to merge.

• S3WebIdentityRole applied without a provider guard (src/backend_auth.rs:104-110, src/registry.rs:197-201): apply_backend_auth has no knowledge of backend_type and doesn't check that the OIDC path is only reached for S3 connections. If a non-S3 connection (Azure, GCS) is misconfigured with S3WebIdentityRole authentication, the function inserts auth_type=oidc/oidc_role_arn/oidc_subject but never sets skip_signature. Depending on how multistore's Azure/GCS backends handle unknown options, the request may go out unsigned (data exposure) or fail opaquely rather than with a clear BackendAuthError. A one-liner guard in resolve_product before the apply_backend_auth call would close this:

  if !matches!(backend_type.as_str(), "s3") {
      if !matches!(connection.authentication, BackendAuth::Unsigned) {
          return Err(ProxyError::BackendAuthError(format!(
              "connection {}: {} auth is not supported for {} backends",
              connection.data_connection_id, connection.authentication.kind(), backend_type
          )));
      }
  }

  Low risk given the Source API controls authentication and you've noted S3-only scope, but worth tracking as S3WebIdentityRole expands.

• anonymous_access: true for all resolutions (src/registry.rs:217): federated connections now back private buckets, yet anonymous_access remains hardcoded true. The confused-deputy comment explains this is intentional (layer-1 authz lives in the subject-scoped Source API fetches), and this is pre-existing, not introduced here. Worth surfacing as a follow-up if multistore's gateway ever adds additional credential checks at that flag.

• deserialize_lenient double-deserializes through serde_json::Value (src/backend_auth.rs:76-80): correct but note the unwrap_or(BackendAuth::Unsupported) is genuinely load-bearing — it catches structurally valid JSON that fails BackendAuth's schema (e.g. s3_web_identity_role missing role_arn), since #[serde(other)] only fires on an unknown type tag, not on a known-but-malformed variant. The existing test lenient_malformed_becomes_unsupported_not_error covers this well.

• OIDC_PROVIDER initialized from first-request config (src/lib.rs:94): fine for Workers (single-threaded Wasm isolates, config is constant for the isolate's lifetime). No issue, just confirming the assumption holds.