diff --git a/MODULE.md b/MODULE.md index ca417f000..e2da81857 100644 --- a/MODULE.md +++ b/MODULE.md @@ -299,8 +299,9 @@ Two client sources, which cannot collide — a registry slug can never spell fetched (bounded size and time, **no redirect at all** per CIMD draft-02 §5.1, and guarded at the dial against every non-public destination — the IANA special-purpose registries for both families, not just loopback and RFC 1918, - with IPv6 bounded to the `2000::/3` global unicast allocation; every other - tenant-directed egress uses the same classifier, listed in + with IPv6 bounded to the `2000::/3` global unicast allocation; the other + guarded egress paths use the same classifier, and the paths it covers and the + OIDC clients it does not are listed in [CALLABLE_SOURCES.md](./module/CALLABLE_SOURCES.md#oauth-and-disclosure)), validated, and used as a public client for that one flow. Failures and invalid documents are **not** cached (§5.2), so a publisher who fixes one is not made diff --git a/module/CALLABLE_SOURCES.md b/module/CALLABLE_SOURCES.md index 19345dd37..b7b810e87 100644 --- a/module/CALLABLE_SOURCES.md +++ b/module/CALLABLE_SOURCES.md @@ -139,11 +139,13 @@ Transport uses the resolved-IP egress guard and same-origin redirect policy. The guard runs at connect time, on the address DNS returned, for the operation call and every OAuth token exchange alike, and admits only an address on the public internet. That judgement is `auth.IsPublicDestination`, the one -classifier for every tenant-directed egress in this host: callable sources -(operation calls and OAuth token exchanges), the documentation crawler, the -object-store connector, outbound webhooks and the Slack notifier that shares -their client, and Client ID Metadata Document fetches. Each of those guards is -tested against the same vectors, `pkg/auth/egresstest`. It reads the +classifier shared by these guarded egress paths: callable sources (operation +calls and OAuth token exchanges), the documentation crawler, the object-store +connector, outbound webhooks and the Slack notifier that shares their client, +and Client ID Metadata Document fetches. Each of those guards is tested against +the same vectors, `pkg/auth/egresstest`. The per-organization OIDC discovery, +JWKS and token clients (`pkg/auth/oidc`) are not guarded. No RPC exposes +`ConfigureOrgIdentityProvider` today, so no tenant can name their destination. It reads the IANA IPv4 and IPv6 special-purpose registries rather than Go's `net.IP` predicates, which call carrier NAT (`100.64.0.0/10`), benchmarking, reserved and the IPv6 transition prefixes "global unicast", and it bounds IPv6 to @@ -152,12 +154,75 @@ Teredo are refused outright rather than decoded, because a network-specific NAT64 prefix cannot be recognised and a decoded public address is still reached through a translator on the host's own network. A v4-mapped `::ffff:a.b.c.d` is judged as `a.b.c.d`. A zoned address, or one the guard cannot parse, is refused. +The one exception is a local runtime's explicit list of loopback addresses, +described in [Local egress](#local-egress); every address it does not list is +still judged by the guard. Named refusal/receipt/audit constructors discard provider error bodies and raw transport errors. Credential reflection in a successful provider body is refused. Audit records actor, source, operation, effect, effect ID and the closed outcome vocabulary (`committed`, `refused`, `unresolved`, `rate_limited`), never input, output, tokens or URLs. Failed operation resolution uses an `UNKNOWN` effect. +## Local egress + +A composition booted locally can prove an OAuth REST source end to end against a +stub provider and API running beside the host. It lists the stub's exact +addresses in `datasource-oauth/local_egress`, a configuration value +([configuration groups](configurations/README.md#datasource-oauth)): + +``` +local_egress=127.0.0.1:18080,[::1]:18081 +``` + +The allowance holds to these rules: + +- **Empty by default.** The shipped value is empty, and an empty value changes + nothing anywhere. +- **Refused outside a local runtime.** A non-empty value refuses startup, naming + the key, before any other startup check runs. A local runtime has to satisfy two + facts the host already reads. First, the Codefly environment is `local` + (`codefly.IsLocal`, which every other local-only exemption reads). Second, the + process is not in a Kubernetes pod: it has no `KUBERNETES_SERVICE_HOST` and no + projected service-account token or CA at their fixed paths. The pod check reads + the same facts as the solution host's in-cluster identity. A host started by + `codefly run` has none of them. A local Kubernetes cluster is refused too. + + What defeating both checks takes: + - **The environment** is one variable, set by whatever renders the deployment. + - **The pod facts** come from the kubelet, and no configuration value can set + or clear them. A Codefly-rendered service pod mounts no service-account token + by default (`automountServiceAccountToken: false`), so in such a pod the only + pod fact is `KUBERNETES_SERVICE_HOST`. + - So a deployment passes both checks only with control of the pod spec: + `CODEFLY__ENVIRONMENT=local` plus an explicit container-env override that + clears `KUBERNETES_SERVICE_HOST`. It cannot pass them through a + configuration value. + - A deployment that can name itself `local` has already lifted the refusals + of the dev identity provider and the ephemeral signing key, which are worse + than this allowance. +- **Explicit loopback `ip:port` entries only.** Each entry is one loopback IP + literal and one port, written canonically. A hostname, range, CIDR, wildcard, + zone, IPv4-mapped spelling, port 0 or non-loopback address refuses the whole + value. Loopback never leaves the host's own network namespace. Even a host that + got past the runtime check could reach only that namespace: never a metadata + service, the cluster network or another tenant. Within that namespace, + loopback reaches everything listening there: + - on a developer's machine, every local listener, so the developer picks the + port; + - in a pod, every container in the pod, sidecars included. A mesh proxy's + admin port is on loopback. +- **Never widens the public-address guard.** The allowance is checked at dial + time against the resolved address and port, ahead of the guard. Any address it + does not list is judged by the guard exactly as before. The allowance lives in + its own file (`apisource/local_egress.go`) beside the guard, so each can be + reviewed separately. +- **A value, never a file.** Codefly delivers it like every other group value. + +It applies to the token exchange (refresh, `client_credentials`, +authorization-code) and the operation call alike. A Docker-runtime local run +reaches a stub through a private address and is not covered. Run the host +natively, which is the `codefly run` default. + ## Generation and qualification Core's option import closure is copied byte for byte under the accounts proto's diff --git a/module/WEBHOOKS.md b/module/WEBHOOKS.md index f0d484284..3d40cc479 100644 --- a/module/WEBHOOKS.md +++ b/module/WEBHOOKS.md @@ -3,11 +3,12 @@ Webhook endpoints must be public `https://` URLs on port 443. Accounts resolves all A/AAAA records when an endpoint is registered and again immediately before each connection. If any answer is not on the public internet, the request is -rejected. That judgement is `auth.IsPublicDestination`, the classifier every -tenant-directed egress in this host shares (see -[CALLABLE_SOURCES.md](./CALLABLE_SOURCES.md#oauth-and-disclosure)): the IANA -IPv4 and IPv6 special-purpose registries, with IPv6 bounded to the `2000::/3` -global unicast allocation. A connection is pinned to the address that was +rejected. That judgement is `auth.IsPublicDestination`, the classifier the +host's guarded egress paths share. The paths it covers, and the OIDC clients it +does not, are listed in +[CALLABLE_SOURCES.md](./CALLABLE_SOURCES.md#oauth-and-disclosure). It reads the +IANA IPv4 and IPv6 special-purpose registries, with IPv6 bounded to the +`2000::/3` global unicast allocation. A connection is pinned to the address that was judged. Redirects are returned as failed attempts and never followed. Kubernetes egress policy is an independent second layer over the special-purpose ranges it lists; the application check is the complete one. diff --git a/module/configurations/README.md b/module/configurations/README.md index 422bba010..9d6d4f3f3 100644 --- a/module/configurations/README.md +++ b/module/configurations/README.md @@ -6,15 +6,25 @@ consuming workspaces override a group rather than editing module-owned files. ## datasource-oauth -Accounts reads three JSON maps. This version supports one deployment app, -keyed by `api`; other provider keys are refused, so configuration cannot replace -a connector-owned linker: +Accounts reads three JSON maps and one local-only list. This version supports +one deployment app, keyed by `api`; other provider keys are refused, so +configuration cannot replace a connector-owned linker: | Key | Delivery | Content | | --- | --- | --- | | `providers` | value | `authorize_url`, `token_url`, and `scopes` for each provider | | `client_id` | value | The deployment's registered client identifier | | `client_secret` | secret | The confidential client secret; omit for public clients | +| `local_egress` | value | Local runtime only: loopback `ip:port` entries a callable source may dial. Empty by default | + +`local_egress` lets a host booted on a developer's machine reach a stub OAuth +provider and API beside it, for example `127.0.0.1:18080,[::1]:18081`. Each +entry is one loopback IP literal and one port, written canonically: no hostname, +range, CIDR, wildcard, zone or port 0. It applies to every callable-source dial, +both the token exchange and the operation call. Any non-empty value refuses +startup unless the Codefly environment is `local` **and** the process is not in +a Kubernetes pod. A deployed composition that sets it does not start. See +[Local egress](../CALLABLE_SOURCES.md#local-egress). The local provider and client-id maps are empty, which disables generic sign-in. The secret map contains only a local placeholder for `api`. For example, diff --git a/module/configurations/local/datasource-oauth.env b/module/configurations/local/datasource-oauth.env index 546ff8b05..86c201adb 100644 --- a/module/configurations/local/datasource-oauth.env +++ b/module/configurations/local/datasource-oauth.env @@ -3,3 +3,8 @@ providers={} # Public app identifiers keyed by the same connector. Empty disables linking. client_id={} +# Local runtime only: loopback ip:port entries a callable source may dial, so a +# host booted on a developer's machine can reach a stub provider beside it. +# e.g. 127.0.0.1:18080,[::1]:18081. Empty by default; any value refuses startup +# outside a local runtime (module/CALLABLE_SOURCES.md, "Local egress"). +local_egress= diff --git a/module/services/accounts/code/pkg/auth/egresstest/vectors.go b/module/services/accounts/code/pkg/auth/egresstest/vectors.go index 75ba31da5..d70745e22 100644 --- a/module/services/accounts/code/pkg/auth/egresstest/vectors.go +++ b/module/services/accounts/code/pkg/auth/egresstest/vectors.go @@ -1,5 +1,5 @@ // Package egresstest is the set of addresses every egress guard in this host -// is held to. Each guard that decides whether a connection may leave for a +// is held to (the paths with a guard are listed in auth/public_address.go). Each guard that decides whether a connection may leave for a // tenant-named destination answers through auth.IsPublicDestination, and each // guard's own test runs these vectors against it, so a guard that stops using // that classifier — or a classifier change that re-admits a range — fails a diff --git a/module/services/accounts/code/pkg/auth/public_address.go b/module/services/accounts/code/pkg/auth/public_address.go index 4fa235798..1cae439cc 100644 --- a/module/services/accounts/code/pkg/auth/public_address.go +++ b/module/services/accounts/code/pkg/auth/public_address.go @@ -14,7 +14,10 @@ import ( // it: callable sources (datasource/apisource), the crawler and object-store // connectors, outbound webhooks and the Slack notifier that shares their client // (business.WebhookEndpointPolicy), and Client ID Metadata Document fetches. -// Each of those guards' tests holds the vectors in auth/egresstest. +// Each of those guards' tests holds the vectors in auth/egresstest. The +// per-organization OIDC discovery, JWKS and token clients (auth/oidc) have no +// guard: nothing exposes ConfigureOrgIdentityProvider over RPC, so no tenant +// names their destination today. // // It does not compose Go's net.IP predicates, because they answer a different // question. net.IP.IsGlobalUnicast is not a public-routability test — it diff --git a/module/services/accounts/code/pkg/business/datasource_local_egress_integration_test.go b/module/services/accounts/code/pkg/business/datasource_local_egress_integration_test.go new file mode 100644 index 000000000..f14311fc3 --- /dev/null +++ b/module/services/accounts/code/pkg/business/datasource_local_egress_integration_test.go @@ -0,0 +1,121 @@ +//go:build !pure + +package business_test + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" + + "accounts/pkg/auth" + "accounts/pkg/business" + "accounts/pkg/datasource/apisource" + "accounts/pkg/datasource/connector/connectortest" + "accounts/pkg/datasource/operations" + gen "accounts/pkg/gen/saas/accounts/v1" + + "github.com/stretchr/testify/require" + "google.golang.org/protobuf/proto" +) + +// #1087, at the service level: an httptest provider and API beside the host, +// reached through a client_credentials source with the real store, the real +// Vault envelope and the REAL guarded transport — neither the token exchange +// nor the operation client is substituted. The only thing that lets the dials +// through is a local runtime's allowance naming the stub's exact address. +func TestClientCredentialsSourceReachesALocalStubThroughTheEgressAllowance(t *testing.T) { + clearData(t) + actor, org := mustUserAndOrg(t, testCtx, "local-egress@example.com", "local-egress", "Acme") + svc, _ := realOAuthService(t) + clientSecret := randomCredential(t) + const accessToken = "stub-access-token-0123456789abcdef" + var tokenCalls, apiCalls atomic.Int32 + stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/token": + tokenCalls.Add(1) + if err := r.ParseForm(); err != nil || r.PostForm.Get("grant_type") != "client_credentials" || + r.PostForm.Get("client_id") != "example-client" || r.PostForm.Get("client_secret") != clientSecret { + w.WriteHeader(http.StatusUnauthorized) + _, _ = w.Write([]byte(`{"error":"invalid_client"}`)) + return + } + _ = json.NewEncoder(w).Encode(map[string]any{"access_token": accessToken, "expires_in": 3600}) + case "/items/a": + apiCalls.Add(1) + if r.Header.Get("Authorization") != "Bearer "+accessToken { + w.WriteHeader(http.StatusUnauthorized) + return + } + _, _ = w.Write([]byte(`{}`)) + default: + http.NotFound(w, r) + } + })) + defer stub.Close() + t.Cleanup(func() { _ = apisource.ConfigureLocalEgress("", false) }) + + config := &business.APIDatasourceConfig{ + BaseURL: stub.URL, + CredentialKind: business.APICredentialKindOAuth2, + OAuth2: &business.APIOAuth2Config{ + Grant: business.OAuth2ClientCredentials, + TokenURL: stub.URL + "/token", + ClientID: "example-client", + }, + } + source, err := svc.AddSource(testCtx, actor, business.AddSourceInput{OrgID: org, Provider: business.DatasourceProviderAPI, CollectionLabel: "Acme", API: config, Credential: clientSecret}) + require.NoError(t, err) + + // No allowance: the public-address guard refuses the token exchange, and + // the stub never sees a request. + require.NoError(t, apisource.ConfigureLocalEgress("", true)) + copy := *source + _, err = svc.ResolveDatasourceTokenForTest(testCtx, ©) + require.Error(t, err, "loopback must stay unreachable without an allowance") + require.Zero(t, tokenCalls.Load()) + + require.NoError(t, apisource.ConfigureLocalEgress(stub.Listener.Addr().String(), true)) + declaration := connectortest.OperationDeclaration() + requestCtx := auth.WithVerifiedDatabaseIdentity(testCtx, actor, org) + grantOperationBoundary(t, requestCtx, actor, org, source.BoundaryNodeID) + _, err = svc.DeclareSourceOperations(requestCtx, actor, org, source.ID, []operations.Declaration{declaration}) + require.NoError(t, err) + receiptStore, closeStore, err := testStore.NewEffectReceipts() + require.NoError(t, err) + defer closeStore() + require.NoError(t, svc.ConfigureSourceOperationReceipts(receiptStore)) + _, guard := svc.SourceOperationReceipts() + const effect, input = "local-egress", `{"id":"a"}` + request := &gen.InvokeSourceOperationRequest{OrgId: org, SourceId: source.ID, Operation: declaration.Name, InputJson: input, EffectId: effect} + ctx := business.WithSourceOperationRecheck(requestCtx, func(ctx context.Context) error { + _, _, err := svc.SourceOperationAuthority(ctx, actor, org, source.ID, declaration.Name, effect, false) + return err + }) + response, err := guard.Handle(ctx, business.SourceOperationMethod, effect, request, func(ctx context.Context) (proto.Message, error) { + var response *gen.InvokeSourceOperationResponse + _, err := svc.InvokeSourceOperation(ctx, actor, org, source.ID, declaration.Name, []byte(input), func(ctx context.Context, result *business.SourceOperationResult) error { + response = &gen.InvokeSourceOperationResponse{OutputJson: string(result.Output), Receipt: &gen.SourceOperationReceipt{EffectId: effect, CommittedAt: result.CommittedAt.UTC().Format(time.RFC3339Nano), Status: "committed", ProviderStatus: uint32(result.ProviderStatus), OutputJson: string(result.Output)}} + return svc.RecordSourceOperationReceipt(ctx, response) + }) + return response, err + }) + require.NoError(t, err, "the listed stub must be reachable for both the token exchange and the operation call") + require.Equal(t, "{}", response.(*gen.InvokeSourceOperationResponse).OutputJson) + require.EqualValues(t, 1, tokenCalls.Load(), "one client_credentials exchange") + require.EqualValues(t, 1, apiCalls.Load(), "one operation call, with the exchanged token") + + // Withdrawn, the same source cannot reach the stub again: a fresh exchange + // on a second source is refused before it is sent. + require.NoError(t, apisource.ConfigureLocalEgress("", true)) + second, err := svc.AddSource(testCtx, actor, business.AddSourceInput{OrgID: org, Provider: business.DatasourceProviderAPI, CollectionLabel: "Acme", API: config, Credential: clientSecret}) + require.NoError(t, err) + copy = *second + _, err = svc.ResolveDatasourceTokenForTest(testCtx, ©) + require.Error(t, err) + require.EqualValues(t, 1, tokenCalls.Load(), "a withdrawn allowance must not reach the stub") +} diff --git a/module/services/accounts/code/pkg/business/webhook_security.go b/module/services/accounts/code/pkg/business/webhook_security.go index e345b6dd6..fe66e0b3a 100644 --- a/module/services/accounts/code/pkg/business/webhook_security.go +++ b/module/services/accounts/code/pkg/business/webhook_security.go @@ -155,8 +155,9 @@ func isAmbiguousNumericHost(host string) bool { } // resolvePublic resolves host and requires every answer to be on the public -// internet, as auth.IsPublicDestination judges it: the one classifier every -// tenant-directed egress in this host shares. An address literal is judged as +// internet, as auth.IsPublicDestination judges it: the classifier the host's +// guarded egress paths share (auth/public_address.go lists them, and the OIDC +// clients that have no guard). An address literal is judged as // written, zone included, and only unmapped once it has been admitted. func (p *WebhookEndpointPolicy) resolvePublic(ctx context.Context, host string) ([]netip.Addr, error) { if addr, err := netip.ParseAddr(host); err == nil { diff --git a/module/services/accounts/code/pkg/datasource/apisource/client.go b/module/services/accounts/code/pkg/datasource/apisource/client.go index b22dae2a5..f3eade36e 100644 --- a/module/services/accounts/code/pkg/datasource/apisource/client.go +++ b/module/services/accounts/code/pkg/datasource/apisource/client.go @@ -10,8 +10,10 @@ // on the initial request and on any redirect, checked against the exact IP // being connected to (not the hostname) so DNS rebinding cannot slip past the // check. "Not on the public internet" is auth.IsPublicDestination, read from -// the IANA special-purpose registries: the one classifier every tenant-directed -// egress in this host answers through. +// the IANA special-purpose registries: the classifier the host's guarded egress +// paths share (auth/public_address.go lists them). +// The one exception is a local runtime's explicit list of loopback addresses +// (local_egress.go), refused at boot anywhere else. package apisource import ( @@ -93,7 +95,9 @@ func New(cfg Config, credential string) *Client { // guardedClient builds an HTTP client that refuses connections to non-public // addresses, on the initial request and every redirect. Both the fetch client -// and the OAuth token exchange dial tenant-supplied hosts, so both use it. +// and the OAuth token exchange dial tenant-supplied hosts, so both use it. A +// local runtime's explicit allowance (local_egress.go) is consulted first and +// admits only the exact addresses it lists. func guardedClient() *http.Client { transport := &http.Transport{ // These clients make a single exchange. An idle connection is neither @@ -102,7 +106,7 @@ func guardedClient() *http.Client { DisableKeepAlives: true, DialContext: (&net.Dialer{ Timeout: 10 * time.Second, - Control: dialControl, + Control: withLocalEgress(dialControl), }).DialContext, } return &http.Client{ diff --git a/module/services/accounts/code/pkg/datasource/apisource/local_egress.go b/module/services/accounts/code/pkg/datasource/apisource/local_egress.go new file mode 100644 index 000000000..c10f44c0d --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/apisource/local_egress.go @@ -0,0 +1,122 @@ +package apisource + +import ( + "errors" + "fmt" + "net/netip" + "strings" + "sync/atomic" + "syscall" +) + +// The local egress allowance: how a composition booted on a developer's machine +// reaches a stub OAuth provider and API running beside it (issue #1087), and +// why no deployed composition can use it. +// +// guardDial refuses every address auth.IsPublicDestinationDialAddress does not +// classify as public. That is right for every deployed host, and it also means +// a local host cannot reach an httptest-style provider on its own machine. The +// allowance is a separate admission in front of that guard, not a change to it: +// an address it does not list is judged by guardDial, and so by the shared +// classifier, exactly as before. The public-address policy is reviewable on its +// own, and this file adds nothing to what it refuses or admits. +// +// What it admits, and only that: +// +// - entries the composition lists explicitly, as `ip:port` — one loopback IP +// literal and one port. No hostname (the guard sees the resolved IP, so the +// entry is written as what is dialed), no range, no CIDR, no wildcard, no +// zone, no IPv4-mapped spelling, no port 0; +// - loopback addresses only. A stub "beside" a local host is on the same +// machine, and loopback is the one address class that never leaves the +// dialing process's own network namespace. Even a host that somehow passed +// the environment check below could reach only its own namespace — never +// the metadata service, the cluster network, or another tenant. +// +// It is refused at boot, rather than ignored, outside a local runtime: a +// deployed composition that sets it is a composition whose operator believes +// the host can reach something it cannot, and the place to say so is startup. +// Whether the runtime is local is the caller's answer (see work.go, +// configureLocalEgress); this package only refuses a non-empty allowance it is +// told is not local. + +// ErrLocalEgressOutsideLocal reports a non-empty allowance on a runtime that +// is not local. It refuses startup. +var ErrLocalEgressOutsideLocal = errors.New("apisource: the local egress allowance is only permitted in a local runtime") + +// ErrInvalidLocalEgress reports an allowance entry that is not exactly one +// loopback IP literal and one port. +var ErrInvalidLocalEgress = errors.New("apisource: invalid local egress entry") + +// localEgress is the process-wide allowance, installed once at boot by +// ConfigureLocalEgress. Nil — every deployed host — admits nothing beyond +// guardDial. +var localEgress atomic.Pointer[map[netip.AddrPort]struct{}] + +// ConfigureLocalEgress parses value, a comma- or whitespace-separated list of +// `ip:port` entries, and installs it as the allowance every callable-source +// client built afterwards consults. An empty value installs no allowance. +// A non-empty value is refused unless local is true, and a malformed entry is +// refused whole: a partially-understood allowance is not installed. +func ConfigureLocalEgress(value string, local bool) error { + allowed, err := parseLocalEgress(value) + if err != nil { + return err + } + if len(allowed) > 0 && !local { + return ErrLocalEgressOutsideLocal + } + if len(allowed) == 0 { + localEgress.Store(nil) + return nil + } + localEgress.Store(&allowed) + return nil +} + +func parseLocalEgress(value string) (map[netip.AddrPort]struct{}, error) { + entries := strings.FieldsFunc(value, func(r rune) bool { + return r == ',' || r == ' ' || r == '\t' || r == '\n' || r == '\r' + }) + allowed := make(map[netip.AddrPort]struct{}, len(entries)) + for _, entry := range entries { + target, err := netip.ParseAddrPort(entry) + switch { + case err != nil: + return nil, fmt.Errorf("%w %q: write exactly one loopback IP literal and a port, as ip:port with an IPv6 literal in brackets", ErrInvalidLocalEgress, entry) + case target.String() != entry: + // A non-canonical spelling (a leading-zero port, an uncompressed + // IPv6 literal) is refused rather than normalised, so the entry an + // operator reads is the address that is admitted. + return nil, fmt.Errorf("%w %q: write it as %s", ErrInvalidLocalEgress, entry, target) + case target.Addr().Zone() != "" || target.Addr().Is4In6(): + return nil, fmt.Errorf("%w %q: no zone or IPv4-mapped spelling", ErrInvalidLocalEgress, entry) + case !target.Addr().IsLoopback(): + return nil, fmt.Errorf("%w %q: only a loopback address may be listed", ErrInvalidLocalEgress, entry) + case target.Port() == 0: + return nil, fmt.Errorf("%w %q: name the port the stub listens on", ErrInvalidLocalEgress, entry) + } + allowed[target] = struct{}{} + } + return allowed, nil +} + +// withLocalEgress wraps next, the dial guard, with the installed allowance: an +// address it lists exactly is admitted, and every other address is handed to +// next unchanged, so the public-address classifier still judges it. With no +// allowance installed it returns next itself. +func withLocalEgress(next func(network, address string, c syscall.RawConn) error) func(network, address string, c syscall.RawConn) error { + allowed := localEgress.Load() + if allowed == nil { + return next + } + return func(network, address string, c syscall.RawConn) error { + if target, err := netip.ParseAddrPort(address); err == nil { + target = netip.AddrPortFrom(target.Addr().Unmap(), target.Port()) + if _, ok := (*allowed)[target]; ok { + return nil + } + } + return next(network, address, c) + } +} diff --git a/module/services/accounts/code/pkg/datasource/apisource/local_egress_test.go b/module/services/accounts/code/pkg/datasource/apisource/local_egress_test.go new file mode 100644 index 000000000..23d7ddef8 --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/apisource/local_egress_test.go @@ -0,0 +1,225 @@ +package apisource + +import ( + "context" + "encoding/json" + "errors" + "net" + "net/http" + "net/http/httptest" + "net/url" + "reflect" + "strconv" + "syscall" + "testing" +) + +// installLocalEgress installs value as a LOCAL runtime's allowance for one test +// and withdraws it afterwards, so no other test in the package dials through it. +func installLocalEgress(t *testing.T, value string) { + t.Helper() + if err := ConfigureLocalEgress(value, true); err != nil { + t.Fatalf("ConfigureLocalEgress(%q, local) = %v", value, err) + } + t.Cleanup(func() { _ = ConfigureLocalEgress("", false) }) +} + +// Every spelling that is not exactly one loopback IP literal and one port is +// refused whole. Each case names the property it holds, so dropping one check +// fails a case named after it. +func TestLocalEgress_RefusesAnythingButAnExplicitLoopbackAddressAndPort(t *testing.T) { + for name, entry := range map[string]string{ + "no port": "127.0.0.1", + "port zero": "127.0.0.1:0", + "wildcard host": "*:18080", + "wildcard port": "127.0.0.1:*", + "port range": "127.0.0.1:18080-18090", + "cidr": "127.0.0.0/8:18080", + "hostname": "localhost:18080", + "unspecified v4": "0.0.0.0:18080", + "unspecified v6": "[::]:18080", + "private": "10.0.0.1:18080", + "link-local metadata": "169.254.169.254:80", + "public": "93.184.216.34:443", + "ipv4-mapped loopback": "[::ffff:127.0.0.1]:18080", + "zoned loopback": "[::1%lo0]:18080", + "leading-zero port": "127.0.0.1:018080", + "unbracketed v6": "::1:18080", + "valid entry beside invalid": "127.0.0.1:18080,10.0.0.1:18080", + } { + t.Run(name, func(t *testing.T) { + localEgress.Store(nil) + t.Cleanup(func() { localEgress.Store(nil) }) + err := ConfigureLocalEgress(entry, true) + if !errors.Is(err, ErrInvalidLocalEgress) { + t.Fatalf("ConfigureLocalEgress(%q) = %v, want ErrInvalidLocalEgress", entry, err) + } + if localEgress.Load() != nil { + t.Fatal("a refused allowance must install nothing") + } + }) + } + for _, entry := range []string{"127.0.0.1:18080", "[::1]:18080", "127.0.0.2:80", "127.0.0.1:18080, [::1]:18081"} { + installLocalEgress(t, entry) + } +} + +// The allowance is refused, not ignored, on a runtime that is not local; an +// empty value is the default everywhere and is no allowance at all. +func TestLocalEgress_RefusedOutsideALocalRuntime(t *testing.T) { + t.Cleanup(func() { _ = ConfigureLocalEgress("", false) }) + if err := ConfigureLocalEgress("127.0.0.1:18080", false); !errors.Is(err, ErrLocalEgressOutsideLocal) { + t.Fatalf("ConfigureLocalEgress outside local = %v, want ErrLocalEgressOutsideLocal", err) + } + if localEgress.Load() != nil { + t.Fatal("an allowance refused outside local must install nothing") + } + for _, empty := range []string{"", " ", " , "} { + if err := ConfigureLocalEgress(empty, false); err != nil { + t.Fatalf("empty allowance %q outside local = %v, want nil", empty, err) + } + if localEgress.Load() != nil { + t.Fatalf("empty allowance %q installed something", empty) + } + } +} + +// The allowance admits the exact address and port it lists and nothing else: +// not the next port, not a neighbouring loopback address, not the other loopback +// family, and never a non-public address the public guard refuses. +func TestLocalEgress_AdmitsOnlyTheListedAddress(t *testing.T) { + installLocalEgress(t, "127.0.0.1:18080") + control := withLocalEgress(dialControl) + if err := control("tcp", "127.0.0.1:18080", nil); err != nil { + t.Fatalf("listed address refused: %v", err) + } + if err := control("tcp6", "[::ffff:127.0.0.1]:18080", nil); err != nil { + t.Fatalf("the listed address dialed in its mapped form refused: %v", err) + } + for _, address := range []string{ + "127.0.0.1:18081", + "127.0.0.2:18080", + "[::1]:18080", + "10.0.0.1:18080", + "169.254.169.254:18080", + "192.168.1.10:18080", + "100.100.100.200:18080", + "[64:ff9b::a9fe:a9fe]:18080", + "not-an-address", + } { + if err := control("tcp", address, nil); !errors.Is(err, ErrBlockedAddress) { + t.Errorf("unlisted %q = %v, want ErrBlockedAddress", address, err) + } + } + // A public address is still judged by the public guard alone: the allowance + // neither admits nor refuses anything it does not list. + if err := control("tcp", "93.184.216.34:443", nil); err != nil { + t.Errorf("public address = %v, want the public guard's answer (allowed)", err) + } +} + +// With no allowance installed the guarded client is exactly guardDial. +func TestLocalEgress_NoAllowanceIsThePublicGuard(t *testing.T) { + if err := ConfigureLocalEgress("", true); err != nil { + t.Fatal(err) + } + next := func(string, string, syscall.RawConn) error { return ErrBlockedAddress } + if reflect.ValueOf(withLocalEgress(next)).Pointer() != reflect.ValueOf(next).Pointer() { + t.Fatal("with no allowance the dial control must be the guard it wraps, itself") + } + if err := withLocalEgress(dialControl)("tcp", "127.0.0.1:18080", nil); !errors.Is(err, ErrBlockedAddress) { + t.Fatalf("loopback without an allowance = %v, want ErrBlockedAddress", err) + } +} + +// Every address the allowance does not list is handed to the guard it wraps, +// unchanged: the shared classifier, not this file, judges it. Only the exact +// listed entry skips the guard. +func TestLocalEgress_EveryUnlistedAddressReachesTheGuard(t *testing.T) { + installLocalEgress(t, "127.0.0.1:18080") + var judged []string + control := withLocalEgress(func(_, address string, _ syscall.RawConn) error { + judged = append(judged, address) + return ErrBlockedAddress + }) + unlisted := []string{"127.0.0.1:18081", "[::1]:18080", "100.100.100.200:18080", "93.184.216.34:443", "not-an-address"} + for _, address := range unlisted { + _ = control("tcp", address, nil) + } + if err := control("tcp", "127.0.0.1:18080", nil); err != nil { + t.Fatalf("listed address = %v, want admitted", err) + } + if !reflect.DeepEqual(judged, unlisted) { + t.Fatalf("the guard judged %q, want exactly the unlisted addresses %q", judged, unlisted) + } +} + +// The issue's done-criterion at this layer: an httptest provider on loopback, +// reached by the REAL guarded clients — the client_credentials token exchange +// and the operation call — with no transport substituted. Without the +// allowance both dials are refused; with an allowance for another port they +// still are; with the listed address both succeed. +func TestLocalEgress_ClientCredentialsReachALocalStubThroughTheRealGuard(t *testing.T) { + var tokenCalls, apiCalls int + stub := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/token": + tokenCalls++ + if err := r.ParseForm(); err != nil || r.PostForm.Get("grant_type") != "client_credentials" || + r.PostForm.Get("client_id") != "example-client" || r.PostForm.Get("client_secret") != "example-secret-value" { + http.Error(w, `{"error":"invalid_client"}`, http.StatusUnauthorized) + return + } + _ = json.NewEncoder(w).Encode(map[string]any{"access_token": "stub-access-token-0123456789", "expires_in": 3600}) + case "/v1/items": + apiCalls++ + if r.Header.Get("Authorization") != "Bearer stub-access-token-0123456789" { + http.Error(w, "", http.StatusUnauthorized) + return + } + _, _ = w.Write([]byte(`{"items":[]}`)) + default: + http.NotFound(w, r) + } + })) + defer stub.Close() + address := stub.Listener.Addr().String() + oauth := OAuth2Config{TokenURL: stub.URL + "/token"} + fetch := func(token string) error { + _, err := New(Config{BaseURL: stub.URL, ResourcePath: "/v1/items", CredentialKind: CredentialKindBearer}, token).Fetch(context.Background()) + return err + } + + _ = ConfigureLocalEgress("", true) + if _, err := ClientCredentials(context.Background(), oauth, "example-client", "example-secret-value"); !errors.Is(err, ErrBlockedAddress) { + t.Fatalf("token exchange without an allowance = %v, want ErrBlockedAddress", err) + } + if err := fetch("stub-access-token-0123456789"); !errors.Is(err, ErrBlockedAddress) { + t.Fatalf("operation call without an allowance = %v, want ErrBlockedAddress", err) + } + + host, port, _ := net.SplitHostPort(address) + other, _ := strconv.Atoi(port) + installLocalEgress(t, net.JoinHostPort(host, strconv.Itoa(other+1))) + if _, err := ClientCredentials(context.Background(), oauth, "example-client", "example-secret-value"); !errors.Is(err, ErrBlockedAddress) { + t.Fatalf("token exchange with another port listed = %v, want ErrBlockedAddress", err) + } + if tokenCalls != 0 || apiCalls != 0 { + t.Fatalf("a refused dial reached the stub (token %d, api %d)", tokenCalls, apiCalls) + } + + installLocalEgress(t, address) + token, err := ClientCredentials(context.Background(), oauth, "example-client", "example-secret-value") + if err != nil { + t.Fatalf("token exchange with the stub listed = %v", err) + } + if err := fetch(token.AccessToken); err != nil { + t.Fatalf("operation call with the stub listed = %v", err) + } + if tokenCalls != 1 || apiCalls != 1 { + t.Fatalf("stub calls (token %d, api %d), want one of each", tokenCalls, apiCalls) + } + if u, _ := url.Parse(stub.URL); u.Hostname() != "127.0.0.1" { + t.Fatalf("httptest stub is on %s, not loopback IPv4", u.Hostname()) + } +} diff --git a/module/services/accounts/code/pkg/datasource/crawler/client.go b/module/services/accounts/code/pkg/datasource/crawler/client.go index eb72c0559..221ee7f86 100644 --- a/module/services/accounts/code/pkg/datasource/crawler/client.go +++ b/module/services/accounts/code/pkg/datasource/crawler/client.go @@ -228,8 +228,8 @@ var dialGuard = guardDial // public internet. It runs after DNS resolution with the concrete IP:port being // dialed, so it blocks the exact address the connection would reach — closing // the DNS-rebinding gap a hostname-only check leaves open. The judgement is -// auth.IsPublicDestinationDialAddress, the host's one classifier for every -// tenant-directed egress; an address it cannot parse is refused. +// auth.IsPublicDestinationDialAddress, the classifier the host's guarded +// egress paths share; an address it cannot parse is refused. func guardDial(_, address string, _ syscall.RawConn) error { if !auth.IsPublicDestinationDialAddress(address) { return ErrBlockedAddress diff --git a/module/services/accounts/code/pkg/datasource/objectstore/client.go b/module/services/accounts/code/pkg/datasource/objectstore/client.go index b1188de0f..8d85fbba4 100644 --- a/module/services/accounts/code/pkg/datasource/objectstore/client.go +++ b/module/services/accounts/code/pkg/datasource/objectstore/client.go @@ -103,8 +103,8 @@ var dialGuard = guardDial // public internet. It runs after DNS resolution with the concrete IP:port being // dialed, so it blocks the exact address the connection would reach — closing // the DNS-rebinding gap a hostname-only check leaves open. The judgement is -// auth.IsPublicDestinationDialAddress, the host's one classifier for every -// tenant-directed egress; an address it cannot parse is refused. +// auth.IsPublicDestinationDialAddress, the classifier the host's guarded +// egress paths share; an address it cannot parse is refused. func guardDial(_, address string, _ syscall.RawConn) error { if !auth.IsPublicDestinationDialAddress(address) { return ErrBlockedAddress diff --git a/module/services/accounts/code/pkg/infra/kubernetes_in_cluster_test.go b/module/services/accounts/code/pkg/infra/kubernetes_in_cluster_test.go new file mode 100644 index 000000000..c384bbd12 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/kubernetes_in_cluster_test.go @@ -0,0 +1,66 @@ +package infra + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +// Each fact the kubelet puts in a pod answers "in a cluster" on its own. A pod +// that mounts a service-account token carries the projected files whatever its +// environment says, so the file branch must hold without the variable. +func TestInClusterAnswersYesOnEachKubeletFactAlone(t *testing.T) { + dir := t.TempDir() + token := filepath.Join(dir, "token") + ca := filepath.Join(dir, "ca.crt") + absentToken := filepath.Join(dir, "absent-token") + absentCA := filepath.Join(dir, "absent-ca.crt") + require.NoError(t, os.WriteFile(token, []byte("projected"), 0o600)) + require.NoError(t, os.WriteFile(ca, []byte("projected"), 0o600)) + + t.Run("service host variable", func(t *testing.T) { + require.True(t, inCluster("10.96.0.1", absentToken, absentCA)) + }) + t.Run("projected token only", func(t *testing.T) { + require.True(t, inCluster("", token, absentCA)) + }) + t.Run("projected CA only", func(t *testing.T) { + require.True(t, inCluster("", absentToken, ca)) + }) + // A path that cannot be checked is not evidence of absence: callers refuse + // a local-only setting on yes, so the doubt goes to yes. + t.Run("uncheckable path", func(t *testing.T) { + require.True(t, inCluster("", filepath.Join(token, "under-a-file"), absentCA)) + }) + // The control: with none of them, the answer is no. Without it an + // inCluster that always said yes would pass every case above. + t.Run("none", func(t *testing.T) { + require.False(t, inCluster("", absentToken, absentCA)) + require.False(t, inCluster(" ", absentToken, absentCA), "a blank variable is not an address") + }) +} + +// InCluster wires the real carriers into inCluster: the kubelet's variable and +// its two fixed projected paths, each of which answers on its own. +func TestInClusterReadsTheKubeletCarriers(t *testing.T) { + require.Equal(t, []string{ + "/var/run/secrets/kubernetes.io/serviceaccount/token", + "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt", + }, inClusterProjectedPaths) + + t.Setenv("KUBERNETES_SERVICE_HOST", "10.96.0.1") + require.True(t, InCluster()) + + dir := t.TempDir() + token := filepath.Join(dir, "token") + require.NoError(t, os.WriteFile(token, []byte("projected"), 0o600)) + saved := inClusterProjectedPaths + t.Cleanup(func() { inClusterProjectedPaths = saved }) + t.Setenv("KUBERNETES_SERVICE_HOST", "") + inClusterProjectedPaths = []string{token, filepath.Join(dir, "absent-ca.crt")} + require.True(t, InCluster(), "a projected token with no variable is still a pod") + inClusterProjectedPaths = []string{filepath.Join(dir, "absent-token"), filepath.Join(dir, "absent-ca.crt")} + require.False(t, InCluster()) +} diff --git a/module/services/accounts/code/pkg/infra/kubernetes_token_review.go b/module/services/accounts/code/pkg/infra/kubernetes_token_review.go index 1d79dd379..247aaba0a 100644 --- a/module/services/accounts/code/pkg/infra/kubernetes_token_review.go +++ b/module/services/accounts/code/pkg/infra/kubernetes_token_review.go @@ -9,6 +9,7 @@ import ( "encoding/json" "errors" "fmt" + "io/fs" "net/http" "os" "strings" @@ -50,6 +51,47 @@ const ( inClusterNamespacePath = "/var/run/secrets/kubernetes.io/serviceaccount/namespace" ) +// InCluster reports whether this process carries any of what the kubelet puts +// in a pod: the API server's address in KUBERNETES_SERVICE_HOST, or the +// projected service-account token or CA bundle at their fixed paths. Any one of +// them is enough to answer yes, and a path that cannot be checked counts as +// present. +// +// It is the host's answer to "is this a pod", read from the same facts +// NewKubernetesClient builds its identity from, and it errs toward yes on +// purpose: callers use it to REFUSE a local-only setting. +// +// What defeating it takes, precisely. The kubelet injects +// KUBERNETES_SERVICE_HOST into every container, and no configuration value can +// set or clear it: workspace values arrive under their own prefixed carriers. +// The projected files are present only when the pod mounts a service-account +// token, and a Codefly-rendered service pod does not by default +// (automountServiceAccountToken: false). So in a default pod the variable is the +// only fact, and an explicit container-env override in the pod spec clears it. +// That takes control of the pod spec, not of a configuration value. +func InCluster() bool { + return inCluster(os.Getenv("KUBERNETES_SERVICE_HOST"), inClusterProjectedPaths...) +} + +// inClusterProjectedPaths are the kubelet's fixed paths InCluster checks. A +// variable only so a test can point it at files it controls; nothing assigns +// it outside a test. +var inClusterProjectedPaths = []string{inClusterTokenPath, inClusterCAPath} + +// inCluster is InCluster over facts it has already located, so a test drives +// each branch through the same code rather than a lookalike. +func inCluster(serviceHost string, projected ...string) bool { + if strings.TrimSpace(serviceHost) != "" { + return true + } + for _, path := range projected { + if _, err := os.Stat(path); err == nil || !errors.Is(err, fs.ErrNotExist) { + return true + } + } + return false +} + // KubernetesClient is the host's read-only window onto the API server. type KubernetesClient struct { endpoint string diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index dc1d1eb62..da2fd639b 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -16,6 +16,7 @@ import ( pgbilling "accounts/pkg/billing/pg" "accounts/pkg/business" "accounts/pkg/datasource" + "accounts/pkg/datasource/apisource" "accounts/pkg/email" "accounts/pkg/eventcatalog" "accounts/pkg/githubconnector" @@ -2651,6 +2652,11 @@ func loadSigningKey( // deployment can still be fixed rather than discovered from a crash loop — and // so a test can reach all of them with nothing running. func requireStartupConfiguration(ctx context.Context, isLocal bool) error { + // First, so a deployed composition that sets the allowance is told so before + // any other refusal can answer in its place. + if err := configureLocalEgress(isLocal, infra.InCluster()); err != nil { + return err + } selection, err := keyservice.Selection(ctx, isLocal) if err != nil { return err @@ -2664,6 +2670,56 @@ func requireStartupConfiguration(ctx context.Context, isLocal bool) error { return requireApplicationBaseURL(isLocal) } +// configureLocalEgress installs the callable-source local egress allowance +// (`datasource-oauth/local_egress`, apisource/local_egress.go), or refuses to +// start when it is set on a runtime that is not local. +// +// "Local" here is two facts the host already has, and both must hold: +// +// - the Codefly environment is local (codefly.IsLocal) — the one environment +// identity every other local-only exemption in this file reads; +// - the process is not in a Kubernetes pod (infra.InCluster) — the kubelet's +// own projection, which the solution host already treats as its in-cluster +// identity. +// +// The environment alone is one variable, delivered by whatever renders the +// deployment, so a deployment could name itself local. The pod fact is put +// there by the kubelet, and no configuration value can set or clear it. A +// Codefly-rendered pod mounts no service-account token by default, so there the +// fact is KUBERNETES_SERVICE_HOST alone: defeating both checks takes the pod +// spec — CODEFLY__ENVIRONMENT=local and an explicit override clearing that +// variable — never a configuration value. A deployment that can name itself +// local has already lifted the refusals of the dev identity provider and the +// ephemeral signing key, both worse than this allowance. A local runtime started by +// `codefly run` has neither fact, so the allowance works exactly where it is +// meant to: on a developer's machine. A local Kubernetes cluster is refused +// too, by the same rule; it is not where a stub provider runs "beside" the host. +// +// Even past both checks the allowance admits only explicit loopback ip:port +// entries, so it never reaches beyond the host's own network namespace. In a +// pod that namespace is shared with every container in the pod, sidecars +// included (a mesh proxy's admin port is on loopback); on a developer's machine +// it is every local listener, chosen by the developer who lists the port. +func configureLocalEgress(isLocal, inCluster bool) error { + value := workspaceEnv("datasource-oauth", "local_egress") + err := apisource.ConfigureLocalEgress(value, isLocal && !inCluster) + switch { + case err == nil: + if strings.TrimSpace(value) != "" { + wool.Get(context.Background()).In("configureLocalEgress").Warn( + "callable sources may dial these loopback addresses past the public-address guard (local runtime only)", + wool.Field("datasource-oauth/local_egress", value)) + } + return nil + case errors.Is(err, apisource.ErrLocalEgressOutsideLocal): + return fmt.Errorf("datasource-oauth/local_egress is set, but this runtime is not local (local environment: %t, in a Kubernetes pod: %t): "+ + "remove it from this composition. The allowance exists only so a host booted on a developer's machine can reach a stub provider beside it; "+ + "a deployed host reaches public addresses only: %w", isLocal, inCluster, err) + default: + return fmt.Errorf("datasource-oauth/local_egress: %w", err) + } +} + // requireApplicationBaseURL refuses to start a deployed runtime without the // origin it treats as its own. // diff --git a/module/services/accounts/code/work_local_egress_test.go b/module/services/accounts/code/work_local_egress_test.go new file mode 100644 index 000000000..b37d0689b --- /dev/null +++ b/module/services/accounts/code/work_local_egress_test.go @@ -0,0 +1,94 @@ +package main + +import ( + "context" + "errors" + "testing" + + "accounts/pkg/datasource/apisource" + "accounts/pkg/infra" + + "github.com/stretchr/testify/require" +) + +// localEgressValue sets `datasource-oauth/local_egress` on both carriers a +// workspace-group read consults, and withdraws any allowance a test installed. +func localEgressValue(t *testing.T, value string) { + t.Helper() + t.Setenv("CODEFLY__WORKSPACE_CONFIGURATION__DATASOURCE_OAUTH__LOCAL_EGRESS", value) + t.Setenv("local_egress", value) + t.Cleanup(func() { _ = apisource.ConfigureLocalEgress("", false) }) +} + +// #1087: a deployed composition that sets the allowance refuses to START, through +// the same function main calls, before any other startup refusal can answer in +// its place. +func TestStartupRefusesLocalEgressOutsideTheLocalEnvironment(t *testing.T) { + localEgressValue(t, "127.0.0.1:18080") + err := requireStartupConfiguration(t.Context(), false) + require.ErrorIs(t, err, apisource.ErrLocalEgressOutsideLocal) + require.Contains(t, err.Error(), "datasource-oauth/local_egress") + + // The environment fact on its own, independent of whether the machine + // running this test is itself a pod. + require.ErrorIs(t, configureLocalEgress(false, false), apisource.ErrLocalEgressOutsideLocal) +} + +// The refusal through doWork itself, the function main runs, and FIRST: a hosted +// boot with no key-service backend refuses on that too, so if the allowance's +// check stopped running before every other startup check, the key-service +// refusal would answer here instead and this test would fail. Nothing is +// running, so reaching a dependency would fail too. +func TestDoWorkRefusesLocalEgressBeforeAnyOtherStartupCheck(t *testing.T) { + clearAuthProviderEnvironment(t) + clearVaultBinding(t) + localEgressValue(t, "127.0.0.1:18080") + + t.Setenv("CODEFLY__ENVIRONMENT", "hosted") + _, err := doWork(context.Background()) + require.ErrorIs(t, err, apisource.ErrLocalEgressOutsideLocal) + require.Contains(t, err.Error(), "datasource-oauth/local_egress") + + // A pod that names itself local is refused by the same boot path. + t.Setenv("CODEFLY__ENVIRONMENT", "local") + t.Setenv("KUBERNETES_SERVICE_HOST", "10.96.0.1") + _, err = doWork(context.Background()) + require.ErrorIs(t, err, apisource.ErrLocalEgressOutsideLocal) +} + +// A local Codefly environment is not enough on its own: the environment is one +// variable whoever renders a deployment sets. A process in a Kubernetes pod is +// refused even when it names itself local. +func TestLocalEgressRefusedInAPodThatNamesItselfLocal(t *testing.T) { + localEgressValue(t, "127.0.0.1:18080") + require.ErrorIs(t, configureLocalEgress(true, true), apisource.ErrLocalEgressOutsideLocal) + + // And through main's path, with the pod fact read from the kubelet's own + // variable rather than handed in. + t.Setenv("KUBERNETES_SERVICE_HOST", "10.96.0.1") + require.True(t, infra.InCluster()) + require.ErrorIs(t, requireStartupConfiguration(t.Context(), true), apisource.ErrLocalEgressOutsideLocal) +} + +// The positive control: a local runtime outside any pod installs the allowance. +// Without it, a configureLocalEgress that refused everything would pass both +// tests above. +func TestLocalEgressInstalledOnALocalRuntime(t *testing.T) { + localEgressValue(t, "127.0.0.1:18080") + require.NoError(t, configureLocalEgress(true, false)) +} + +// The default is empty everywhere, so a deployed host with no allowance starts +// exactly as before. +func TestLocalEgressEmptyIsAcceptedEverywhere(t *testing.T) { + localEgressValue(t, "") + require.NoError(t, configureLocalEgress(false, true)) +} + +// A malformed entry refuses even a local start, naming the key. +func TestLocalEgressMalformedRefusesALocalStart(t *testing.T) { + localEgressValue(t, "localhost:*") + err := configureLocalEgress(true, false) + require.True(t, errors.Is(err, apisource.ErrInvalidLocalEgress), "got %v", err) + require.Contains(t, err.Error(), "datasource-oauth/local_egress") +}