diff --git a/AGENTS.md b/AGENTS.md index 2fe1d66..5592950 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -444,10 +444,33 @@ embedding it. answer is recorded **where the answer lands** rather than by whichever caller was still waiting — the ask is detached, so it can finish with nobody listening. +- **The mint reports the dynamic half of the registration; nothing else + does.** `upstream` (where the gateway dials this runtime) and `manifest` (the + frontend manifest it serves) travel in the mint body, recorded by the host + under the identity the projected token proves. This is not the registration + this runtime removed: it adds no request, no timer and no credential, and it + cannot make a workload present — the host completes only a declaration its + own delivery already applied. `upstream` is the **advertised** address + (`CODEFLY__SELF_ENDPOINT__…`, through core's accessor), never the listen + address a cell localizes to `localhost`; a present carrier is never read as + absent, and a deployed runtime context refuses a loopback origin at boot. + `brokerMintBodyMaxBytes` bounds the request at boot, under what the host + reads, so the bound that fires is the one that can name the manifest. +- **`412` is "admitted, installed nowhere", and it is a state, not an error.** + The host issues this execution's credential only after the install step is + acknowledged, so the step must run without it — it acts under the capability + the host carries in — and `/health` must answer 200, both from what the + controller already recorded (`uninstalled`) without asking again. + `awaitInstallation` is the one asker, through `askOnce` like every other + ask: an ask made around the controller left a terminal answer latched in the + source and reported by nobody. A viewer's route still answers 503. Do not + widen the exemption to any other unavailable state: a host that cannot be + reached is not an uninstalled solution. - **The execution mint follows the host's module broker contract natively.** `saas-sdk-go/moduleauthority` at `217bf9fe3f10` is the wire reference; its - capability dependency is incompatible with this runtime. Send exactly - `prefix: Manifest.ID` to `/modules/_work-context`, with the projected token + capability dependency is incompatible with this runtime. Send + `prefix: Manifest.ID` — and beside it only `upstream` and `manifest`, below — + to `/modules/_work-context`, with the projected token as bearer and `X-Codefly-Internal-Token` from the declared workspace secret `module-authority/INTERNAL_TOKEN`. Resolve it in `loadConfig`, refuse absence in `validate()`, and never resurrect a legacy secret reader. The projection diff --git a/README.md b/README.md index 3aaaf7c..c2db60d 100644 --- a/README.md +++ b/README.md @@ -65,10 +65,37 @@ Four things follow, and each is a boot failure rather than a degraded run: Execution minting uses the host's module broker protocol natively. The reference is `saas-sdk-go/moduleauthority` at `217bf9fe3f10`; importing that package is incompatible with this runtime's SDK capability API, so no dependency was added. -The request is `POST /modules/_work-context`, with exactly +The request is `POST /modules/_work-context`, with `{"prefix":""}`, `Authorization: Bearer `, and `X-Codefly-Internal-Token`. The prefix is the declared solution ID, separate from -the frozen SPIFFE principal. The answer carries `token`, `expiresAt`, +the frozen SPIFFE principal. Beside the prefix the body reports the dynamic half +of this runtime's registration, since nothing self-registers: `upstream`, the +https origin the gateway dials to reach this runtime, and `manifest`, the +frontend manifest this runtime serves. The host records both under the identity +the projected token proves and reaches the origin only under this workload's +SVID. The whole request is bounded at 64 KiB, and a manifest that takes it over +is refused at boot, naming the manifest. + +`upstream` is the address Codefly **advertises** for this service's http +endpoint — the `CODEFLY__SELF_ENDPOINT______HTTP__HTTP` carrier +— and never the address it listens on. A rendered cell localizes a service's own +`CODEFLY__ENDPOINT__` carrier to `localhost`, so reporting that would have the +gateway proxy the solution to the gateway's own loopback. With no carrier, a +process on a developer machine reports the loopback port it actually binds +(`PORT` included). In a deployed runtime context a loopback origin, a missing +carrier and a carrier that does not hold exactly a host and a port each fail the +boot; a present carrier is never treated as absent. + +A host that admits this workload but has not been asked to install it anywhere +answers `412`. That is not a refusal. The runtime serves holding no credential: +`/health` answers `200`, and the [install step](#install-step) runs, because it +acts under the capability the host carries in and the host issues this +execution's credential only after that step is acknowledged. Neither asks the +host again. The runtime asks every 20 seconds (jittered), and at once — then a +few times in quick succession — after it acknowledges an install step, and holds +the first credential issued. A viewer's route still answers `503` until then. A +`401`/`403` +stays what it was: a judgement, which fails the boot and is never retried. The answer carries `token`, `expiresAt`, `principalId`, and `tenant`. Core alone reads the returned capability, through its structural `Decode` (named `Inspect` before core v0.15); the runtime does not implement a signed encoding or verify an issuer's authority, and that read diff --git a/boot_test.go b/boot_test.go index dc134f0..30f7648 100644 --- a/boot_test.go +++ b/boot_test.go @@ -97,6 +97,10 @@ type hostMint struct { // as the path it would have used. callers []string paths []string + // answer, when set, decides each workload mint in place of status and + // recoverAfter: the status to answer, or zero to issue. It is called on + // the host's goroutines, so what it reads must be safe to read there. + answer func() int // recoverAfter, when positive, is how many times status is answered before // the host starts issuing: an issuer catching up with a presence // generation that has just been applied. @@ -360,7 +364,7 @@ func (m *hostMint) serve(w http.ResponseWriter, r *http.Request) { } if r.URL.Path == "/modules/_work-context" { var body map[string]string - if r.Method != http.MethodPost || r.Header.Get("Content-Type") != "application/json" || r.Header.Get("Accept") != "application/json" || r.Header.Get("X-Codefly-Internal-Token") != "broker-perimeter-token" || !strings.HasPrefix(r.Header.Get("Authorization"), "Bearer ") || json.NewDecoder(r.Body).Decode(&body) != nil || len(body) != 1 || body["prefix"] != testSolutionID { + if r.Method != http.MethodPost || r.Header.Get("Content-Type") != "application/json" || r.Header.Get("Accept") != "application/json" || r.Header.Get("X-Codefly-Internal-Token") != "broker-perimeter-token" || !strings.HasPrefix(r.Header.Get("Authorization"), "Bearer ") || json.NewDecoder(r.Body).Decode(&body) != nil || body["prefix"] != testSolutionID || !onlyBrokerBodyKeys(body) { http.Error(w, "invalid broker request", http.StatusBadRequest) return } @@ -368,7 +372,12 @@ func (m *hostMint) serve(w http.ResponseWriter, r *http.Request) { atomic.AddInt64(&m.mints, 1) // recoverAfter 0 means this host never issues; a positive one means it // issues once that many asks have been answered with status. - if m.status != 0 && (m.recoverAfter == 0 || atomic.LoadInt64(&m.mints) <= m.recoverAfter) { + if m.answer != nil { + if status := m.answer(); status != 0 { + w.WriteHeader(status) + return + } + } else if m.status != 0 && (m.recoverAfter == 0 || atomic.LoadInt64(&m.mints) <= m.recoverAfter) { w.WriteHeader(m.status) return } diff --git a/broker_body_test.go b/broker_body_test.go new file mode 100644 index 0000000..19bf793 --- /dev/null +++ b/broker_body_test.go @@ -0,0 +1,103 @@ +package solution + +import ( + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" +) + +// The mint request carries the declared prefix and, beside it, the dynamic half +// of this runtime's registration: where it serves and the manifest it serves. +// Nothing else — a field the host does not read is a field nobody reviewed. +var brokerBodyKeys = map[string]bool{"prefix": true, "upstream": true, "manifest": true} + +func onlyBrokerBodyKeys(body map[string]string) bool { + for key := range body { + if !brokerBodyKeys[key] { + return false + } + } + return true +} + +// requireBrokerBody holds a mint body to the protocol: exactly the declared +// prefix, an https origin when an upstream is reported, and this solution's own +// manifest when one is reported. +func requireBrokerBody(t *testing.T, raw []byte, prefix string) map[string]string { + t.Helper() + var body map[string]string + if err := json.Unmarshal(raw, &body); err != nil { + t.Fatalf("broker body is not a flat JSON object of strings: %s", raw) + } + if body["prefix"] != prefix || !onlyBrokerBodyKeys(body) { + t.Fatalf("broker body must carry the declared prefix and only the dynamic half beside it: %s", raw) + } + if upstream, reported := body["upstream"]; reported { + parsed, err := url.Parse(upstream) + if err != nil || parsed.Scheme != "https" || parsed.Host == "" || parsed.Port() == "" || (parsed.Path != "" && parsed.Path != "/") || parsed.RawQuery != "" || parsed.User != nil { + t.Fatalf("a reported upstream must be an https origin and nothing more: %q", upstream) + } + } + if manifest, reported := body["manifest"]; reported { + var document struct { + ID string `json:"id"` + } + if err := json.Unmarshal([]byte(manifest), &document); err != nil || document.ID != prefix { + t.Fatalf("a reported manifest must be this solution's own: %s", manifest) + } + } + return body +} + +// The host records a solution's backend origin and frontend manifest from the +// mint, under the identity the projected token proves, since nothing +// self-registers. A runtime that knows where it serves therefore says so, and a +// runtime that does not — no resolved endpoint — reports no origin rather than +// a guessed one. +func TestTheMintReportsWhereThisRuntimeServesAndItsManifest(t *testing.T) { + for _, tc := range []struct { + name, upstream string + }{ + {"with a resolved endpoint", "https://localhost:22631"}, + {"without one", ""}, + } { + t.Run(tc.name, func(t *testing.T) { + s, host, _ := brokerFixture(t) + s.cfg.upstream = tc.upstream + options := s.mintOptions("accounts", func() (*x509.CertPool, error) { return x509.NewCertPool(), nil }, func(*tls.CertificateRequestInfo) (*tls.Certificate, error) { return &tls.Certificate{}, nil }) + options.Authority = fixedAuthority{audience: testAudience} + source, err := newBrokerCredentialSource(options) + if err != nil { + t.Fatal(err) + } + var sent map[string]string + source.client.Transport = mintRoundTrip(func(r *http.Request) (*http.Response, error) { + raw, err := io.ReadAll(r.Body) + if err != nil { + t.Fatal(err) + } + sent = requireBrokerBody(t, raw, testSolutionID) + r.Body = io.NopCloser(strings.NewReader(string(raw))) + response := httptest.NewRecorder() + host.serve(response, r) + return response.Result(), nil + }) + if _, err := source.Credential(context.Background()); err != nil { + t.Fatal(err) + } + if got, reported := sent["upstream"]; got != tc.upstream || reported != (tc.upstream != "") { + t.Fatalf("upstream reported as %q (present: %v), want %q", got, reported, tc.upstream) + } + if sent["manifest"] == "" { + t.Fatal("the mint did not report this solution's manifest") + } + }) + } +} diff --git a/broker_credential.go b/broker_credential.go index b3c7819..5610c39 100644 --- a/broker_credential.go +++ b/broker_credential.go @@ -7,6 +7,7 @@ import ( "crypto/x509" "encoding/json" "errors" + "fmt" "io" "net" "net/http" @@ -45,12 +46,20 @@ func (c brokerCredential) Attach(r *http.Request) error { return workcontext. type brokerMintOptions struct { URL, Prefix, InternalToken, ProjectionAudience string - Audience workcontext.AuthorityValue - Authority workcontext.AuthorityPin - ProjectedToken workcontext.ProjectedTokenSource - TrustAnchor func() (*x509.CertPool, error) - ClientCertificate func() (*tls.Certificate, error) - AdmittedPeers func() ([]string, error) + // Upstream is the origin this runtime serves its backend at, reported with + // the mint as the dynamic half of its registration: the host records it + // only for the workload the projected token proves, and reaches it only + // under this workload's SVID. Empty reports nothing. + Upstream string + // Manifest is the solution's frontend manifest, as this runtime serves it: + // the other dynamic half of a solution's registration. Empty for a module. + Manifest string + Audience workcontext.AuthorityValue + Authority workcontext.AuthorityPin + ProjectedToken workcontext.ProjectedTokenSource + TrustAnchor func() (*x509.CertPool, error) + ClientCertificate func() (*tls.Certificate, error) + AdmittedPeers func() ([]string, error) } type brokerCredentialSource struct { @@ -76,11 +85,37 @@ func newBrokerCredentialSource(options brokerMintOptions) (*brokerCredentialSour if options.ProjectionAudience != "accounts" { return nil, errors.New("module-authority/PROJECTION_AUDIENCE must be accounts for the module broker") } + // At boot, where it names the cause. The host bounds what it reads, and an + // oversize body is answered there as a malformed request — which this + // client can only classify as a refusal, at boot and again at every + // renewal: a crash loop that says "request failed" about a manifest. + if size := len(options.body()); size > brokerMintBodyMaxBytes { + return nil, fmt.Errorf("the mint request is %d bytes and the module broker reads at most %d: the frontend manifest this solution reports with it is too large", size, brokerMintBodyMaxBytes) + } c := &brokerCredentialSource{options: options} c.client = c.httpClient(u) return c, nil } +// brokerMintBodyMaxBytes bounds the mint request this client will send. It is +// under what the host reads, so the bound that fires is the one that can say +// why. +const brokerMintBodyMaxBytes = 64 << 10 + +// body is the mint request: the declared prefix, and the dynamic half of this +// runtime's registration when it has one to report. +func (o brokerMintOptions) body() []byte { + body := map[string]string{"prefix": o.Prefix} + if o.Upstream != "" { + body["upstream"] = o.Upstream + } + if o.Manifest != "" { + body["manifest"] = o.Manifest + } + payload, _ := json.Marshal(body) + return payload +} + func (c *brokerCredentialSource) Credential(ctx context.Context) (executionCredential, error) { c.mu.Lock() defer c.mu.Unlock() @@ -117,7 +152,7 @@ func (c *brokerCredentialSource) obtain(ctx context.Context) (executionCredentia if strings.TrimSpace(projected) == "" { return workcontext.Credential{}, &brokerMintError{class: brokerProjection} } - payload, _ := json.Marshal(map[string]string{"prefix": c.options.Prefix}) + payload := c.options.body() req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.options.URL, bytes.NewReader(payload)) if err != nil { return workcontext.Credential{}, classifyBrokerTransport(err) diff --git a/broker_credential_test.go b/broker_credential_test.go index e538da3..304cd3f 100644 --- a/broker_credential_test.go +++ b/broker_credential_test.go @@ -4,7 +4,6 @@ import ( "context" "crypto/tls" "crypto/x509" - "encoding/json" "errors" "fmt" "io" @@ -13,7 +12,6 @@ import ( "net/http/httptest" "net/url" "path/filepath" - "reflect" "strings" "sync/atomic" "syscall" @@ -72,10 +70,7 @@ func TestBrokerExecutionProtocolConformance(t *testing.T) { if err != nil { t.Fatal(err) } - var body map[string]any - if json.Unmarshal(raw, &body) != nil || !reflect.DeepEqual(body, map[string]any{"prefix": "widgets-go"}) { - t.Fatalf("broker body must contain exactly the declared prefix: %s", raw) - } + requireBrokerBody(t, raw, "widgets-go") r.Body = io.NopCloser(strings.NewReader(string(raw))) return wire.RoundTrip(r) }) diff --git a/credential.go b/credential.go index 4c5db26..1985664 100644 --- a/credential.go +++ b/credential.go @@ -4,9 +4,11 @@ import ( "context" "crypto/tls" "crypto/x509" + "encoding/json" "errors" "fmt" "log" + "math/rand/v2" "net/http" "sync" "time" @@ -287,6 +289,8 @@ func (s *Server) mintOptions( ) brokerMintOptions { return brokerMintOptions{ URL: s.cfg.mintURL, + Upstream: s.cfg.upstream, + Manifest: s.manifestJSON(), Prefix: s.manifest.ID, InternalToken: s.cfg.brokerInternalToken, Audience: workcontext.AuthorityValue{Name: AuthorityGroup, Key: AuthorityAudienceKey}, @@ -409,6 +413,15 @@ func (s *Server) openCredential(ctx context.Context) error { // wait, and accepting a late success would make it advisory. return fmt.Errorf("obtain this execution's credential from %s: a credential arrived after the %s window closed, so this boot does not hold one", s.cfg.mintURL, window) + case awaitingInstallation(err): + // Admitted by the host, installed nowhere yet. The install step is + // delivered to this runtime's own listener and carries its own + // authority, so this boot serves — the install endpoint and health + // — holding no credential. serve keeps asking (awaitInstallation) + // and holds the first one the host issues. + s.noteCredentialAnswer(err) + log.Printf("solution %q: no organization has installed this solution yet; serving its install endpoint, and holding a credential once one has", s.manifest.ID) + return nil case !errors.Is(err, workcontext.ErrMintUnavailable): // A judgement, or a configuration this client will not even send. return fmt.Errorf("obtain this execution's credential from %s: %w", s.cfg.mintURL, err) @@ -439,6 +452,148 @@ func (s *Server) openCredential(ctx context.Context) error { // mustDeadline is the deadline of a context that has one. openCredential sets // it, so the fallback is only there to keep the arithmetic total. +// manifestJSON is the frontend manifest this runtime serves, reported with +// the mint so the host's registration carries it without a self-registration. +func (s *Server) manifestJSON() string { + if s.manifest.ID == "" { + return "" + } + encoded, err := json.Marshal(s.manifestMap()) + if err != nil { + return "" + } + return string(encoded) +} + +// awaitInstallation asks the host for this execution's credential for as long +// as this process holds none because nothing has installed the solution, and +// returns when ctx ends or the credential is refused for good. +// +// It exists because an uninstalled solution has no other asker. Health and the +// install step answer from what is already known (see uninstalled), and a +// viewer's request cannot arrive: the host routes nobody to a solution +// installed nowhere. So without this the first credential would wait for a +// request the host will not send. +// +// Every ask goes through the controller (askOnce): single-flight with any +// viewer's ask, the answer recorded where every other reader looks, and a +// terminal answer ending the process exactly as it does on any other path. An +// ask made around the controller left a refusal latched in the source and +// reported by nobody. +// +// Two cadences. Uninstalled and idle it asks slowly, with jitter so replicas +// that booted together stop asking together. After an install step has been +// acknowledged (installAcknowledged) it asks at once and then a few times in +// quick succession, because the host activates the installation only after it +// has read this runtime's answer — the first ask can race it. +func (s *Server) awaitInstallation(ctx context.Context) { + idle := time.NewTimer(awaitInstallationWait()) + defer idle.Stop() + for { + followUps := []time.Duration(nil) + select { + case <-ctx.Done(): + return + case <-s.credentialRefusedC(): + return + case <-s.installed: + followUps = awaitInstallationFollowUps + case <-idle.C: + } + for ask := 0; s.uninstalled(); ask++ { + credential, err := s.askOnce(ctx, platformRequestTimeout) + if err == nil { + seal := credential.Seal() + log.Printf("solution %q: installed; holding one execution credential, sealed to installation %s revision %d and build incarnation %d, valid until %s", + s.manifest.ID, seal.GetInstallationId(), seal.GetInstallationRevision(), seal.GetBuildIncarnation(), + credential.ExpiresAt().UTC().Format(time.RFC3339)) + break + } + if errors.Is(err, ErrCredentialRefused) || ask >= len(followUps) { + // Refused: askOnce has recorded it and the process is + // ending. Otherwise still uninstalled, or the host is not + // answering — the idle cadence asks again. + break + } + select { + case <-ctx.Done(): + return + case <-s.credentialRefusedC(): + return + case <-time.After(followUps[ask]): + } + } + if !idle.Stop() { + select { + case <-idle.C: + default: + } + } + idle.Reset(awaitInstallationWait()) + } +} + +// awaitInstallationEvery is how often an uninstalled solution asks again, and +// awaitInstallationJitter the fraction of it one ask may arrive early by. +const ( + awaitInstallationEvery = 20 * time.Second + awaitInstallationJitter = 0.2 +) + +// awaitInstallationFollowUps paces the asks after an acknowledged install +// step: the host activates the installation once it has read the answer, so +// the credential is a moment away rather than a cadence away. +var awaitInstallationFollowUps = []time.Duration{ + 250 * time.Millisecond, 500 * time.Millisecond, time.Second, 2 * time.Second, 4 * time.Second, +} + +func awaitInstallationWait() time.Duration { + return time.Duration(float64(awaitInstallationEvery) * (1 - awaitInstallationJitter*rand.Float64())) +} + +// awaitingInstallation reports the host's answer that nobody has installed +// this solution yet. +func awaitingInstallation(err error) bool { + var broker *brokerMintError + return errors.As(err, &broker) && broker.class == brokerAwaitingInstallation +} + +// uninstalled reports that this process holds no credential because the host, +// when last asked, said nothing has installed this solution. +// +// It is a reading of what the controller already recorded and asks nobody. A +// probe or an install step that asked would be one more mint attempt per +// request against a host that has already answered. +func (s *Server) uninstalled() bool { + if s.terminalErr.Load() != nil { + return false + } + s.credentialMu.Lock() + defer s.credentialMu.Unlock() + if held := s.credentialHeld; held != nil && held.Token() != "" && time.Now().Before(held.ExpiresAt()) { + return false + } + return awaitingInstallation(s.credentialAskErr) +} + +// noteCredentialAnswer records an answer the controller did not itself +// receive — the boot's — so every reader sees the same last answer. +func (s *Server) noteCredentialAnswer(err error) { + s.credentialMu.Lock() + s.credentialAskErr = err + s.credentialMu.Unlock() +} + +// installAcknowledged tells awaitInstallation that an install step has just +// been answered with success, so the credential is worth asking for now. +// Never blocks: one pending signal is as good as several. +func (s *Server) installAcknowledged() { + select { + case s.installed <- struct{}{}: + default: + } +} + func mustDeadline(ctx context.Context) time.Time { if at, ok := ctx.Deadline(); ok { return at diff --git a/installation.go b/installation.go index 9761231..f9a1ed1 100644 --- a/installation.go +++ b/installation.go @@ -5,6 +5,7 @@ import ( "encoding/json" "errors" "io" + "log" "net/http" "strings" @@ -78,6 +79,10 @@ const ( func (s *Server) handleInstallation(w http.ResponseWriter, r *http.Request) { token, err := workcontext.FromHeaders(r.Header) if err != nil { + // Named in this process's log, never in the answer: the host's job + // retries and dead-letters on a bare 401, and the operator reading + // the installation needs to know what the carrier got wrong. + log.Printf("solution %q: install step refused: %v", s.manifest.ID, err) http.Error(w, "invalid work context", http.StatusUnauthorized) return } @@ -86,9 +91,18 @@ func (s *Server) handleInstallation(w http.ResponseWriter, r *http.Request) { http.Error(w, "invalid installation", http.StatusBadRequest) return } - if err := s.actingForAViewer(r.Context()); err != nil { - installationOutcome{status: http.StatusServiceUnavailable}.write(w) - return + // An uninstalled solution holds no execution credential and needs none + // here: the step acts under the capability the host carried in, and the + // host issues this execution's credential only once the step has been + // acknowledged. Refusing the step for want of that credential is a + // solution nobody can ever install. Every other state is held to what a + // viewer's call is held to. + uninstalled := s.uninstalled() + if !uninstalled { + if err := s.actingForAViewer(r.Context()); err != nil { + installationOutcome{status: http.StatusServiceUnavailable}.write(w) + return + } } // The supplied capability is the step's bearer. No viewer session is // involved, and no viewer capability is minted for an install step. @@ -96,7 +110,13 @@ func (s *Server) handleInstallation(w http.ResponseWriter, r *http.Request) { gw.carriedContext = token gw.orgID = installation.OrganizationID ctx := context.WithValue(r.Context(), mcpViewerKey{}, gw) - s.installOnce(ctx, installation).write(w) + outcome := s.installOnce(ctx, installation) + outcome.write(w) + if outcome.status == http.StatusNoContent { + // After the answer is written: the host activates the installation + // when it reads it, and only then is there a credential to hold. + s.installAcknowledged() + } } // decodeInstallation bounds the entire body, including whitespace after the diff --git a/mint_errors.go b/mint_errors.go index 39e6322..6eeb86a 100644 --- a/mint_errors.go +++ b/mint_errors.go @@ -68,6 +68,10 @@ const ( brokerTransport brokerInvalidAnswer brokerProjection + // brokerAwaitingInstallation: the host admits this workload and has not + // been asked to install it anywhere yet. The install step arrives on this + // runtime's own listener; until then there is nothing to mint for. + brokerAwaitingInstallation ) type brokerTransportCategory uint8 @@ -92,6 +96,8 @@ func (e *brokerMintError) Error() string { return fmt.Sprintf("module broker: HTTP %d: request refused", e.status) case brokerNotEstablished: return "module broker: HTTP 503: execution not established" + case brokerAwaitingInstallation: + return "module broker: HTTP 412: no organization has installed this solution yet" case brokerOtherStatus: return fmt.Sprintf("module broker: HTTP %d: request failed", e.status) case brokerInvalidAnswer: @@ -113,7 +119,7 @@ func (e *brokerMintError) Error() string { } } func (e *brokerMintError) retryable() bool { - return e.class == brokerNotEstablished || e.class == brokerTransport && + return e.class == brokerNotEstablished || e.class == brokerAwaitingInstallation || e.class == brokerTransport && (e.transport == brokerTransportTimeout || e.transport == brokerTransportRefused || e.transport == brokerTransportReset) } func (e *brokerMintError) Unwrap() error { @@ -132,6 +138,8 @@ func brokerStatusError(status int) *brokerMintError { e.class = brokerRefused case http.StatusServiceUnavailable: e.class = brokerNotEstablished + case http.StatusPreconditionFailed: + e.class = brokerAwaitingInstallation } return e } diff --git a/solution.go b/solution.go index affc191..36037c8 100644 --- a/solution.go +++ b/solution.go @@ -324,7 +324,10 @@ type Server struct { // cannot act for anyone and nothing it does locally changes that. terminal chan struct{} terminalOnce sync.Once - terminalErr atomic.Pointer[error] + // installed signals awaitInstallation that an install step was just + // acknowledged. One slot: a pending signal is as good as several. + installed chan struct{} + terminalErr atomic.Pointer[error] // The one in-flight credential ask and what it last produced. See askOnce: // without single-flight, a credential entering its renewal window turns // every concurrent viewer request into its own audited mint. @@ -384,7 +387,12 @@ type config struct { assetsDir string // mintURL is the explicitly configured module broker endpoint. The prefix // comes from Manifest.ID; authority audiences remain frozen separately. - mintURL string + mintURL string + // upstream is the https origin peers reach this runtime at, reported at + // mint (see brokerMintOptions.Upstream), and upstreamErr is why there is + // none. See advertisedOrigin. + upstream string + upstreamErr error brokerInternalToken string // projectedTokenPath is the file the platform projects this workload's // service-account token into. It is re-read before every renewal, because @@ -520,6 +528,134 @@ func env(key, fallback string) string { // "localhost:8080"), and other endpoints as URLs; url.Parse reads the bare form // as scheme "localhost" with no port, which left every rendered solution // refusing to boot with an empty listen port. Both forms resolve here. +// advertisedOrigin is the https origin this runtime reports to the host as +// where it serves: what the gateway dials to proxy to it. +// +// It is the address Codefly ADVERTISES for this service's http endpoint — the +// CODEFLY__SELF_ENDPOINT__ carrier, read through core's accessor — and never +// the one it listens on. The two differ exactly where it matters: a rendered +// cell localizes a service's own CODEFLY__ENDPOINT__ carrier to +// "localhost:8080", so a runtime that reported its listen address told the +// gateway to proxy the solution to the gateway's own loopback. +// +// Without the carrier, a process on a developer machine reports the loopback +// port it actually listens on — PORT included, since that is the port the +// listener binds — because the gateway beside it reaches it there. In a +// deployed runtime context that answer is a loopback address no peer can +// dial, and validate() refuses the boot naming the carrier. +// +// The scheme is always https: the listener is TLS-only, whatever scheme the +// carrier's value was written with. +func advertisedOrigin(ctx context.Context, self, port string, environment []string) (string, error) { + module, service := os.Getenv(resources.ModulePrefix), os.Getenv(resources.ServicePrefix) + info := &resources.EndpointInformation{Module: module, Service: service, Name: "http", API: "http"} + carrier := resources.SelfEndpointAsEnvironmentVariableKey(info) + if module != "" && service != "" { + instance, err := resources.FindSelfNetworkInstanceInEnvironmentVariables(ctx, info, environment) + switch { + case err == nil: + origin, err := httpsOrigin(instance.Address) + if err != nil { + return "", fmt.Errorf("the advertised address in %s is not one a peer can dial: %w", carrier, err) + } + return origin, nil + case !errors.Is(err, resources.ErrEndpointCarrierAbsent): + // Present and not an address. Never treated as absence: falling + // back to the listen address here is how a typo becomes a + // solution proxied to the gateway's own loopback. + return "", fmt.Errorf("the advertised address in %s cannot be read: %w", carrier, err) + } + } + // No carrier. The listen address, on the port actually bound. + host := "localhost" + if listening := hostnameOf(self); listening != "" { + host = listening + } + if port == "" { + // Nothing to report, and nothing to add: validate() refuses an + // unresolved listen port in its own words. + return "", nil + } + origin, err := httpsOrigin(net.JoinHostPort(host, port)) + if err != nil { + return "", fmt.Errorf("no advertised address in %s, and the listen address is not one a peer can dial: %w", carrier, err) + } + return origin, nil +} + +// httpsOrigin is "https://host:port" for an address written as host:port or as +// a URL, and an error for anything that is not exactly a host and a port. +// +// Strict, because this value is handed to another component to dial. A path, a +// query, userinfo, a missing port, a port that is not one, and a host that +// names no machine (the unspecified address a listener binds) are all refused +// rather than trimmed: each is a destination that is not the one somebody +// meant. +func httpsOrigin(raw string) (string, error) { + hostport := strings.TrimSpace(raw) + if strings.Contains(hostport, "://") { + u, err := url.Parse(hostport) + if err != nil { + return "", errors.New("it does not parse as a URL") + } + if u.User != nil || u.RawQuery != "" || u.Fragment != "" || strings.Trim(u.Path, "/") != "" { + return "", errors.New("it carries more than a host and a port") + } + hostport = u.Host + } + host, port, err := net.SplitHostPort(hostport) + if err != nil { + return "", errors.New("it is not a host and a port") + } + if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 { + return "", errors.New("its port is not a port") + } + if host == "" || strings.ContainsAny(host, "/@?# ") { + return "", errors.New("it names no host") + } + if ip := net.ParseIP(host); ip != nil && ip.IsUnspecified() { + return "", errors.New("it is the unspecified address, which names no machine") + } + return "https://" + net.JoinHostPort(host, port), nil +} + +// hostnameOf is the host of a listen address written as host:port or as a URL, +// or "" when it names none — a bare ":8080", or the unspecified address. +func hostnameOf(raw string) string { + hostport := raw + if strings.Contains(raw, "://") { + u, err := url.Parse(raw) + if err != nil { + return "" + } + hostport = u.Host + } + host, _, err := net.SplitHostPort(hostport) + if err != nil { + return "" + } + if ip := net.ParseIP(host); ip != nil && ip.IsUnspecified() { + return "" + } + return host +} + +// loopbackOrigin reports whether origin names this machine: localhost (or a +// name under .localhost) or a loopback IP. Such an origin is reachable only +// from the process's own host. +func loopbackOrigin(origin string) bool { + u, err := url.Parse(origin) + if err != nil { + return false + } + host := strings.ToLower(strings.TrimSuffix(u.Hostname(), ".")) + if host == "localhost" || strings.HasSuffix(host, ".localhost") { + return true + } + ip := net.ParseIP(host) + return ip != nil && ip.IsLoopback() +} + func listenPort(self string) string { if strings.Contains(self, "://") { if u, err := url.Parse(self); err == nil { @@ -700,12 +836,15 @@ func loadConfig(ctx context.Context, id string, initErr error) config { public := strings.TrimRight(env("PUBLIC_URL", ""), "/") // Own endpoint: the port Codefly assigned this service, not a fixed default. + self := address(ctx, "", "", "http", "http") port := env("PORT", "") - if port == "" { - if self := address(ctx, "", "", "http", "http"); self != "" { - port = listenPort(self) - } + if port == "" && self != "" { + port = listenPort(self) } + // Where peers reach this runtime, reported with the mint: the address + // Codefly ADVERTISES for this endpoint, never the one it listens on. See + // advertisedOrigin. + upstream, upstreamErr := advertisedOrigin(ctx, self, port, os.Environ()) // Host endpoints, resolved via the SDK (no localhost:port literals). gatewayURL := strings.TrimRight(env("GATEWAY_URL", resolveGateway(ctx, hostModule, hostGateway)), "/") @@ -727,6 +866,8 @@ func loadConfig(ctx context.Context, id string, initErr error) config { assetsDir: env("ASSETS_DIR", "../fe-remote/dist"), // The broker path is published, but its origin remains explicit. mintURL: strings.TrimSpace(env(CredentialMintURLEnvironmentVariable, "")), + upstream: upstream, + upstreamErr: upstreamErr, dialled: dialledAddresses(gatewayURL, strings.TrimSpace(env(CredentialMintURLEnvironmentVariable, ""))), projectedTokenPath: projectedTokenPath, identityCertFile: workloadPath(ctx, IdentityCertFileEnvironmentVariable, WorkloadIdentityCertFileKey), @@ -967,6 +1108,23 @@ func (c config) validate() error { if p, err := strconv.Atoi(c.port); err != nil || p < 1 || p > 65535 { return fmt.Errorf("unresolved listen port %q: set PORT or ensure the SDK resolves this service's http endpoint", c.port) } + // Where the gateway is told to dial this runtime. Checked at boot because + // it is reported with the mint, where the host could only store it: a + // wrong origin is a solution that installs and then answers nothing. + if c.upstreamErr != nil { + if deployedRuntimeContext(c.runtimeContext) { + return fmt.Errorf("no origin to report to the host in the deployed runtime context %q: %w — Codefly's render injects the address peers reach this service at as %s______HTTP__HTTP, and a deployed runtime never falls back to the address it listens on", + c.runtimeContext, c.upstreamErr, resources.SelfEndpointPrefix) + } + return fmt.Errorf("no origin to report to the host: %w", c.upstreamErr) + } + // A loopback origin reported from a deployment is reachable by no peer: + // the gateway, dialling its own localhost, would proxy this solution to + // itself. + if deployedRuntimeContext(c.runtimeContext) && loopbackOrigin(c.upstream) { + return fmt.Errorf("the origin this runtime would report to the host, %q, is a loopback address in the deployed runtime context %q: the gateway would proxy this solution to itself. Codefly's render injects the reachable address as %s______HTTP__HTTP", + c.upstream, c.runtimeContext, resources.SelfEndpointPrefix) + } // Both of these carry credentials: the mint presents the projected token // that attests which workload this process is, and the gateway carries the // viewer's bearer and the capability minted for them. So each has to be @@ -1459,7 +1617,7 @@ func New(manifest Manifest) *Server { // unset announced that product's capability contract to the host. manifest.Contract = manifest.ID } - return &Server{manifest: manifest, handlers: make(map[string]RequestHandler), terminal: make(chan struct{})} + return &Server{manifest: manifest, handlers: make(map[string]RequestHandler), terminal: make(chan struct{}), installed: make(chan struct{}, 1)} } // Handle registers a solution endpoint. Chainable. @@ -1643,6 +1801,11 @@ func (s *Server) serve(ctx context.Context, ln net.Listener) error { } _ = srv.Close() }() + // For as long as this server serves, and no longer: Serve hands this + // function a context nothing cancels. + serving, stopped := context.WithCancel(ctx) + defer stopped() + go s.awaitInstallation(serving) log.Printf("solution %q listening on :%s (gateway=%s, profile=%s)", s.manifest.ID, s.cfg.port, redactedURL(s.cfg.gatewayURL), s.cfg.profile) if serveErr := srv.Serve(ln); !errors.Is(serveErr, http.ErrServerClosed) { @@ -1686,6 +1849,14 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) { // and hands the same one back until its renewal point, so this is a // mutex and a comparison except at the renewal the credential's own expiry // dictates. Nothing here runs on a timer this runtime chose. + // Admitted and installed nowhere is healthy: this process serves the one + // thing it can be asked for, its install step, and a probe that failed it + // would have the orchestrator restart a solution for not being installed. + // Answered from what the host last said, without asking it again. + if s.uninstalled() { + w.WriteHeader(http.StatusOK) + return + } if s.credential != nil || s.brokerCredential != nil { // Through credentialWithin, so the deadline is real: passing a context // to the SDK's client does not bound it, because what blocks is a diff --git a/uninstalled_test.go b/uninstalled_test.go new file mode 100644 index 0000000..716e556 --- /dev/null +++ b/uninstalled_test.go @@ -0,0 +1,304 @@ +package solution + +import ( + "context" + "crypto/tls" + "crypto/x509" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/codefly-dev/core/resources" +) + +// postInstall sends one well-formed install step to a booted runtime as the +// admitted caller, and returns the status it answered. +func postInstall(t *testing.T, b *booted) int { + t.Helper() + r := installRequest(t, installBody) + r.URL.Scheme, r.URL.Host = "https", strings.TrimPrefix(b.base, "https://") + r.RequestURI = "" + resp, err := b.client.Do(r) + if err != nil { + t.Fatal(err) + } + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + return resp.StatusCode +} + +func healthOf(t *testing.T, b *booted) int { + t.Helper() + resp, err := b.client.Get(b.base + HealthPath) + if err != nil { + t.Fatal(err) + } + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + return resp.StatusCode +} + +// TestAnUninstalledSolutionServesItsInstallStepAndThenHoldsACredential is the +// order an installation actually happens in, against a host that behaves as +// the real one does: it answers 412 while nothing has installed the solution, +// delivers the install step, and issues only once that step was acknowledged. +// +// A runtime that needs its execution credential to answer the step can never +// be installed by such a host — each waits for the other — and one whose +// health fails while uninstalled is restarted for not being installed. +func TestAnUninstalledSolutionServesItsInstallStepAndThenHoldsACredential(t *testing.T) { + var active atomic.Bool + var steps atomic.Int32 + host := newHostMint(t, &hostMint{answer: func() int { + if active.Load() { + return 0 + } + return http.StatusPreconditionFailed + }}) + s := New(Manifest{ID: testSolutionID}).OnInstall(func(context.Context, Installation) error { + steps.Add(1) + return nil + }) + b := boot(t, s, host) + t.Cleanup(b.client.CloseIdleConnections) + b.client.Timeout = 5 * time.Second + + if !s.uninstalled() { + t.Fatal("a boot the host answered 412 does not read as uninstalled") + } + if _, held := s.heldCredential(); held { + t.Fatal("an uninstalled boot holds a credential") + } + asked := host.count() + + // Healthy, and answered from what the host already said. + for range 3 { + if status := healthOf(t, b); status != http.StatusOK { + t.Fatalf("health while uninstalled = %d, want 200: a failing probe has the orchestrator restart a solution for not being installed", status) + } + } + if got := host.count(); got != asked { + t.Fatalf("three probes asked the host %d more time(s); a probe of an uninstalled solution asks nobody", got-asked) + } + + // The step is delivered while the host still answers 412, exactly as the + // host orders it, and the host activates only after reading the answer. + if status := postInstall(t, b); status != http.StatusNoContent { + t.Fatalf("install step while uninstalled = %d, want 204: the host issues nothing until this step is acknowledged, so refusing it is a solution nobody can install", status) + } + if steps.Load() != 1 { + t.Fatalf("the install step ran %d time(s), want 1", steps.Load()) + } + time.Sleep(100 * time.Millisecond) + active.Store(true) + + // Held promptly — well inside the idle cadence, because the acknowledged + // step is what says the credential is now worth asking for. + deadline := time.Now().Add(awaitInstallationEvery / 2) + for { + if _, held := s.heldCredential(); held { + break + } + if time.Now().After(deadline) { + t.Fatalf("no credential held %s after the install step was acknowledged and the host began issuing", awaitInstallationEvery/2) + } + time.Sleep(20 * time.Millisecond) + } + if s.uninstalled() { + t.Fatal("a runtime holding a credential still reads as uninstalled") + } + if status := healthOf(t, b); status != http.StatusOK { + t.Fatalf("health once installed = %d, want 200", status) + } +} + +// TestARefusalAfterAnUninstalledBootEndsTheProcess: the host admitted this +// workload at boot and judges it differently when next asked. That answer has +// to end the process the way it does on every other path; an ask made around +// the controller logged it and left a process serving on a refused build. +func TestARefusalAfterAnUninstalledBootEndsTheProcess(t *testing.T) { + var refuse atomic.Bool + host := newHostMint(t, &hostMint{answer: func() int { + if refuse.Load() { + return http.StatusForbidden + } + return http.StatusPreconditionFailed + }}) + s := New(Manifest{ID: testSolutionID}).OnInstall(func(context.Context, Installation) error { return nil }) + b := boot(t, s, host) + t.Cleanup(b.client.CloseIdleConnections) + b.client.Timeout = 5 * time.Second + + refuse.Store(true) + if status := postInstall(t, b); status != http.StatusNoContent { + t.Fatalf("install step = %d, want 204", status) + } + err, ended := b.ended(10 * time.Second) + if !ended { + t.Fatal("the host refused this execution's credential after an uninstalled boot and the process kept serving") + } + if err == nil || !strings.Contains(err.Error(), "stopped serving") { + t.Fatalf("serve returned %v; a refusal is reported as the reason the process ended", err) + } +} + +// TestAnInstalledRuntimeStillHoldsTheInstallStepToItsCredential: the exemption +// is for the one state that needs it. A runtime whose credential the host has +// stopped issuing for any other reason answers the step as it answers a viewer. +func TestAnInstalledRuntimeStillHoldsTheInstallStepToItsCredential(t *testing.T) { + var steps atomic.Int32 + s, _, _ := brokerFixture(t) + s.OnInstall(func(context.Context, Installation) error { steps.Add(1); return nil }) + s.noteCredentialAnswer(&brokerMintError{class: brokerTransport}) + s.credentialMu.Lock() + s.credentialQuietUntil = time.Now().Add(time.Minute) + s.credentialMu.Unlock() + if s.uninstalled() { + t.Fatal("a host that cannot be reached reads as an uninstalled solution") + } + w := httptest.NewRecorder() + s.handleInstallation(w, installRequest(t, installBody)) + if w.Code != http.StatusServiceUnavailable || steps.Load() != 0 { + t.Fatalf("install step with no obtainable credential = %d after %d run(s), want 503 and none", w.Code, steps.Load()) + } +} + +func TestAwaitInstallationCadenceIsJitteredDownward(t *testing.T) { + low := time.Duration(float64(awaitInstallationEvery) * (1 - awaitInstallationJitter)) + distinct := map[time.Duration]bool{} + for range 200 { + wait := awaitInstallationWait() + if wait > awaitInstallationEvery || wait < low { + t.Fatalf("wait %s outside [%s, %s]", wait, low, awaitInstallationEvery) + } + distinct[wait] = true + } + if len(distinct) < 2 { + t.Fatal("every replica waits the same time, so replicas that boot together ask together") + } +} + +// selfCarrier is the advertised-address carrier for the service these tests +// run as. +func selfCarrier(t *testing.T, value string) []string { + t.Helper() + t.Setenv(resources.ModulePrefix, "wiki") + t.Setenv(resources.ServicePrefix, "backend") + key := resources.SelfEndpointAsEnvironmentVariableKey(&resources.EndpointInformation{ + Module: "wiki", Service: "backend", Name: "http", API: "http", + }) + if value == "" { + return nil + } + return []string{key + "=" + value} +} + +// TestTheReportedOriginIsTheAdvertisedAddress: a rendered cell hands a service +// its own endpoint as the address it listens on, localized to localhost, and +// the address its peers reach it at under a separate carrier. Reporting the +// former tells the gateway to proxy the solution to the gateway's own loopback. +func TestTheReportedOriginIsTheAdvertisedAddress(t *testing.T) { + for _, tc := range []struct { + name, self, port, advertised, want string + }{ + {"a rendered cell", "localhost:8080", "8080", "http://backend.platform-wiki.svc.cluster.local:8080", "https://backend.platform-wiki.svc.cluster.local:8080"}, + {"a cell whose listen port was overridden", "localhost:8080", "9443", "http://backend.platform-wiki.svc.cluster.local:8080", "https://backend.platform-wiki.svc.cluster.local:8080"}, + {"an advertised host and port with no scheme", "localhost:8080", "8080", "backend.platform-wiki.svc:8080", "https://backend.platform-wiki.svc:8080"}, + {"an advertised IPv6 address", "localhost:8080", "8080", "http://[fd00::12]:8080", "https://[fd00::12]:8080"}, + {"no carrier, on a developer machine", "localhost:22631", "22631", "", "https://localhost:22631"}, + {"no carrier, and a listen port overridden by PORT", "http://127.0.0.1:22631", "9443", "", "https://127.0.0.1:9443"}, + {"no carrier, and a listen address naming no host", ":8080", "8080", "", "https://localhost:8080"}, + {"no carrier, listening on the unspecified address", "0.0.0.0:8080", "8080", "", "https://localhost:8080"}, + {"no carrier and no resolved endpoint", "", "8080", "", "https://localhost:8080"}, + } { + t.Run(tc.name, func(t *testing.T) { + got, err := advertisedOrigin(context.Background(), tc.self, tc.port, selfCarrier(t, tc.advertised)) + if err != nil || got != tc.want { + t.Fatalf("advertisedOrigin = %q, %v; want %q", got, err, tc.want) + } + }) + } +} + +// TestAnAdvertisedAddressThatIsNotOneIsRefusedNotReplaced: a carrier that is +// present and wrong is never read as absent. Falling back to the listen +// address is how a typo becomes a loopback upstream. +func TestAnAdvertisedAddressThatIsNotOneIsRefusedNotReplaced(t *testing.T) { + for name, advertised := range map[string]string{ + "not an address": "backend", + "a path": "http://backend.platform-wiki.svc.cluster.local:8080/api", + "userinfo": "http://ops@backend.platform-wiki.svc.cluster.local:8080", + "a query": "http://backend.platform-wiki.svc.cluster.local:8080?x=1", + "the unspecified address": "http://0.0.0.0:8080", + "a port that is not one": "backend.platform-wiki.svc:0", + } { + t.Run(name, func(t *testing.T) { + got, err := advertisedOrigin(context.Background(), "localhost:8080", "8080", selfCarrier(t, advertised)) + if err == nil || got != "" { + t.Fatalf("advertisedOrigin(%q) = %q, %v; want a refusal and nothing to report", advertised, got, err) + } + }) + } +} + +// TestADeployedRuntimeRefusesToReportItsOwnLoopback is the boot refusal: in a +// deployed runtime context a loopback origin is reachable by no peer, whether +// it came from a missing carrier or was advertised. +func TestADeployedRuntimeRefusesToReportItsOwnLoopback(t *testing.T) { + load := func(t *testing.T, runtimeContext, advertised string) (config, error) { + t.Helper() + bootableEnvironment(t) + t.Setenv(resources.RuntimeContextPrefix, runtimeContext) + for _, entry := range selfCarrier(t, advertised) { + key, value, _ := strings.Cut(entry, "=") + t.Setenv(key, value) + } + c := loadConfig(context.Background(), "solution-under-test", nil) + return c, c.validate() + } + t.Run("a developer machine reports its loopback port", func(t *testing.T) { + c, err := load(t, resources.RuntimeContextNative, "") + if err != nil || c.upstream != "https://localhost:8090" { + t.Fatalf("upstream %q, %v; want the loopback port this process listens on", c.upstream, err) + } + }) + t.Run("a cell reports the advertised address", func(t *testing.T) { + c, err := load(t, "kubernetes", "http://backend.platform-wiki.svc.cluster.local:8080") + if err != nil || c.upstream != "https://backend.platform-wiki.svc.cluster.local:8080" { + t.Fatalf("upstream %q, %v; want the advertised cluster-local origin", c.upstream, err) + } + }) + for name, advertised := range map[string]string{ + "a cell with no carrier": "", + "a cell advertising localhost": "http://localhost:8080", + "a cell advertising a loopback IP": "http://127.0.0.1:8080", + } { + t.Run(name, func(t *testing.T) { + _, err := load(t, "kubernetes", advertised) + if err == nil || !strings.Contains(err.Error(), resources.SelfEndpointPrefix) { + t.Fatalf("validate = %v; want a refusal naming the %s carrier", err, resources.SelfEndpointPrefix) + } + }) + } +} + +// TestAnOversizeMintRequestIsRefusedAtBoot: the host bounds what it reads and +// answers an oversize body as malformed, which this client can only classify +// as a refusal — at boot and at every renewal after it. +func TestAnOversizeMintRequestIsRefusedAtBoot(t *testing.T) { + s, _, _ := brokerFixture(t) + options := s.mintOptions("accounts", func() (*x509.CertPool, error) { return x509.NewCertPool(), nil }, func(*tls.CertificateRequestInfo) (*tls.Certificate, error) { return &tls.Certificate{}, nil }) + options.Authority = fixedAuthority{audience: testAudience} + options.Manifest = `{"pad":"` + strings.Repeat("a", brokerMintBodyMaxBytes) + `"}` + if _, err := newBrokerCredentialSource(options); err == nil || !strings.Contains(err.Error(), "manifest") { + t.Fatalf("an oversize mint request was accepted at boot (%v); it must be refused naming the manifest", err) + } + options.Manifest = `{"pad":"` + strings.Repeat("a", brokerMintBodyMaxBytes/2) + `"}` + if _, err := newBrokerCredentialSource(options); err != nil { + t.Fatalf("a mint request inside the bound was refused: %v", err) + } +}